Log4j Vulnerability

Critical Vulnerability in Apache Log4J. This affects you.

A new Remote Code Execution (RCE) vulnerability (identified as CVE-2021-44228) has been discovered in the Apache Java module, log4j. More vulnerabilities, CVE-2021-45046  and CVE-2021-45105, were discovered that also require critical review. Without wishing to sound alarmist, this may be the most serious vulnerability we’ve faced in the last decade. Once the dust has settled, it may be the most serious vulnerability in the history of modern cyber security.

If you’re reading this, it almost certainly impacts you.

Update: 18 December 2021

• Added information on Log4j version 2.17 and CVE-2021-45105.

Update: 17 December 2021

• Updated to mention the first claim that Conti ransomware operators are gaining access through the Log4j vulnerability.
• Updated to include reference to show that CVE-2021-45046 could potentially lead to remote code execution under certain conditions.
• Added that deleting JndiLookup class is dangerous and must only be considered as a final solution.
• Mention that Log4j version 2.17 is planned for release some time.
• Updated timeline graphic to match the SharePoint site’s version.

Update: 16 December 2021

• Updated to reflect that CVE-2021-45046 can “allow for exfiltration of sensitive data in certain circumstances” and reiterate that one should upgrade to 2.16.0 when possible
• Updated to report that more than 300 products are already reported vulnerable on the Dutch government CERT list, and will probably reach 500 before next week.
• Updated to advertise that Orange Cyberdefense is now sharing our own Indicators of Compromise (IoC) publicly via GitHub at https://github.com/Orange-Cyberdefense/log4shell_iocs

Update: 15 December 2021

• Updated to emphasize that we are now recommending an upgrade to 2.16 to mitigate the threat
• Updated to report that a third CVE number was assigned to a vulnerability bypass of the 2.15 version under certain non-default configurations: CVE-2021-45046. Apparently not an RCE at this point. It’s fixed by version 2.16 already.
• Updated to reflect JNDI lookup supports protocols such as LDAP, RMI, DNS, or even IIOP, and we’ve finally noticed exploitation attempts using IIOP today.
• Updated to emphasize that log4j is a library, so we cannot 100% confirm this via a list of “installed” software.
• Updated to report on some newly-confirmed products, including IBM, AWS, and Zoho, and to report new trackers from CISA and NCSC-NL.
• Updated to clarify that if the exploitation is successful, the JDNI string is not actually logged (because it’s interpreted by the Log4j logger).
• Updated to reflect that Web Application Firewalls (WAF) can offer reasonable protection against the majority of know attacks.
• Updated to report that ‘Khonsari’, a new simple ransomware strain, was seen deployed after a log4j attack.

Update: 14 December 2021

• Updated with an example attack string.
• Updated to clarify that the vulnerable platform itself does not have to be internet-facing. Any remotely accessible interface that offers a path to log4j can be exploited as a vector.
• Updated to share the iPhone / iCloud attack example to illustrate the point above.
• Updated to emphasize the Environment Variables attack scenario.
• Updated with an improved image to illustrate the attack surface.
• Updated with a timeline.
• Updated with incident trend data from our Nordic CyberSOC.
• Updated with new patching details.
• Updated with a link to the SwitHalk Blueteam cheatsheet, which we contribute to.
• Updated to include a vulnerability decision tree.
• Added a reference to the work-around formatMsgNoLookups=true.

What it means

TL;DR: You need to locate and update all instances of log4j and update to version 2.17.0 to mitigate this threat fully.

Apache log4j is a Java-based logging framework that is used in a multitude of custom applications, off-the-shelf software, security products, and cloud applications like Steam and Apple iCloud.

The list of applications impacted by this vulnerability is vast and growing hourly as our understanding of the problem develops. The log4j component is a highly popular logging library for Java and widely used within Java environments.

The security community has been working non-stop since Friday 10^th to get a handle on the extent of the problem, and there is some emerging understanding of what technologies are affected, but this is only the start of a very long story and we should anticipate that more affected systems will be revealed for several weeks to come.

The vulnerability is triggered via a variety of programmatic paths to the vulnerable log4j component, as the Minecraft and Apple examples below illustrate. The attacker only needs an interface that accepts string input from outside, and somehow forwards that string to the underlying log4j component.

The attack string takes the general form

${jndi:protocol://server}

‘JNDI’ stands for ‘Java Naming and Directory Interface’. It’s an API (Application Program Interface) that is used to fetch resources from a ‘database’, including Lightweight Directory Access Protocol (LDAP), Domain Name Service (DNS) and Java Remote Method Invocation (RMI). There are many more protocol layers available, as we will emphasize later.

The log4j library recognizes, interprets, and executes this JNDI request when it is included in a logged message, resulting in the Java system on the logging platform executing a variety of actions, including connecting to a remote server to download Java code or perform further resource retrieval. This is when the attackers can slip in malicious code.

The format of this log4j lookup string in Java is complex and nuanced. For example, instead of ${ jndi:} one could use ${${lower:jn}${lower:di, which results in the same thing. This clearly makes detection very difficult.

Here are some examples of the form such an attack-string might take:

The attack surface is almost limitless, as long as the attack string finds it way to a vulnerable component.

In one instance, the vulnerability has been exploited on the popular Minecraft gaming platform simply by typing a specially crafted message into the game’s chat window. In another revealing example a researcher illustrated that the Apple iCloud infrastructure could be attacked by inserting the appropriately formatted string in Device Name field in the iPhone Settings interface. There are also anecdotes about the exploit being triggered via an SMS text message to an automated backend system. In these kinds of examples, the successful exploit is confirmed via a DNS call back, in which a command is issued to resolve a DNS name in a domain the attacker can monitor.

Log4 fun and profit

The exploit for this vulnerability is being referred to as ‘log4shell’ and involves creating a carefully crafted text string that finds its way through to the log4j logging component, where it causes the specified command to be executed on the underlying operating system.

The string that triggers the exploit can be crafted in a variety of a different formats – a technique known as obfuscation. This makes detecting an attack via logs or on the network a significant challenge.

The exploit is what’s known as ‘blind’, meaning that the attacker doesn’t receive direct feedback on whether it succeeded. A typical approach would therefore be to attempt to trigger a ‘reverse’ connection from the exploited component to a server the attacker controls on the internet from which remote command and control can be established. Preventing outbound connections from internet-facing servers would help to mitigate this approach, but it is by no means the only option the attacker has.

There are many avenues open to an attacker that don’t require real-time command and control, so this challenge really shouldn’t feature in your risk calculus. One popular vector so far has surprisingly been to deploy crypto coin mining malware.

The researcher Marcus Hutchins has warned that if system environment variables are set, they could also be referenced and fetched inside the Java context. In one example he provides, an AWS_SECRET_ACCESS_KEY is set on the server and fetched via the exploit. Environment variables could also include database connection strings & passwords, and other secrets.

We believe this variation of the attack would be even more difficult to detect, since it doesn’t require any additional network traffic or risk logging on the affected platform.

We also note that the Java Naming and Directory Interface (JNDI – used to trigger the exploit) supports protocols such as LDAP, RMI, DNS, or even ‘Internet Inter-ORB Protocol (IIOP) – sed to facilitate network interaction between distributed programs written in different programming languages. Our CyberSOCs have now observed exploitation attempts using IIOP.

We shouldn’t make the mistake of underestimating the creativity of our adversaries or the level of mayhem that this vulnerability will enable.

Attack activity

News of the vulnerability became publicly known on Friday, December 10^th. However, there is evidence to suggest that there have been attempts to exploit the issue since at least December 1^st, suggesting the exploit has been in the wild for at least 9 days before it was publicly disclosed. We may yet discover that the exploit is even older than that.

Since being publicly disclosed on the 10^th, several exploits have been developed and published, and there are widespread reports of generic scans and targeted attacks.

The bottom line for you is simple: Firstly, you may already have been targeted using this vulnerability sometime in the past. Secondly, your internet-facing systems will certainly be facing scans for generic manifestations of the vulnerability and will be compromised if possible, even for something as banal as a botnet or a cryptocurrency miner. Finally, you should anticipate that the vulnerability will be exploited in more sophisticated, targeted attacks in the near future, and for lateral movement across your networks once an initial compromise has been accomplished in some other way.

News of the vulnerability became publicly known on Friday December 10^th. However, there is evidence to suggest that there have been attempts to exploit the issue since at least December 1^st, suggesting the exploit has been in the wild for at least 9 days before it was publicly disclosed. We may yet discover that the exploit is even older than that.

Since being publicly disclosed on the 10^th, several exploits have been developed and published, and there are widespread reports of generic scans and targeted attacks.

Incidents detected at clients by our Nordic and DACH CyberSOC and SOC

In a December 13th Bitdefender reported that they have detected an attempt to use the log4j vulnerability to deploy a new ransomware family they called ‘Khonsari’ (https://businessinsights.bitdefender.com/technical-advisory-zero-day-critical-vulnerability-in-log4j2-exploited-in-the-wild). It later emerged that ‘Kharen Konsari’ is a real person and that the phone number and the email shown in the ransom note probably point to her. The whole incident may simply be an elaborate ruse, therefore. Although we have not yet witnessed it, we do anticipate that ransomware crews start exploiting this vulnerability in anger in the near future. Attacks of this kind are already being attributed to state actors like Phosphorus, Nemesis Kitten and Hafnium.

On December 17 AdvIntel published a blog claiming that the Conti ransomware group has started targeting the Log4j vulnerability, tracked as CVE-2021-44228. AdvIntel specifically mentions that Conti is interested in VMware vCenter Server and vCenter Cloud Gateway that also contains this vulnerability.

The bottom line for you is simple: Firstly, you may already have been targeted using this vulnerability sometime in the past. Secondly, your internet-facing systems will certainly be facing scans for generic manifestations of the vulnerability and will be compromised, if possible, even for something as banal as a botnet or a cryptocurrency miner. Finally, you should anticipate that the vulnerability will be exploited in more sophisticated, targeted attacks in the near future, and for lateral movement across your networks once an initial compromise has been accomplished in some other way.

What we know about the impact

As we mentioned before, our understanding of the scale of the problem is still in its early stages. Steel yourself for a long, ongoing struggle as the scope continues to expand.

As of December 16^th we added ivanti MobileIron to the list of vulnerable platforms, which is also being actively exploited in the wild. The total list of known vulnerable systems now exceeds 300 and will probably reach 500 by the end of the week

Information about vulnerable, possibly-vulnerable and not-vulnerable products is in fact changing so rapidly that, rather than attempt to keep this resources updated, we have opted to share any new intelligence we collect in collaboration via GitHub here:

• https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592
• US Cybersecurity & Infrastructure Security Agency (CISA):
  https://github.com/cisagov/log4j-affected-db
• The Dutch government Nationaal Cyber Security Centrum – NCSC
  https://github.com/NCSC-NL/log4shell .

We expect to see these two lists merging over time.

As the vendors are still in the process of investigating this vulnerability, our CERT is engaged to continuously update our Vulnerability Intelligence Database (). We will update via the GitHub page above, via our Managed Vulnerability Intelligence Watch service, or directly via the services you engage us for.

What we are doing

Our multi-disciplinary crisis management team has been engaged with this issue since we published our first World Watch advisory on Friday. We have been meeting regularly to communicate updates and track progress regularly all weekend. Our team includes:

• CERT
• Research
• The Office of the CISO
• Internal cybersecurity
• Internal IT
• Global operations – Managed Detection and Response and Vulnerability Management
• Penetration Testing
• Computer Security Incident Response Team (CSIRT)

Our efforts have focused on the following:

• Tracking the scope and the scale of the issue, variations, attack vectors, affected products, and potential mitigations
• Engaging with our vendors and partners
• Updating our customers via our World Watch advisory service
• Updating our detection capabilities across our world-wide SOCs and CyberSOC
• Updating our vulnerability scanning platform with the most current detections available
• Triaging and patching our own systems to gauge the counter any potential impact to our environment and ensure the safety and reliability of our customer services

We will continue to work tirelessly to ensure the safety of our systems, and the integrity of our services. And to keep you, our customers, informed, updated and supported as we work together to tackle this challenge and ensure there are no serious incidents.

What you should do

We can’t emphasize enough how urgent this issue is, and how vital it is that you take urgent action.

You need to locate and upgrade all instances of log4j to at least version 2.17.0 to mitigate all identified threats.

Given that this vulnerability has been actively exploited for several days now, however, rash and panicked responses probably won’t serve us either.

We advise our clients to prepare for a long process as new versions of the vulnerability emerge, more vendors publish details, and attackers continue to innovate and adapt.

Log4j version 2.16.0 was released on 14 December 2021. This addressed an incomplete fix of the remote code execution vulnerability fixed in version 2.15.0. A third CVE number has been assigned (CVE-2021-45046) to the vulnerability bypass of the 2.15 version under certain non-default configurations. As a result, version 2.15 and older are considered unsafe we strongly recommend upgrading to at least version 2.16.0.

A vulnerability affecting version 1.x has also been reported but is not as bad as the one impacting v2.x as a specific set of conditions are required to exploit it. This bug has been assigned a different CVE number (CVE-2021-4104), but the version is End of Life and should be upgraded if possible to the latest branch.

On December 17, the Apache Log4j team released version 2.17.0 to address a local denial of service vulnerability tracked as CVE-2021-45105. This vulnerability received the fourth CVE designation since the start of December 2021 for this library. The vulnerability is a result of specific configuration that could result in “uncontrolled recursive self-referential lookups” due to a logic flaw in the pattern handling section of this library. This flaw could be used by attackers to render an application unusable or make it unstable. Apache provided mitigates that can be used to resolve this issue through configuration.

To address all these vulnerable versions, the recommended version of Log4j as of 17^th December 2021 is 2.17.0 or greater for Java 8 and version 2.12.2 or greater is still the preferred versions for Java 7. CVE-2021-45105 does not affect version 2.12.2.

Also note that Java 7 is end of life July 2022 (Released in 2011) and Java 8 EOL Dec 2030.

Every environment is different, and businesses will have to conceive their own response based on their unique environments. We would propose the following priorities be considered:

1. Focus on the internet and mitigate. Your first step should be to identify internet-facing attack surfaces that might provide a path to a vulnerable system. A vulnerability scanner can be deployed for this purpose, and most have already been updated with appropriate signatures. However, the vulnerability is nuanced, and scanners will likely only identify the most basic cases. A better approach may be to engage a penetration testing team to manually search for vulnerable instances. You’ll want to equip the team with login credentials to any internet-facing web applications. Ultimately, however, you’ll want to identify any systems that use log4j, manually if necessary. If that’s not clear, identify any systems that use Java in some form and assume they are vulnerable. Patching log4j is the ideal course of action, but there are other work-arounds that reduce the risk if patching is not possible.Careful deployment of Web Application Firewalls (WAF) for example, are not a perfect solution, but can add some value by limiting the majority of know attack vectors against web interfaces.The initial temporary mitigation for the vulnerability involved setting a specific property, formatMsgNoLookups to true, but researchers found ways that bypass this. Relying on this property alone is considered risky and could very well lead to successful exploitation.
2. Limit outbound internet connections. As we mentioned already, a popular and dangerous technique used by attackers is to establish and outbound connection from a compromised system to their command-and-control server. You want to prevent this. Blocking outbound connections won’t prevent systems from getting compromised, but it will complicate the attacker’s ability to verify their exploit, or leverage a compromise in a useful way.
3. Communicate with your vendors. Many of the systems you own or use will be vulnerable. Unless you have a clear view of what software components they use, you will have to depend on them to advise you on the risk and potential mitigations. You’re focus is to inventory your estate and establish a channel to your vendors.
4. Patch log4j everywhere. Once you’ve controlled the risk of attack from the internet, you’ll want to thoroughly inventory every technology you have to identify all the systems that may be vulnerable. As we previously mentioned, an internal vulnerability scan is a good start, but the nature of the problem really requires that you find log4j wherever it may be hiding. Speak to your vendors, and assume that anything running Java needs to be examined closely. Remember that this issue is independent of the version of Java you are running. Upgrading to the latest Java version is recommended, but does not address the log4j vulnerability. Also remember that that log4j is a library, not an application, so we cannot entirely depend on a list of “installed” software to identify it. Fortunately, the library version is reflected in the library’s file name, which makes searching much easier.
5. Deleting the offending class from the Log4j module. Please try to avoid taking this route. This action must only be considered as a last resort and after you have evaluated the impact. The Apache Log4j team provides instructions on how to delete the JndiLookup class. Manually removing the class could break or destabilise the application. This is an extreme solution to the problem, but it must be treated a temporary. You’ll need to keep track of the modified artefacts to ensure you do not reintroduce the vulnerability later.Modifying signed Java artefacts such as JAR, WAR, or EAR files will cause the application to fail at start-up as the integrity of the components were impacted. This behaviour is as designed and will protect against code modification. When dealing with nested Java code artefacts in WAR and EAR files you’ll need to reclusively go through each file to locate the JndiLookup class. There may also be cached copies of the extracted WAR or EAR files on the system. These will need to be removed. Restarting the affected application is, in most cases, the only way to get the changes to reflect as the Java Classloader will cache this file in memory.
6. Detect attacks and limit the impact. Detection is certain to be a constant game of cat and mouse for several weeks to come. Our initial efforts are likely to be naïve, but our capabilities will improve over time. Set yourself up to implement detections on your network, firewalls, webservers, proxies, Web Application Firewalls, and servers. Expect to have to deal with a high level of false positives and to have to adapt and test repeatedly. Note that if the exploitation is successful, the JDNI string is NOT actually logged because it’s interpreted by the log4j logger. Threat Hunting or detection from logs should therefore be aware of these kinds of ‘blank’ spaces. We are publishing an updated list of Indicators of Compromise (IoC) from our intelligence platforms for public consumption here: https://github.com/Orange-Cyberdefense/log4shell_iocs.
7. Defence in depth. The threat of a 0day exploit is not new and we’ve long understood that we need to engineer our controls on the assumption that some system will be compromised. Our response has always been ‘defence in depth’. This means we patch everything we can, deploy endpoint protection, enforce strong authentication and limit user privileges, limit network traffic in, out, and across your network, and search for anomalous behaviours wherever you can. Deception, for example via Thinkst ‘Canaries’, will improve your ability to detect and respond to the worst case. And speaking of the ‘worst case’, consider establishing a relationship with a professional CSIRT that you can access if you detect signs of compromise.

Follow the simple flowchart below when you work through your entire environment. Remember, we are not only concerned about external applications – we also need to find related internal applications as well. Remember how data traverses the network.

Vulnerability decision tree

Initially it was thought that a later version of Java had protection but that was found not to be the case on Tuesday December 14^th.

The terrifying scope and scale of this issue is a ‘first’ for the industry. But like other security threat firsts in our space, our collective efforts will allow us to tackle the threat, manage it, and emerge wiser and stronger on the other end.

Please note there will be updates to this blog.

Incident Response Hotline