A Holiday Story, Internet Edition: The Impact Of Assessing And Addressing Log4j Installations Proactively

A look at our log4j data.

On December 10th, a critical, once in a generation security flaw was discovered in log4j. With a deep heritage in understanding and assessing vulnerabilities, Tenable began our efforts right away to create many different ways to audit for this flaw, knowing it was and remains a race against the clock before many of these methods are discovered and exploited by bad actors. The resulting static and dynamic checks that Tenable created assess for vulnerability via well documented vectors as well as some more arcane ones that you may not have heard of – with even more to come. For instance, while most of the public discussion has been about exploiting log4j via the HTTP(s) protocol (either through headers or by sending malformed input), we started to implement many more protocols. Here are some of our initial findings in terms of prevalence (this is still in flux with more protocols coming):

“Raw”, above, basically means connect to any open port, send the ${jdni://} payload and watch what happens. And, yes, believe it or not, we did find a handful of pop3 and imap servers that somehow send their logs to vulnerable log4j-enabled servers. What this graph shows is that the surface area for potential exploitation of Log4j goes well beyond web protocols and it’s critical that customers assess the widest range of TCP/IP-based protocols possible.

One novel and effective detection method we use to dynamically discover vulnerable hosts consists of making the vulnerable ones perform a DNS resolution against a Tenable-operated DNS server (that process is fully documented here), effectively reporting back that asset is at-risk. This is a very reliable method which anonymizes the source of the data. It was designed in such a way that Tenable would not hold a “master list” of vulnerable log4j installations while still enabling customers to leverage this highly effective method to determine vulnerable systems.

We received hundreds of thousands of queries per second, and that volume remains steady as of this writing. These queries are made regardless of whether the remote servers are vulnerable or not (they are made by the scanners as part of the probe). That volume is expected – we have thousands of customers regularly scanning their environments.

Servers which are vulnerable to log4j do a different DNS query. Among other things, we can use this data as a proxy to estimate the overall volume of vulnerable servers over time:
 

As can be seen on this graph, the number of vulnerable servers increased steadily as our customers rushed to detect them, and then decreased as more and more systems were patched. It’s worth noting that scanning activity did not substantially decrease during the same period:

Three quarters of the servers we detected with our dynamic check have been fixed right in time for our customers to go home on 12/24 and spend the long weekend with their families - a huge win for already tired security teams. That being said, this is a critical vulnerability and we’ve been behind our customers the entire way to help them detect and remediate this flaw as fast as possible. 

Over the next few weeks and months, security researchers and attackers will continue to discover new attack vectors for log4j. There is still a lot to be done and this problem is by no means fixed. Those who were early to audit this problem and remediate it got to spend the end of the year with their families and not doing incident response or reinstalling their environment. More importantly, their proactive approach and processes will help them stay on top of log4j as it continues to evolve.

In the meantime, let’s celebrate how quickly many in our industry are reacting to this flaw and keeping the infrastructure safe.

Learn more:

• One in 10 Assets Assessed Are Vulnerable to Log4Shell
• Assess Log4Shell Like an Attacker With Tenable’s Dynamic Detections
• Frequently Asked Questions About Log4Shell and Associated Vulnerabilities