What’s Next for Log4j: Tales from the Trenches

The recently discovered flaw in Apache’s popular open source logging library for Java, Log4j, could wreak havoc for years to come. Analysts are predicting it could take as long as five years to finish patching related security flaws because of the widespread adoption of the logging library and the complexity involved in maintaining third-party software libraries.

In a recent Rezilion panel webinar, security experts discussed what organizations need to consider when it comes to using open source software and mitigating their exposure to Log4j, and what organizations can expect to see from the fallout.

Where Things Stand Now With Log4j

There has been a mad scramble across the industry to detect, identify, and patch as fast as you can, or even take systems offline if they can’t be patched — given the magnitude of systems that needed to be patched, said panelist Julius Musseau, cofounder and CTO of MergeBase.

By now, probably 99.99% of systems have been patched and the vulnerability has been fixed, he said. But the one in 10,000 or 100,000 vulnerabilities that are still unpatched are still vulnerable. Some of those vulnerable systems are going to continue to be exploited, which people will discover in the months ahead, Musseau said.

Based on past breaches, Musseau anticipates it will be six months before further implications of Log4j make more news.

Josh Bressers, vice president of security at Anchore, agreed but said he’s hopeful since there haven’t been any new incidents for a couple of months. He cited the Biden Administration’s Executive Order, which has laid the groundwork for improving cybersecurity across the board. “There’s the old saying about ‘Never let a good catastrophe go to waste.’”

Log4j has put security front and center in a way that you don’t always see, Bressers said. “And so, I’m very hopeful that in a year or two, we’re going to look back and say, ‘Thank goodness we made all these changes.’”

Log4j Blindspots in Scanners Continue

Research Rezilion conducted using multiple open source and commercial scanning tools, which were assessed against benchmark packaged Java files, revealed that there were a lot of blindspots when it comes to detecting Log4j.

Bressers said he wasn’t particularly surprised by that because there are “a lot of odd ways to construct your Java archives and your dependencies, and you can mix and match.”

For him, the survey findings have helped push the whole open source community forward because now, the next time a vulnerability is discovered, people will know to look in a database.

Musseau said the survey findings are significant because they reveal the different ways Log4j could be hidden and camouflaged on your systems and how effective various scanners are at uncovering different variations of Log4j in the wild.

“What Rezilion did was very helpful for all the scanners to improve awareness about all these variations,’’ Musseau said.

What Security Teams Should Be Doing Now About Log4j

Musseau recommended that organizations run more than one scanner for extra assurance. In terms of maturing their processes, he advised asking suppliers to generate a Software Bill of Materials (SBOM), “where your software is literally reporting back to you what’s inside it. That’s really something I think the industry as a whole needs to do to mature.”

Bressers echoed that, saying an SBOM is critical because Log4j “really made supply chain security a front of line problem.” While it may already be, “sometimes it’s easy to ignore something until you can’t, and I don’t think we can anymore.”

Organizations have reached a point where not knowing what’s in their environment and in their software is not an option anymore, he said.

SBOMs are just starting to take hold now, “and I’m very excited to see where they go in the future because we’re just starting out and there’s already some amazing tooling and fantastic ideas I’m seeing in the space,” Bressers said.

Because blindspots still exist, Yotam Perkal, director of vulnerability research at Rezilion, said the vulnerability has raised the issue of the importance of supporting the open source community. He recommended that organizations keep monitoring and make sure that Log4j doesn’t crawl back.

Open source security needs improvement, said Bressers, who was recently elected to serve on the OpenSSF Technical Advisory Council. He added that the council is always looking for help. It behooves people to participate because so much is built on open source software and everyone benefits, he noted.

How To Prepare for the Next Log4j

An important metric for organizations to track is the mean time to patch, Musseau said. That means when a vulnerability is announced, how quickly can you turn around and get a patch applied to the affected systems. He suggested organizations run exercises around that.

Whatever comes next, Bressers said he’s learned to be ready for it.

“Whatever we think we’re going to be ready for, the universe is going to throw a curveball and whatever we’re ready to do isn’t what’s going to happen.” So security teams must have a certain amount of flexibility and be able to quickly change direction on the fly, he said.

Perkal said to bear in mind that software tools are not perfect. He said security teams should be skeptical and “take every result you get with a grain of salt.”