How Vulnerability Validation Resolved Bugs Without Patching

By Artum Zolotushko, Software Development Team Leader at Rezilion

As every responsible company does, we too scan our containerized applications for vulnerabilities before deploying them in production. In a recent scan, our security team found 56 high and critical vulnerabilities coming from container base-image and open-source components.

Unfortunately for me, I was tasked with fixing these 56 vulnerabilities. I was given 48 hours to do that as a large customer was waiting for this release. Before we could ever ship to production, these vulnerabilities would need to be addressed, but at what cost? I knew that the time and effort it would typically take to address these vulnerabilities would cause a significant delay in our release schedule. Furthermore, a quick look at the vulnerability list produced by the scanner our security team used clearly showed that many of these weren’t really applicable in the specific context of my application.

Usually, it would be impossible to know which of these vulnerabilities would even present a real risk, let alone the hours it would take to repair just one. I knew I’d need to find a faster, easier way if we were to meet our deadlines. Luckily, I didn’t have to look too far. I was able to validate the vulnerability scan results using Rezilion’s platform. This allowed me to narrow my focus from identified vulnerabilities to exploitable vulnerabilities, greatly reducing patching efforts without compromising the product’s security.

What is vulnerability validation?

Before taking a look at what I did, we should first address the question of, “What is Vulnerability Validation?” Simply put, it’s a technical analysis that finds out if a specific vulnerability in a piece of code is exploitable in the specific context in which this code is deployed. For example, when the code or packages are deployed inside a container, most are not going to be used – some of this is code bloat, some could be in the operating system. The vulnerable code can be deployed on a container but never loaded to memory – and therefore technically not exploitable. Another example is a remote code execution vulnerability that requires network access to the vulnerable code – but that code is being executed by a process that doesn’t communicate in any manner. 

Vulnerability Validation is always deterministic; it provides a clear-cut yes or no answer to the question, “Is this vulnerability exploitable?” Developers perform their own version of vulnerability validation daily whenever they get a new pile of vulnerabilities dumped on them. 

For my part, I first ran Rezilion’s platform determine which vulnerabilities were not loaded to memory. I saw that 55 out of 62 high/critical vulnerabilities were not loaded and, therefore, not exploitable. However, when trying to fix the remaining seven vulnerabilities, I experienced some difficult challenges. When looking to see if there were official patches available, I found that some had nothing available. Others required massive changes to my architecture. So I decided to see if I could invalidate some of them. Here’s what I found:

• Two out of these seven vulnerabilities were in a file that indeed loaded to memory, but the specific vulnerable function wasn’t called. So two gone, five left! 
• Two vulnerabilities were indeed loaded to memory but are only exploitable on hosts with an interactive user and not on containers. Fortunately, our code was running inside a container.  That’s two more gone.
• Three other vulnerabilities were only exploitable on ARMv7 processors, and we were running on an Intel processor. Three more gone. No more vulnerabilities left.

By running our platform, I was automatically able to invalidate 49/56 high and critical vulnerabilities and reduce the number of vulnerabilities I had to patch by 89%. As an added benefit, through this process, I discovered new techniques to invalidate six more vulnerabilities, leading to those techniques being added to Rezilion’s platform so that everyone will be able to execute them automatically. Most importantly, I saved many hours of manual security work – without compromising on security. Our sales team was able to move forward in their process with the customer prospect, proving that it’s NEVER a bad idea to dogfood your work. 🙂