I’m sure we’re all familiar with the idea made popular by Malcolm Gladwell’s book Outliers that it takes 10,000 hours to truly master something. Based on the paper “The Role of Deliberate Practice in the Acquisition of Expert Performance“, the research data indicates that people who are experts in a field got to that level by practicing up to 4 hours a day, 5 days a week for 10 years. What would you do if you could free up 10,000 hours? Are there tasks at work you would like to clear off your plate so you can spend more time on what you enjoy? During a recent bake off against a legacy application security testing platform, we found a way to give one organization back a huge chunk of time.
What is a legacy AppSec tool?
At Qwiet AI, we often use the term “legacy” when referring to the previous generation of application security testing tools. There are a few identifiers that are shared across legacy application security testing tools, but for now let’s focus on the most important one: scanning methodology. Legacy tools utilize a “localized” scanning method that scans code in separate blocks without taking into account the data flow throughout the application. This method often suffers from very low accuracy, especially when it comes to reachability due to lack of insight into the data flow in the application.
Qwiet AI uses a patented scanning method based on creating a Code Property Graph (CPG) of the target application. By combining an Abstract Syntax Tree, Control Flow Graph, and a Data Flow Graph, scans become more holistic, taking the entirety of the application into account, including the path data travels through the code. This provides a much more accurate detection of vulnerabilities and their reachability. (For more information on the CPG, please see an overview in this blog post or read the research paper behind the concept)
The Bake Off
The customer, a Fortune 100 tech company took a sample of 10 applications across their organization to provide a wide variety of implementations, size, and languages to use for testing. Qwiet AI worked with the customer to get the product deployed, but no other efforts were made to tune the detection policies. The 10 applications totaled 4,968,150 lines of code.
Speed
The first metric to compare was scan times. Scanning all 10 applications with Qwiet AI took a total of 28 minutes, with the legacy competitor taking 383 minutes (not including 2 apps that the legacy vendor’s report didn’t include scan times). That’s a average of 4 minutes per app for Qwiet AI and 48 minutes per app for the competitor. The longest scan times in the test were 10 minutes for Qwiet AI and 108 minutes for the legacy competitor.
Findings
The next step was to measure the total findings. For the 10 applications, Qwiet AI found a total of 522 vulnerabilities whereas the legacy competitor found 2,928. Of course, the initial reaction to seeing Qwiet AI return around 1 ⁄ 5 the results found by the legacy solution was a bit concerning. Given the accuracy of our CPG based scanning method, Qwiet AI was confident in our findings and the customer then went through the results to determine the false positive rate and the results were quite surprising.
False Positives
As the customer went through the results, it soon became apparent that the legacy solution had an extremely high number of false positives. For 10 applications totaling 4.9 million lines of code, Qwiet AI returned 28 false positives while the legacy competitor returned 2,339. That’s a false positive rate of 79.9% for the legacy solution, with 8% for Qwiet AI.
While some false positives are to be expected when dealing with code, a false positive rate close to 80% goes well beyond tolerable levels. In the AppSec space, each finding needs to be researched and remediated and false positives can dramatically increase the time an organization spends on their remediation efforts.
The customer informed us that it takes an average of 4 hours to research and fix each findings from their AppSec platform. With this sample set of 10 applications, results from the legacy vendor (including false positives) would take approximately 11,712 hours to research and remediate. Qwiet AI’s findings would take approximately 2,088 hours to remediate, saving the customer 9,624 hours of developer time. So almost 10,000 hours.
Which brings us back to the initial question, what would you do with 10,000 hours? Qwiet AI is helping organizations save time (and money) by quickly returning highly accurate results. Can we save you 10,000 hours? Give our preZero platform a spin yourself or reach out for a demo.
For the full breakdown of the bake off, feel free to download the Legacy Bake Off Case Study.
