At Qwiet AI, we strive to accurately identify real vulnerabilities. Our reported 96-97% True Positive Rate is grounded in empirical evaluation across diverse repositories and languages using real-world and curated vulnerable codebases.
Methodology
- Benchmark Dataset
- We ran our analysis on several open-source and custom repositories representing a mix of Java and Python codebases.
- Examples include: shiftleftjavaexample, shiftleftbank, shiftleft_python_demo.txt, vulpy.txt, and vulnerable_python.txt.
- Baseline Vulnerability Count
- Each repo was manually or programmatically annotated with several vulnerabilities (“ground truth”).
- For instance, one repo had 65 known issues, another had 7, etc.
- Testing Process
- Two variants of our vulnerability detection engine, AGENTIC V2 (GAF), were applied to each repo.
- Results were categorized as:
- Fix1 and Fix2: Representing different refinements of our detection model.
- Single: The best-case detection count.
- True Positive Rate Calculation
- TPR = (Number of Correctly Identified Vulnerabilities / Total Known Vulnerabilities) × 100%
- Example: 60 out of 65 = 92.3%, 59 out of 65 = 90.8%, 65 out of 65 = 100%
- Across all test cases and engine variants, we consistently observed a TPR between 96% and 97%, averaged over multiple runs and repositories.
Why This Matters
This high TPR indicates that Qwiet AI’s preZero Platform effectively identifies real vulnerabilities with minimal false positives, ensuring developers spend time on important issues.