Exploitable Bugs per 1000 Lines of Code

TL;DR

Calculating exploitable bugs per thousand lines of code (KLOC) is tricky. There’s no perfect formula, but you can get a reasonable estimate by combining static analysis results with dynamic testing and vulnerability reports. This guide shows how.

1. Understand the Challenges

• Code Complexity: More complex code has more potential bugs.
• Language Matters: Some languages are inherently safer than others (e.g., Rust vs. C).
• Quality of Code: Well-written, reviewed code will have fewer bugs.
• Testing Coverage: More thorough testing finds more bugs.
• Definition of ‘Exploitable’: What counts as exploitable? Remote code execution is different from a minor information leak.

2. Static Analysis

Static analysis tools scan your code without running it, looking for potential vulnerabilities. These tools provide an initial bug count.

• Choose a Tool: Examples include SonarQube, Coverity, and Bandit (for Python).
• Run the Scan: Follow the tool’s documentation to scan your codebase.
• Filter Results: Static analysis often reports false positives. Focus on high-severity issues that are likely to be exploitable.

Example (SonarQube):

# No specific command, this is a web UI based tool. Configure your project and run an analysis.

3. Dynamic Testing

Dynamic testing runs your code with various inputs to find bugs in real-time.

• Fuzzing: Provide random, invalid, or unexpected inputs to see if the application crashes or behaves unexpectedly. Tools include AFL and libFuzzer.
• Penetration Testing: Simulate a real attack to identify vulnerabilities.
• Automated Security Scanners: Use tools like OWASP ZAP or Burp Suite to scan web applications for common vulnerabilities (SQL injection, XSS).

Example (AFL – simplified):

afl-fuzz -i input_dir -o output_dir //program_to_test

4. Vulnerability Reports & Bug Tracking

Review any existing vulnerability reports or bug tracking data for your project.

• Public Databases: Check databases like the National Vulnerability Database (NVD) for known vulnerabilities in your dependencies.
• Internal Bug Tracker: Analyse past bugs to identify patterns and common issues.

5. Calculate Lines of Code (LOC)

Determine the total number of lines of code in your project.

• Use a Tool: Tools like `cloc` are useful for counting LOC across multiple languages.
• Command Example:

cloc .

6. Combine the Data

Now, calculate the exploitable bugs per KLOC.

• Total Exploitable Bugs: Sum the number of confirmed exploitable bugs from static analysis (after filtering), dynamic testing, and vulnerability reports.
• KLOC Calculation: Divide the total exploitable bugs by the LOC in thousands.

Formula:

Exploitable Bugs per KLOC = (Total Exploitable Bugs / Total Lines of Code) * 1000

7. Interpretation and Context

• Acceptable Range: There’s no single ‘good’ number. A lower number is better, but context matters.
• Industry Benchmarks: Research industry benchmarks for similar projects.
• Trend Analysis: Track the bugs per KLOC over time to see if your code quality is improving.

8. Important Considerations

• False Positives/Negatives: Both static and dynamic analysis can produce inaccurate results. Manual review is crucial.
• Severity Levels: Weight bugs based on their severity (critical, high, medium, low).
• Regular Updates: Repeat this process regularly as your codebase evolves.