Linux Crash Triage from Fuzzing

TL;DR

Fuzzing finds crashes! This guide shows how to quickly figure out what they mean on Linux, so you can focus on the important ones. We’ll cover collecting info, symbolising crashes, and prioritisation.

1. Setting up Crash Collection

Before you start fuzzing, make sure your system is set up to catch crashes. This usually involves a crash handler like apport or systemd-coredump.

• Apport (Ubuntu/Debian): Usually installed by default. Check with

  sudo apt install apport

  . Configure in /etc/default/apport to save crash reports.

• Systemd-coredump: More modern, often preferred. Enable it with

  sudo systemctl enable systemd-coredump.service

  and configure in /etc/systemd/coredump.conf (location of core dumps is important!).

Ensure you have enough disk space for the core dumps! Crashes can be large.

2. Reproducing the Crash

The first step is always to try and reproduce the crash reliably. This makes debugging much easier.

• Save the Input: The fuzzer should have saved the input that caused the crash. Keep this safe!
• Run Again: Try running your program with the same input again. If it crashes consistently, great! If not, you might need to increase fuzzing time or adjust parameters.

3. Gathering Crash Information

Once reproducible, collect as much information as possible.

• Core Dump: This is the most important file. It’s a snapshot of your program’s memory at the time of the crash. Location depends on your crash handler (e.g., /var/crash for Apport, /var/lib/systemd/coredump for systemd-coredump).
• Backtrace: Use a debugger like gdb to get a backtrace from the core dump. This shows the function call stack at the crash point. Example:

  gdb /path/to/your/program /path/to/core_dump

  Inside gdb, use the command

  bt full

  to get a detailed backtrace with local variables.

• Program Logs: Check your program’s logs for any clues leading up to the crash.

4. Symbolising the Crash

Backtraces show addresses, not function names. Symbolisation converts these addresses into meaningful function calls.

• Debug Symbols: You *must* have debug symbols for your program and any libraries it uses. Build with the -g flag (e.g., gcc -g myprogram.c -o myprogram).
• Symbol Server/Paths: Tell gdb where to find the symbols.
  • If symbols are in the same directory as your program, gdb will usually find them automatically.
  • Otherwise, use

    set solib-search-path /path/to/symbols

    inside gdb.

• System Symbol Maps: Systemd-coredump can often symbolise crashes automatically if debug symbols are installed on your system (e.g., using apt-get install -dbg). Use the command

  systemd-analyze core /path/to/core_dump

  .

5. Prioritising Crashes

Not all crashes are equal. Here’s how to prioritise:

• Severity: Did the crash cause a denial of service, data corruption, or security vulnerability?
• Reproducibility: How easily can you reproduce it? Consistent crashes are higher priority.
• Crash Location: Crashes in critical code paths (e.g., parsing functions, security-sensitive routines) are more important. The backtrace helps identify this.
• Impacted Functionality: What part of your program is affected?

6. Tools to Help

• GDB (GNU Debugger): Essential for analysing core dumps and getting backtraces.
• Systemd-analyze: Good for quickly symbolising systemd-coredump crashes.
• Crashpad: A cross-platform crash reporting library.