Blog | G5 Cyber Security

Detecting Keyloggers with Heuristic Analysis

G5 Cyber Security

TL;DR

Heuristic analysis can detect keyloggers, but it’s not foolproof. It looks for suspicious behaviour rather than specific known malware signatures. This guide explains how to set up basic heuristic rules and tools to identify potential keylogging activity on your system.

What is Heuristic Analysis?

Traditional antivirus relies on signatures – unique patterns in known viruses. Heuristics look for things that viruses do, like modifying system files or intercepting keyboard input. If a program acts suspiciously, it’s flagged even if it hasn’t been seen before.

Step-by-step Guide

  1. Understand Keylogger Behaviour: Keyloggers typically need to:
    • Intercept keyboard events (what you type).
    • Log those events.
    • Send the logs somewhere (a file, email, network).

    Heuristic rules will focus on detecting these actions.

  2. Monitor System Calls: Keyloggers often use specific system calls to access keyboard input and files.
    • Windows: Use tools like Process Monitor (ProcMon) from Sysinternals Suite. Filter for events related to ReadKeyboard, GetKeyState, file creation/modification in common log locations (e.g., %TEMP%, user’s Documents folder).
    • Linux: Use strace to monitor system calls made by a process. 
      sudo strace -p <process_id> -f -o keylogger_trace.log

      Look for calls like open(), read(), write() on suspicious files or network connections.

  3. Check for Hooking: Keyloggers frequently use keyboard hooks to intercept input.
    • Windows: Tools like API Monitor can help identify processes installing keyboard hooks. Look for calls to SetWindowsHookEx() with the WH_KEYBOARD_LL flag.
    • Linux: Hooking is less common on Linux, but you can monitor shared library loading using ldd or lsof -p <process_id> | grep libhook (if a hooking library is used).
  4. Scan for Suspicious File Activity:
    • Look for new files appearing in unusual locations, especially with names like keylog.txt, log.dat or random character strings.
    • Check file modification dates – a keylogger might be constantly updating its log file.
    • Use tools like ClamAV (cross-platform) to scan for known malware signatures as an additional layer of protection. 
      clamscan -r /path/to/scan
  5. Network Monitoring: If the keylogger sends logs over the network:
    • Use Wireshark to capture network traffic. Filter for suspicious connections (e.g., unencrypted HTTP requests, connections to unknown IP addresses).
    • Look for data being sent in plain text – a poorly written keylogger might not encrypt its logs.
  6. Process Behaviour Analysis:
    • Examine the processes running on your system using Task Manager (Windows) or top/htop (Linux). Look for processes with unusual names, high CPU usage, or no associated window.
    • Use a sandbox environment to run suspicious files and observe their behaviour in isolation. Tools like Cuckoo Sandbox automate this process.
  7. Implement YARA Rules: YARA is a tool for creating descriptions of malware families based on textual or binary patterns.
    • Create rules to detect common keylogger characteristics (e.g., specific strings used in the code, file headers). Example: 
      rule Keylogger_Simple
{
 meta:
description = "Detects a simple keylogger"
 strings:
 $string1 = "GetKeyState" nocase
 $string2 = "WriteFile" nocase
 condition:
 all of ($string1, $string2)
}
    • Scan files or processes with your YARA rules. 
      yara -r /path/to/scan keylogger_rules.yar
  8. Regularly Review Logs: The effectiveness of heuristic analysis depends on consistent monitoring and review of logs generated by the tools mentioned above.

Important Considerations

Exit mobile version