Monitor Executable System Calls

TL;DR

Yes, several tools can alert you when an executable uses specific system functions (API calls). This is useful for detecting suspicious behaviour. We’ll cover Sysmon for Windows and auditd for Linux.

Windows: Using Sysmon

Sysmon (System Monitor) is a free tool from Microsoft that logs detailed system activity, including executable API calls. It’s the best option for most Windows users.

Step 1: Download and Install Sysmon

• Download Sysmon from the Microsoft website.
• Extract the downloaded archive.
• Run Sysmon.exe as an administrator.

Step 2: Configure Sysmon

Sysmon uses a configuration file (XML) to define what it logs. The default config is often too noisy, so we’ll create a custom one.

• Download an example configuration from GitHub or create your own.
• Edit the XML file to specify which events you want to monitor. Key areas include:
• ProcessCreation: Logs process creation events, including command line arguments.
• FileCreate: Logs file creation events.
• NetworkConnect: Logs network connections.
• RawAccessRead: Logs raw read access to files (can be noisy).
• To monitor specific API calls, use the EventFiltering section within the configuration file. For example, to log all calls to CreateFileW:

  <EventFiltering>
   <RuleGroup name="API Call Monitoring" groupRelation="or">
    <ApiCall eventId="1" apiName="CreateFileW"/>
   </RuleGroup>
  </EventFiltering>

• Save the configuration file (e.g., sysmonconfig.xml).
• Run Sysmon with your config:

  Sysmon.exe -c sysmonconfig.xml

Step 3: View Sysmon Logs

• Sysmon logs are written to the Windows Event Log (Event Viewer).
• Open Event Viewer (search for ‘Event Viewer’ in the Start Menu).
• Navigate to Applications and Services Logs > Microsoft > Windows > Sysmon > Operational.
• Filter events by event ID to find specific API calls you’re interested in.

Linux: Using auditd

auditd is the Linux auditing system. It can log system calls made by processes.

Step 1: Install and Start auditd

• Install auditd using your distribution’s package manager (e.g., sudo apt install auditd on Debian/Ubuntu, sudo yum install audit on CentOS/RHEL).
• Start the auditd service: sudo systemctl start auditd.
• Enable it to start automatically at boot: sudo systemctl enable auditd.

Step 2: Configure auditd

Configure rules in /etc/audit/rules.d/audit.rules.

• To monitor all calls to the open() system call:

  -a always,exit -F arch=b64 -S open -k opensyscall

  • -a always,exit: Log on exit of the syscall.
  • -F arch=b64: Only log for 64-bit architectures (adjust as needed).
  • -S open: The system call to monitor.
  • -k opensyscall: A key for filtering logs later.
• Reload the rules: sudo auditctl -R /etc/audit/rules.d/audit.rules.

Step 3: View auditd Logs

• Logs are stored in /var/log/audit/audit.log.
• Use the ausearch command to filter logs:

  sudo ausearch -k opensyscall

  This will show all events tagged with the ‘opensyscall’ key.