Detecting a Hacked Web Server

TL;DR

This guide shows you how to automatically check if your web server has been compromised. We’ll use log analysis, file integrity monitoring and malware scanning.

1. Set up Log Monitoring

Your web server logs are the first place to look for suspicious activity. We’ll focus on access logs (who visited what) and error logs (what went wrong).

• Choose a log management tool: Options include ELK Stack (Elasticsearch, Logstash, Kibana), Graylog, or Splunk. For simpler setups, you can use tail -f to watch the logs directly on the server.
• Configure logging: Make sure your web server is logging enough information. For Apache, check your httpd.conf file; for Nginx, look at nginx.conf. Pay attention to remote IP addresses and user agents.
• Alerting: Set up alerts for common attack patterns. Examples include:
  • Multiple failed login attempts from the same IP address.
  • Requests for unusual files (e.g., /wp-admin if you don’t have a WordPress site).
  • Error codes like 404s or 500s in large numbers.

Example alert using grep and email (basic):

tail -f /var/log/apache2/access.log | grep "POST /wp-admin" | mail -s "Suspicious Activity on Web Server" [email protected]

2. Implement File Integrity Monitoring (FIM)

Hackers often change core website files. FIM helps detect these changes.

• Choose a tool: AIDE, Tripwire, or Samhain are popular choices. For simpler setups, you can use md5sum.
• Create a baseline: Scan your important website files and directories to create a snapshot of their current state.

  md5sum -r /var/www/html > baseline.txt

• Schedule regular scans: Run the scan daily or weekly, comparing the results to the baseline. Any changes indicate potential tampering.

  md5sum -c baseline.txt

3. Run Malware Scans

Malware can hide in your website files. Regular scans are essential.

• Choose a scanner: ClamAV is a free, open-source option. Commercial scanners like Sucuri or SiteLock offer more features.
• Scan your website: Scan all website files and directories.

  clamscan -r /var/www/html

• Automate scans: Schedule regular scans using cron.

  0 3 * * * clamscan -r /var/www/html > /var/log/malware_scan.log

  (This runs a scan every day at 3am and logs the results.)

4. Check for Backdoors

Hackers often install backdoors to regain access.

• Look for unusual files: Search for files with suspicious names or extensions (e.g., .php.bak, webshell.php).
• Review recently modified files: Use ls -lt to find files that have been changed recently.

  ls -lt /var/www/html

• Check process lists: Look for unexpected processes running on your server using ps aux.

5. Monitor Network Traffic

Unusual network activity can indicate a compromise.

• Use tools like tcpdump or Wireshark: Capture and analyze network traffic for suspicious patterns.
• Look for outbound connections to unknown IPs: Hackers may be using your server to send spam or launch attacks.