WAF Event Loss: Is Caching to Blame?

TL;DR

Yes, caching can cause WAF logged events to drop. This happens because caches sit in front of your web application and WAF, potentially intercepting requests before they reach the WAF for inspection. Understanding where you’re caching (browser, CDN, reverse proxy) and how it interacts with your WAF is crucial.

Understanding the Problem

Your Web Application Firewall (WAF) needs to see every request to properly analyse it for threats. Caching stores copies of responses to reduce server load and improve performance. If a request hits a cache, the cached response is served instead of going to your application and WAF. This means the WAF never sees that request.

Steps to Diagnose & Fix Event Loss

1. Identify Your Caching Layers: Where are you caching content? Common places include:
   • Browser Cache: Controlled by HTTP headers.
   • CDN (Content Delivery Network): Services like Cloudflare, Akamai, Fastly.
   • Reverse Proxy: Like Nginx or Apache acting as a cache in front of your application server.
2. Check Browser Caching:
   • Use your browser’s developer tools (usually F12) to see if responses are coming from the cache.
   • Look at the Cache-Control and Expires headers in the HTTP response.
   • Force a refresh bypassing the cache (Ctrl+Shift+R or Cmd+Shift+R). This should send the request directly to your server, triggering WAF logging.
3. Investigate CDN Caching:
   • Log into your CDN provider’s dashboard.
   • Check their caching rules and settings. Many CDNs allow you to bypass the cache for specific URLs or request headers (e.g., those containing authentication tokens).
   • Purge the CDN cache if necessary, but be aware this can temporarily increase load on your origin server.
   • Configure the CDN to forward all requests to the WAF, even cached ones. This is often done through a ‘Cache-Bypass’ or similar feature.
4. Examine Reverse Proxy Caching:
   • If you use Nginx as a reverse proxy, check your configuration file (usually nginx.conf).
   • Look for the proxy_cache_path directive and related settings.
   • Ensure that requests are not being cached unnecessarily. You might need to adjust caching rules based on URL patterns or request headers.
   • Example Nginx Configuration snippet:

     
     proxy_cache_path /var/cache levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m use_temp_path=off;
     server {
         ...
         proxy_cache my_cache;
         proxy_cache_bypass $http_pragma $http_authorization;
         ... 
     }

   • The proxy_cache_bypass directive is important. It tells Nginx to bypass the cache if certain headers are present (like authentication).
5. WAF Configuration:
   • Some WAFs have settings related to caching. Check your WAF documentation.
   • Ensure that the WAF is configured to inspect all traffic, even if it appears to be cached.
   • Consider using a ‘challenge’ mechanism for requests that might be cached (e.g., CAPTCHA or JavaScript challenge).
6. Test Thoroughly:
   • After making changes, test your application thoroughly to ensure that WAF logging is working as expected.
   • Simulate different scenarios (e.g., first request, subsequent requests, cache expiration).
   • Monitor your WAF logs for any gaps or inconsistencies.

Important Considerations

• Cache Invalidation: Regularly invalidate caches when you deploy new code or security rules.
• Dynamic Content: Avoid caching dynamic content that requires WAF inspection (e.g., pages with user-specific data).
• Header Forwarding: Ensure that all relevant request headers are forwarded to the WAF, even from cached responses.