Stop Bots Bypassing Your Forms

TL;DR

Bots are submitting data directly to your web service, skipping your form’s security checks. This guide shows you how to block them using a combination of techniques: rate limiting, CAPTCHAs, referrer checking, and validating expected request formats.

1. Understand the Problem

Bots often bypass forms by directly sending POST requests to your web service endpoint (e.g., an API). This avoids client-side validation like JavaScript checks or form field requirements. They can also avoid CAPTCHAs if they submit data quickly and automatically.

2. Implement Rate Limiting

1. Identify the source: Determine how bots are accessing your service (IP address, API key, user agent).
2. Set limits: Restrict the number of requests allowed from a single source within a specific timeframe. For example, limit to 10 requests per minute per IP address.
3. Implement in your web server or application framework: Most frameworks have built-in rate limiting features.

   # Example using Flask and Flask-Limiter (Python)
   from flask_limiter import Limiter
   app = Flask(__name__)
   limiter = Limiter(app, key=lambda req: req.remote_addr)
   
   @app.route('/your-endpoint')
   @limiter.limit('10 per minute')
   def your_endpoint():
       # Your endpoint logic here
       return "Success!"
   

4. Monitor and adjust: Regularly check logs for blocked requests and fine-tune the rate limits to avoid blocking legitimate users.

3. Add CAPTCHA Verification

1. Choose a CAPTCHA provider: Google reCAPTCHA (v2 or v3) is popular, but other options exist.
2. Integrate into your form: Display the CAPTCHA challenge on your web page.
3. Verify server-side: Crucially, always verify the CAPTCHA response on your server before processing any data.

   # Example using reCAPTCHA v2 (Python)
   import requests
   
   def verify_recaptcha(recaptcha_response):
       url = 'https://www.google.com/recaptcha/api/siteverify'
       payload = {
           'secret': 'YOUR_SECRET_KEY',
           'response': recaptcha_response,
           'remoteip': request.remote_addr
       }
       r = requests.post(url, data=payload)
       data = r.json()
       return data['success']
   

4. Consider invisible reCAPTCHA (v3): This provides a score based on user interaction without requiring explicit challenges, but still requires server-side verification and careful threshold setting.

4. Check the HTTP Referrer Header

The Referer header indicates the page that linked to your web service endpoint. Bots often omit this header or send an invalid one.

1. Expect a valid referrer: Your form should set a specific referrer when submitting data.
2. Validate server-side: Check if the Referer header matches your expected domain.

   # Example (Python)
   if request.headers.get('Referer') != 'https://yourdomain.com/form':
       return "Invalid Referrer", 403
   

3. Be aware of limitations: Users can disable referrer headers in their browsers, so don’t rely on this as the sole security measure.

5. Validate Request Format and Data

Bots may send data that doesn’t conform to your expected format.

1. Define a strict schema: Specify exactly what data you expect (field names, types, allowed values).
2. Validate server-side: Use a validation library or write custom code to ensure the incoming request matches your schema.

   # Example using Marshmallow (Python)
   from marshmallow import Schema, fields, ValidationError
   class YourSchema(Schema):
       name = fields.Str(required=True)
       email = fields.Email(required=True)
   
   try:
       schema = YourSchema()
       data = schema.load(request.form)
   except ValidationError as err:
       return str(err), 400
   

3. Check for unexpected parameters: Bots might include extra, malicious fields in the request.

6. Monitor Logs and Adapt

Regularly review your server logs for suspicious activity (failed CAPTCHAs, blocked requests, invalid referrers). Adjust your security measures as needed to stay ahead of evolving bot techniques.