Black Box Testing: Beyond Fuzzing

TL;DR

Fuzzing is great for finding crashes, but it’s not the only black box testing method. Techniques like boundary value analysis, equivalence partitioning, decision table testing, and state transition testing can uncover different types of bugs without needing to know how the software works internally.

Black Box Testing Alternatives

Black box testing means you’re testing a system *without* looking at its code. You treat it like a ‘black box’ – you give it inputs and see what outputs you get, checking if they match expectations. Fuzzing is one approach, but here are others:

1. Boundary Value Analysis (BVA)
• Focuses on testing the edges of input ranges. Bugs often occur at these limits.
• Example: If an age field accepts values 18-65, test 17, 18, 19, 64, 65, and 66.
2. Equivalence Partitioning
• Divides input data into groups (partitions) that should behave similarly. You only need to test one value from each partition.
• Example: For a country field, partitions might be ‘UK’, ‘USA’, ‘Other’. Test one valid UK entry, one valid USA entry, and one invalid entry.
3. Decision Table Testing
• Useful when the behaviour of the system depends on multiple conditions.
• Create a table listing all possible combinations of inputs (conditions) and their expected outputs (actions).
• Example: A discount calculation based on customer type (new/existing) AND order value (over/under £50). The decision table maps these to the correct discount.
4. State Transition Testing
• Tests how a system moves between different states in response to inputs.
• Example: A vending machine has states like ‘Idle’, ‘Selecting Product’, ‘Dispensing Product’. Test transitions between these states with valid and invalid coins/selections.
5. Use Case Testing
• Tests the system from a user’s perspective, following typical workflows.
• Example: ‘User logs in’, ‘User adds item to basket’, ‘User completes checkout’.
6. Error Guessing
• Based on experience and intuition, try inputs that are likely to cause errors.
• Example: Entering very long strings, special characters, or negative numbers where they aren’t expected.
7. Forced Entry Testing
• Attempting to use functions/features in ways not intended by the developers. This can reveal hidden functionality or vulnerabilities.
• Example: Trying to access admin features without proper authentication, submitting invalid data types to APIs.

Tools & Techniques

While these methods don’t require code access, tools can help:

• Web Application Scanners: Tools like OWASP ZAP or Burp Suite can automate some boundary value and error guessing tests.
• API Testing Tools: Postman allows you to construct complex API requests for testing various input combinations.

Example – Boundary Value Analysis with Python

Let’s say we need to test a function that calculates shipping cost based on weight (0-10kg). We can use a simple script:

def calculate_shipping(weight):
  if 0 <= weight <= 10:
    return weight * 2.5
  else:
    return -1 # Error case

# Test cases for BVA
test_weights = [-1, 0, 1, 5, 9, 10, 11]
for weight in test_weights:
  cost = calculate_shipping(weight)
  print(f"Weight: {weight}, Cost: {cost}")

This script tests values at the boundaries and slightly beyond to check for errors.