Blog | G5 Cyber Security

Malicious Machine Learning Models

G5 Cyber Security

TL;DR

Yes, machine learning models can contain malicious code or be used to deliver harmful payloads. This isn’t usually about viruses *inside* the model file itself, but more about how a model is trained, what data it uses, and how it’s deployed. Attackers can sneak in backdoors, manipulate predictions, or use models as part of larger attacks.

How Machine Learning Models Can Be Malicious

  1. Data Poisoning: This is one of the most common ways to make a model malicious. Attackers inject bad data into the training set.
    • What happens? The model learns from this corrupted data and starts making incorrect or biased predictions, potentially leading to harmful outcomes.
    • Example: Imagine a spam filter trained with emails deliberately labelled as ‘not spam’ that contain phishing links. The filter will then let those emails through.
  2. Backdoor Attacks: Attackers embed hidden triggers into the model.
    • What happens? The model behaves normally most of the time, but when it encounters a specific input (the trigger), it produces a pre-defined malicious output.
    • Example: A facial recognition system might misidentify someone as a known criminal when they wear a particular pair of glasses.
  3. Model Stealing & Repurposing: While not directly ‘malicious code’, stealing a model allows attackers to understand its vulnerabilities.
    • What happens? Attackers can then craft inputs specifically designed to exploit those weaknesses. They might also repackage the stolen model with malicious components in a new application.
    • Example: Stealing a credit scoring model and using it to create a fraudulent loan application system.
  4. Supply Chain Attacks: Compromised pre-trained models or libraries.
    • What happens? Attackers inject malicious code into popular machine learning components that many developers use. This can affect a large number of applications.
    • Example: A compromised TensorFlow package containing hidden malware.
  5. Exploiting Model Serving Infrastructure: The software used to *run* the model is vulnerable.
    • What happens? Attackers target vulnerabilities in the model serving framework (e.g., TensorFlow Serving, TorchServe) to gain control of the server or inject malicious code.
    • Example: A remote code execution vulnerability in a model deployment tool.

How to Protect Against Malicious Machine Learning Models

  1. Data Validation & Sanitisation: Carefully check your training data for errors, inconsistencies, and malicious inputs.
    • Technique: Use anomaly detection algorithms to identify unusual data points.
  2. Regular Model Auditing: Regularly inspect your models for unexpected behaviour or hidden triggers.
    • Technique: Test the model with a wide range of inputs, including adversarial examples (inputs designed to fool the model).
  3. Secure Model Serving: Use secure and up-to-date model serving frameworks.
    • Example: Ensure your TensorFlow Serving instance is patched against known vulnerabilities. 
      docker pull tensorflow/serving:latest
  4. Input Validation at Runtime: Validate all inputs to the model before processing them.
    • Technique: Implement strict input filtering and sanitisation rules.
  5. Monitor Model Performance: Track key metrics like accuracy, precision, and recall for unexpected changes.
    • Example: Set up alerts if the model’s prediction rate drops significantly.
  6. Use Trusted Sources: Only use pre-trained models from reputable sources.
    • Technique: Verify the integrity of downloaded models using checksums and digital signatures.
Exit mobile version