Dataset Poisoning: How Training Data Attacks Work and What to Do About Them

Most AI security conversations focus on what happens after a model is deployed: prompt injection, jailbreaks, guardrail bypasses. These are real and important risks. But there is a category of attack that occurs long before deployment, at the point where the model learns from data. If an attacker can influence the data a model trains on, they can influence the model's behaviour in ways that are very difficult to detect once the model is in production. This is dataset poisoning, and it is a supply-chain risk that most AI security programmes underweight.

The reason it gets underweighted is that it feels like a problem for model developers, not for the organisations that deploy models. That distinction has blurred significantly. Many organisations are now fine-tuning pre-trained models on their own data, building RAG systems that continuously ingest new content, or relying on third-party datasets curated by external providers. In each of these cases, the organisation has meaningful exposure to training data manipulation, and they may not have the controls in place to detect it.

How Poisoning Attacks Work

In a classic backdoor poisoning attack, the attacker injects a small number of carefully crafted examples into the training data. These examples associate a specific trigger (a word, phrase, or pattern) with a desired output that would not otherwise occur. During training, the model learns to produce that output when it encounters the trigger. In deployment, the model behaves normally on standard inputs but produces the attacker-specified output when the trigger is present.

The trigger can be designed to be subtle. It might be a specific phrase in a document, a particular formatting pattern, or even a typo that a human reviewer would not notice as meaningful. The poisoned behaviour might be producing incorrect answers on a specific topic, classifying certain inputs as safe when they are not, or generating outputs that subtly favour a particular product, vendor, or point of view. The sophistication of the attack can be calibrated to the attacker's goals and their level of access to the training pipeline.

Fine-Tuning and RAG Increase Exposure

Fine-tuning a pre-trained model on organisation-specific data is common practice for deploying LLMs in enterprise settings. The fine-tuning data often comes from internal sources: customer interactions, internal documents, domain-specific text. If any part of that data pipeline can be influenced by an external party (a vendor whose communications are included in the training set, a third-party dataset used to supplement internal data, or an automated scraping process that pulls from external sources) there is exposure to poisoning.

RAG systems have a different but related exposure. They do not train on external content in the same sense, but they retrieve and process that content at inference time, and the content can carry indirect prompt injection payloads. More relevantly from a poisoning perspective, if the knowledge base that the RAG system retrieves from is continuously updated from sources that an attacker can influence, the attacker can manipulate the information the model presents to users without ever touching the model weights themselves. This is functionally similar to a poisoning attack in its impact, even though the mechanism is different.

Training Data Supply-Chain Hygiene

Addressing dataset poisoning risk requires treating training data with the same supply-chain scepticism that good security programmes apply to software dependencies. This means knowing where training data comes from, maintaining provenance records for each data source, validating that data sources have not been modified between curation and use, and monitoring model behaviour for anomalies that might indicate poisoned learning.

For organisations fine-tuning models, this means having a defined and controlled data pipeline, not an ad-hoc process where data is assembled from wherever is convenient. It means having human review of data samples, particularly from external sources. It means versioning training datasets so that if anomalous behaviour is detected post-deployment, the contributing data can be identified and audited. It also means testing models on anomalous inputs after fine-tuning to check for unexpected output patterns that might indicate poisoned examples in the training set.

• Backdoor poisoning embeds triggers in training data that cause specific outputs in deployment
• Fine-tuning on data from external or third-party sources creates direct poisoning exposure
• RAG knowledge bases that pull from attacker-influenceable sources are functionally similar risks
• Training data provenance -- knowing where data came from and whether it has been modified -- is foundational
• Post-fine-tuning behavioural testing should probe for unexpected output patterns
• Version control for training datasets enables auditability if anomalies are detected

Dataset poisoning sits at the intersection of AI security and supply-chain risk management. Cyberlinx can help your organisation assess and address both dimensions. Contact us at info@cyberlinx.com.au to discuss your AI data pipeline security.