Data Poisoning

Data poisoning is the deliberate corruption of the data a model learns from, so that the resulting model behaves the way an attacker intends. It can target the original training corpus, a fine-tuning set, a feedback loop, or the documents a retrieval system searches — and unlike an attack on a running system, it leaves no trace in the request path.

Four places it enters

Pre-training data is largely scraped, so anyone who can publish can contribute to it. This is the least controllable and the least targeted.

Fine-tuning data is the most dangerous for an enterprise, because it is small, specific and usually assembled internally from sources nobody audits — support tickets, historic decisions, an export from a system somebody had write access to.

Feedback loops that learn from usage can be steered by whoever generates enough of the usage.

Retrieval corpora are the underrated one. A poisoned document in a knowledge base is not training-time poisoning at all — it is indirect prompt injection with a longer shelf life, and it affects behavior without touching any model weights.

How it differs from model poisoning, and from drift

Model poisoning tampers with the artifact — the weights, a checkpoint, a serialized file in a registry. Data poisoning tampers with the input to training, so the artifact is legitimately produced and its provenance checks out.

Model drift looks similar from the outside — behavior changes without a deployment — but drift has no adversary. Distinguishing them matters: drift is corrected by retraining, and poisoning is not, because retraining on the same poisoned source reproduces the result.

What a runtime control can and cannot do

Be clear about the limit. A gateway sits between an application and a model at inference time. It cannot inspect a training corpus and it cannot detect a backdoor in weights it did not produce. Any vendor claiming otherwise is describing something else.

What a runtime control does contribute is threefold. It bounds the consequence: a poisoned model with narrow tool access and inspected output does less than one with broad authority — see agent authority scope. It records behavior over time, so a shift that testing missed shows up in production evidence. And it governs which models are callable at all, so an unvetted model from an unverified source is not reachable from a regulated workload — see model allowlists.

The controls that actually address poisoning sit upstream: provenance for training data, signing and verification for model artifacts, and an inventory of what went into what — see AI bill of materials. Techniques and mitigations are catalogued in MITRE ATLAS, and data and model poisoning is a named entry in the OWASP Top 10 for LLM Applications.

Related terms

Tool poisoning · Model drift · AI bill of materials · MITRE ATLAS · Full glossary