Jailbreak

A jailbreak is an input crafted to make a model disregard the behavioral constraints its provider trained into it — producing content it would normally refuse, or revealing instructions it was told to keep. The target is the model's own alignment, not the application around it.

How it differs from prompt injection

These two get used interchangeably and they attack different things.

A prompt injection targets the application: it inserts instructions the developer did not write into a context the model was told to trust, so the model follows the attacker instead of the developer. A jailbreak targets the model: it persuades the model to abandon its own refusal behavior.

The practical consequence is who can fix it. A prompt injection is an architecture problem the enterprise can address by constraining what the model is permitted to do. A jailbreak is a property of the provider's artifact, and the enterprise cannot patch it — the model updates when the vendor updates it, and a technique that fails today may work after the next release, or the reverse.

Why alignment alone is not a control

Model refusal behavior is probabilistic, opaque and outside your change control. It varies between providers, shifts silently between versions, and produces no record of what it declined or why. An institution relying on it has outsourced a control to a vendor and cannot evidence that the control operated.

That is the argument for putting enforcement in the path rather than trusting the model: guardrails you own, with a version and a record, evaluated on every call. When alignment holds, the guardrail is redundant. When it does not, the guardrail is the only thing that was ever going to fire.

What actually reduces the risk

Three things, in rough order of durability.

Constrain capability rather than content. A jailbroken model that can only return text is a content problem. One with tool access is an action problem. Scoping what the model may reach — see agent authority scope — bounds the damage regardless of whether the jailbreak succeeds.

Inspect the output path. Jailbreaks are judged by what comes back. Output-side inspection catches the result even when the input looked ordinary, which is also where it overlaps with DLP for AI.

Test adversarially and continuously. A model that resisted a technique last quarter may not resist it now. That is the case for AI red teaming on a cadence rather than as a pre-launch gate, with findings fed back into enforceable policy. Jailbreak techniques are catalogued in MITRE ATLAS, which is the closest thing to a shared reference in this area.

Related terms

Prompt injection · AI guardrails · AI red teaming · MITRE ATLAS · Full glossary