How to Qualify a New AI Model Before It Takes Production Traffic: Shadow, Compare, Canary, Promote

A new AI model earns production traffic in five steps: confirm it is eligible under the same policy as the model it would replace, shadow it on a sample of live requests with its answers withheld from users, compare it with the incumbent on a replay set it has never seen, give it a small canary share of live traffic, then promote it with a rollback path kept ready. Each step has an exit test, and every transition is written to the record. The method applies to a new frontier version, an open-weight model hosted in-house and a specialized model trained for one job.

Enterprises now run several kinds of model at once, and new ones arrive every quarter. Qualification measures each one on the enterprise's own traffic, under its own policy and at its own cost per successful workload.

Regulatory references verified as of October 31, 2026.

Why a benchmark score is the wrong gate

A public benchmark measures a model on a fixed set of tasks chosen by someone else. Your traffic is a different distribution: your document formats, your tool schemas, your retrieval sources, your users' phrasing.

Price per token misleads in the same way. It measures what a call costs, and says nothing about whether the call produced a usable result. The hybrid AI estate guide sets out why enterprises source by workload. Qualification decides which source serves each workload class.

Supervisors expect the same discipline for traditional models. SR 26-2, the revised interagency guidance on model risk management issued on April 17, 2026 by the Federal Reserve, the OCC and the FDIC, supersedes SR 11-7. It states that generative and agentic AI models "are not within the scope of this guidance," and two of its practices still apply to model qualification: "Ongoing model monitoring involves an evaluation of the extent to which a model is performing as expected given potential changes in products, exposures, activities, clients, data relevance, or market conditions." "Outcomes analysis compares model outputs to corresponding real-world outcomes to assess model performance relative to model objectives and business use." The NIST AI Risk Management Framework asks the same of AI systems: under MEASURE 2.4, functionality and behavior "are monitored when in production," and under MANAGE 2.4, mechanisms exist "to supersede, disengage, or deactivate AI systems" whose performance or outcomes are inconsistent with intended use.

Define success before you compare anything

Qualification compares a candidate with the incumbent on one class of work at a time. A workload class is a category of request with a shared success test: classify a document, extract fields, draft a letter, answer from a knowledge base, write code.

Success is decided by validators applied to the outcome. Use checks a machine can run on every response:

• Schema validation. Structured output parses and matches the schema. Tool-call arguments are valid for the tool.
• Groundedness. Claims in an answer trace to the sources retrieved for it.
• Downstream acceptance. No person overrides the output or edits it beyond a set margin.
• No re-request. The requester does not send a near-duplicate request within a short window, a sign that the first answer missed.

From those validators comes the measure that compares models fairly: cost per successful workload, the expected cost of one outcome that passes the success test. With retries at the same model, it equals the cost per attempt divided by the probability of success. With a fallback to a second model, it equals the cost of the first attempt plus the probability of failure times the expected cost of reaching success through the fallback.

A hypothetical shows why the unit matters. Model A costs 0.5 units per call and Model B costs 1 unit. Model A passes the extraction test 40 percent of the time; Model B passes it 90 percent of the time. Per successful extraction, Model A costs 1.25 units and Model B costs about 1.11. Model B is the cheaper choice per finished extraction.

Start with eligibility, under the same policy

A candidate enters qualification when it could lawfully serve the class today. The test is the policy that governs the incumbent, applied to the candidate's attributes:

• Location. Jurisdiction, region, and whether it runs on-premises, in your cloud account or with an external provider. Data with a residency constraint cannot reach a model outside the permitted region.
• Data handling. Retention, logging and permitted use of inputs and outputs under the provider's terms.
• License. For open-weight models, the license and its use restrictions.
• Scope. The data partitions and entitlement sets the model may serve.

Authorization and placement belong at one control point. The check that decides whether a requester may make a request and the choice of which model does the work read the same context: the requester's authority, the data classification, residency and handling constraints, the budget and the workload class. Made in two places, they drift: the router sends data to a model the requester's policy forbids. Made together, both parts land in one decision record. The third-party risk management guide covers the provider terms that feed this test.

The five steps from candidate to promotion

1. Candidate. The model is eligible for the class under the incumbent's policy. Register it with its location, terms, license, scope and cost model, and set its state for the class to none.
2. Shadow. A sample of live requests of the class is also sent to the candidate. Users receive the incumbent's answer; the candidate's answer is withheld and scored. Choose the sample by a hash of the request identifier against a set percentage, so a given request is consistently in or out. Mirrored requests and candidate responses stay inside the boundary the policy permits, which matters when the candidate is an external provider.
3. Compare. Score candidate and incumbent on validator outcomes, groundedness and cost per successful workload, and run a held-out replay: recorded requests of the class that neither model was tuned on. Require the candidate to outperform the incumbent on that replay before it moves on. Until the class has a measured win rate, an automatic scorer can rank responses on features such as length, detected intent and conversation stage.
4. Canary. A small, bounded share of live requests of the class goes to the candidate, and users see its answers. Guardrails watch validator failure rate and latency, and a breach ends the canary.
5. Promote. The candidate becomes the default for the class and stays under the same monitoring. Promotion is per class: a model promoted for extraction is still a candidate for drafting.

Rollback sits beside all five. A validator failure rate above its limit, a regression in cost per successful workload, a policy event such as a change to the provider's terms, or a latency breach restores the prior source.

Google's SRE workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation," and adds: "When conducting canary evaluation, you must be able to compare the canary signals to the control signals." Model qualification adds one step before the canary: shadow, because a model can return a successful status code with a wrong answer.

Keep the judge out of the request path

A common shortcut is to ask a second large model which answer is better. Researchers who studied the method report "position, verbosity, and self-enhancement biases" in model judges (Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena).

Keep scoring offline. Validators run on every response at low cost. The held-out replay runs on a schedule. If a team wants an evaluation model to score groundedness or hallucination on shadow pairs, it scores recorded pairs, away from the user's request. No judge model sits in the path a user waits on.

A worked example

An illustrative claims department sends four classes of work through one control point: classifying first notices of loss, extracting fields from forms, drafting coverage letters and answering bind questions.

For classification, a cache answers repeats. A specialized model on the insurer's own hardware enters as a candidate, passes shadow and compare, runs a canary, and is promoted as the default, lowering cost per successful workload for the class. Coverage letters can follow the same path to a local model. Novel or high-stakes cases still go to an external frontier model that the policy permits.

For bind questions in this example, the frontier model stays the default. Its success rate makes it cheaper per successful answer despite a higher price per token. In this pattern, every call carries one decision record that shows who was authorized, which sources were eligible, which was chosen and why.

What to have in place before the first candidate

• A registry of every model source with its terms, scope, cost model and qualification state per class.
• A workload classifier that assigns each request a class.
• Validators per class, wired to every response, with results stored beside the cost of the call.
• A replay set per class, recorded from governed traffic and kept out of any tuning.
• Metering that ties cost to outcome, reconciled against provider invoices.
• A rollback that is one change, tested before the canary starts.
• One decision record per request.

Questions to ask about your own model choices

• Which workload classes run on which model today, and who chose?
• What is the success test for each class, and does a machine run it on every response?
• When a provider ships a new version, where does it enter, and what does it have to beat?
• If a promoted model regresses on Tuesday, how long until the prior source is back?

How APERION approaches model qualification

Owned intelligence, governed from the first call. Enterprises will rent some intelligence and own more of it. APERION future-proofs the hybrid AI estate: every agent and every model an enterprise rents or owns, governed from one control point inside its own walls. The control point sits in the path between agents and every model and tool they call, rented or owned, and keeps each action inside the authority a person delegated.

The same control point that checks an agent's authority can also place the work on a model, with both parts written to one record. Smartflow routes across providers with ordered fallback, chooses between a lower-cost and a more capable tier, routes across cache, a local model and a frontier model under client policy, and can run shadow evaluation when it is enabled. Canary, promotion and cost per successful workload are the pattern this guide describes, measured on your own traffic. The unit that matters is cost per successful workload.

The Trust Fabric sets out the four layers underneath: verified identity, access governance, runtime governance, and audit and evidence. The thesis is set out on AI governance.

A verified person, the authority that person delegated, the agent that used it, the policy that applied, the decision and the record of all of it. That chain is the product.

Frequently asked questions

What is shadow testing for an LLM?

Shadow testing sends a sample of live production requests to a candidate model as well as the current one. Users receive the current model's answer. The candidate's answer is withheld and scored against the same validators, so the team measures the candidate on its own traffic with no user exposure. Mirrored requests must stay inside the boundary the data policy permits, which matters when the candidate is an external provider.

How is a canary deployment different from shadow testing for AI models?

In shadow, the candidate answers but users never see its output. In a canary, a small, bounded share of live requests goes to the candidate and users receive its answers. Shadow measures quality with no exposure. A canary measures behavior under live conditions with limited exposure, and guardrails on validator failure rate and latency end it on a breach. Run shadow and comparison first, then the canary.

What should a model promotion gate require?

A promotion gate should require that the candidate is eligible under the incumbent's policy, beats the incumbent on a held-out replay of recorded requests for the class, matches or improves cost per successful workload, and stays within validator and latency guardrails through a canary. Promotion applies to one workload class at a time, and a tested rollback to the prior source should be ready before the gate opens.

What is cost per successful workload?

Cost per successful workload is the expected cost of one outcome that passes the success test for its class. With retries, it equals cost per attempt divided by the probability of success. It replaces price per token as the comparison unit, because a cheaper call that fails often can cost more per finished task than a pricier call that succeeds. Validators such as schema checks and groundedness decide success.

Should an LLM judge decide which model answer is better?

Keep any judge model out of the request path. Studies of model judges report position, verbosity and self-enhancement biases, and a judge on every request adds a second call and a second model to govern. Score responses with validators that run on every call, compare models on a held-out replay offline, and use an automatic scorer on response features until the class has a measured win rate.

When should a promoted model be rolled back?

Roll back when the validator failure rate rises above its limit, cost per successful workload regresses, a policy event occurs, such as a change in the provider's data-handling terms, or latency breaches its target. The rollback restores the prior source for that workload class and is written to the decision record with the evidence that triggered it. Test the rollback before the canary starts.

Sources

• Board of Governors of the Federal Reserve System, SR 26-2: Revised Guidance on Model Risk Management, April 17, 2026, and its attachment, Revised Guidance on Model Risk Management (Section II, footnote 3; Section V, Model Validation and Monitoring, under Ongoing Model Monitoring and Outcomes Analysis). Opened October 31, 2026.
• NIST, AI Risk Management Framework Core (MEASURE 2.4, MANAGE 2.4). Opened October 31, 2026.
• Google, The Site Reliability Workbook, Canarying Releases. Opened October 31, 2026.
• Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXiv:2306.05685, June 2023 (revised December 2023). Opened October 31, 2026.

Related reading

• What is a hybrid AI estate?
• Third-party risk management for AI model providers
• Enterprise AI gateway: definitive guide
• What is an AI control plane?
• The Trust Fabric