What belongs in an AI cost release gate?

Scope

Use this gate before releasing a model, prompt, router, context policy, cache, provider, capacity mode, or tool workflow that materially changes unit cost. The gate is cost-centered but cannot be separated from outcome quality and operating risk. A broader evaluation gate may incorporate this record as one input.

Why it happens

Cost review is often a spreadsheet signed before quality testing. Evaluation is a separate report. Reliability, privacy, and capacity live in other systems. A release can therefore look cheaper in one document while increasing retries, escalations, stale cache hits, or sensitive-data exposure elsewhere.

Another failure is approving an average. Average cost hides expensive slices and tail events. A low overall number can coexist with an unbounded tool loop, a premium fallback used for one tenant, or a rare but costly repair path.

Diagnosis

Take the last material AI release and try to reconstruct the approval. Can you identify the exact model and prompt versions, representative tasks, price date, cache behavior, retry limits, quality floors, serious failures, capacity assumptions, and person authorized to accept the trade-off? Can you tell which observed metric would have caused rollback?

If those answers live only in chat or are spread across unversioned dashboards, the organization had a review conversation, not an enforceable gate.

Solution

Define an operating envelope by task slice. For each slice, record expected volume, cost per verified outcome, resolution rate, latency, human intervention, and upper bounds for retries and tool work. Establish non-negotiable quality and policy floors before comparing cost.

Run a production-like canary with budget caps. Reconcile provider usage to task IDs and outcome records. Test fallback paths and one provider or capacity impairment. Set automatic stops for runaway cost, serious failures, and outcome degradation.

Assign an owner who can approve, expire, or revoke an exception. Re-review when provider terms, model versions, task mix, or policy changes—not only on a calendar.

Artifact

The cost-and-evaluation gate record contains:

Section	Required evidence
Release identity	Model, prompt, router, tools, cache, policy, provider, and code versions
Decision and scope	Task slices, traffic percentage, regions, tenants, and duration
Unit economics	Cost per attempt and verified outcome, attribution rules, price date, and range
Quality floors	Outcome metric, serious failures, calibration, and slice thresholds
Operating limits	Latency, concurrency, tokens, retries, tools, fallback, and human review
Data controls	Sensitive fields, retention, cache isolation, and provider processing boundary
Capacity and failure	Quotas, provisioned base, overflow, impairment test, and degraded mode
Canary and rollback	Sample, observation window, stop metrics, prior version, and owner
Exception	Rationale, approver, compensating control, expiry, and removal condition

Common mistakes

• Approving model price without measuring cost per verified outcome.
• Letting improved average quality offset a serious unauthorized or incorrect effect.
• Omitting retries, fallbacks, tool cost, and human repair from the envelope.
• Using current provider price without a checked date and sensitivity range.
• Granting permanent exceptions with no owner or expiry.