Question-led guide · decision
When should I route to a smaller or larger model?
A task-family routing experiment that compares model policies at equal quality, escalation, refusal, latency, cost, and serious-failure conditions.
Direct answer
Route by task family and verified policy performance, not by prompt length or model price alone. Start with the least costly model that meets the acceptance and serious-failure thresholds for a well-defined slice. Escalate on observable conditions such as unsupported input, low-confidence deterministic checks, policy ambiguity, or failed tool results. Compare the complete route—including retries and escalation—against a larger-model baseline before release.
Scope
This method applies when several models can serve the same workflow and the team can define task families, quality criteria, serious failures, and outcome cost. It does not apply when regulation or policy mandates one approved model, or when the smaller model cannot support required context, tools, modalities, or data controls.
Why it happens
Routers often begin with an attractive rule: use the small model unless the prompt is long or “complex.” Complexity labels then become unreviewed model guesses. Easy-looking inputs can contain high-risk ambiguity; long inputs can require only deterministic extraction. The route also changes the distribution of failures because escalation happens after an initial model has already shaped context or state.
Diagnosis
- Partition work by user outcome, environment, tool set, risk, and expected reasoning—not by token count alone.
- Build a larger-model baseline on the same task portfolio and operating budgets.
- Identify eligibility constraints: modality, context, latency, region, data policy, and tool support.
- Define observable escalation and refusal conditions before running the experiment.
- Record first-call result, escalation reason, full trajectory, final outcome, latency, and fully loaded cost.
- Review slices where the router disagrees with the eventual successful path.
- Set stop conditions for serious failures, coverage gaps, and drift.
Solution
Treat the router as a versioned policy. A minimal policy can use deterministic eligibility and task-family rules before any learned score. Add a classifier only when it improves a measured decision and its uncertainty can trigger a safe fallback.
Evaluate at least three policies: larger-model baseline, smaller-model-only where eligible, and smaller-then-escalate. Compare acceptance, serious failures, abstention, escalation, latency distribution, fully loaded cost per accepted outcome, and coverage. Release by slice; do not average a safe low-risk route together with an unsafe high-risk route.
Artifact
Use this experiment card:
| Field | Required entry |
|---|---|
| Decision | Which task families may begin on the smaller model? |
| Baseline | Model, prompt, tools, policy, date, and portfolio version |
| Eligibility | Hard technical and governance requirements |
| Escalation | Observable reasons, maximum attempts, state carried forward |
| Quality | Acceptance rubric and deterministic checks |
| Safety | Serious failures that block the route |
| Economics | Full path cost per accepted outcome |
| Operations | Drift signals, review cadence, rollback policy |
Common mistakes
- Comparing per-token price instead of full successful trajectories.
- Allowing the first model to cause effects before escalation is decided.
- Using one global confidence threshold across unrelated task families.
- Counting escalated successes as small-model successes.
- Ignoring refusal quality, tail latency, review load, and serious-failure slices.
Evidence
Different task types can justify starting with different model capability levels.
OpenAI's practical agent guide recommends establishing a performance baseline and notes that simpler tasks may be handled by smaller models while harder decisions may require greater capability.
Primary source · official-doc · checked Aug 25, 2026
Limit: The guide is provider-authored and does not establish a universal router or threshold for another product.
Routing policies must be evaluated in the environment and workflow where they will operate.
OpenAI evaluation guidance emphasizes task-specific evaluation criteria, representative data, and continuous evaluation rather than generic impressions.
Primary source · official-doc · checked Aug 25, 2026
Limit: Evaluation guidance does not make offline task sets identical to production behavior.
The correct comparison unit is the complete routing policy, including escalation and failure.
A policy-level comparison prevents a cheap first call from hiding expensive retries, escalations, reviews, or serious failure costs.
Signal Studio author framework · reviewed Aug 25, 2026
Limit: The framework requires local outcome and cost data and does not guarantee savings.
Limitations
This guide does not recommend named models or stable price thresholds. Model behavior, provider features, and prices change. High-risk decisions may require human review regardless of model size, and a route that passes an offline portfolio can still drift in production.
FAQ
- Should prompt length choose the model?
- Prompt length may constrain eligibility or cost, but it does not describe task difficulty, risk, tool behavior, or quality. Use it as one feature, not the decision contract.
- Can the router use another model as a judge?
- It can provide one signal, but calibrate that signal against deterministic checks and domain review, and include the router model's latency, cost, and failure modes.
Related guides
Continue within AI cost engineering, or use one of these adjacent diagnostics:
English editorial review: Codex native-English editorial review, .