Why did token cost fall while cost per resolved task rose?

Scope

Use this guide after a model, prompt, caching, or routing change lowers reported token spend but finance, support, or operations sees no corresponding improvement. It diagnoses the divergence. The full design of a durable unit-cost metric is covered separately in the cost-per-outcome guide.

Why it happens

Token price is a component price. A resolved task is a system outcome. Between them sit context assembly, repeated calls, tool execution, validation, fallbacks, queues, user abandonment, human review, and repair. Optimizing one component can move cost elsewhere.

A smaller model may generate a cheaper first answer but call tools twice, fail validation more often, or escalate cases that were previously automated. Aggressive context reduction can lower input tokens while increasing rework. Faster timeouts can create duplicate attempts. Even a true improvement can be hidden if the new cohort contains harder tasks.

Diagnosis

Choose one release boundary and build comparable before-and-after cohorts. Match or stratify by task type, difficulty proxy, customer tier, language, tool availability, and policy version. For every task, reconstruct the attempt tree rather than keeping only the final call.

Check five bridges:

1. Price bridge: model and provider rates, cache classification, discounts, and currency.
2. Usage bridge: calls, input/output tokens, tools, retries, fallbacks, and concurrency.
3. Quality bridge: validation failures, incorrect effects, reopenings, and regressions.
4. Labor bridge: review, escalation, support, and repair minutes.
5. Outcome bridge: verified resolution, time to resolution, abandonment, and delayed reversal.

If the cohorts differ materially, report the mix shift before attributing the change to the model.

Solution

Assign a stable task ID before the first model call and propagate it through model usage, tools, workflow state, review, billing allocation, and outcome verification. Record every attempt with a reason: primary, validation retry, transport retry, fallback, human-requested revision, or recovery.

Define resolution independently of model completion. A support draft is resolved when accepted under policy; an operational task is resolved when the intended effect is committed and verified; a research task may require citation checks. Add a fixed observation window for reopenings and compensation.

Use a waterfall to explain the change instead of one ratio. A team can then see that model cost fell by $X while tool load, review, or unresolved work rose by $Y.

Artifact

Complete this worksheet for each matched cohort:

Line	Before	After	Attribution question
Attempted tasks			Did volume or task mix change?
Verified resolutions			Is the outcome definition unchanged?
Primary model cost			Did price or token use change?
Retry and fallback cost			Which failure classes created extra calls?
Tool and infrastructure cost			Did work move outside the model API?
Human review and repair			What labor rate and time source were used?
Reversal/reopening cost			What happened after the first success signal?
Total attributable cost			Which shared-cost allocation rule applies?
Cost per attempt			Is this moving differently from outcome cost?
Cost per verified resolution			What confidence interval or range is credible?

Common mistakes

• Dividing provider spend by successful API responses.
• Dropping failed and abandoned tasks from both cost and outcome reporting.
• Comparing releases with different task mix and calling the difference a model effect.
• Counting cached tokens at the wrong rate or ignoring shared-capacity allocation.
• Treating a short-term completion as resolved before validation or the reversal window.