How should I trace model latency, streaming, and token usage?

Scope

Use this profile for synchronous or streaming model calls inside an agent. It gives operators phase and usage evidence for one attempt and for aggregates across attempts. The outer workflow should separately record user request, routing, tool work, validation, and final outcome.

Why it happens

One span duration often combines local admission, rate-limit waiting, network setup, provider queueing, generation, client buffering, streaming consumption, and post-processing. A “latency regression” has no actionable location. Streaming makes the ambiguity worse: the user may care about first useful token and completion, while the backend records only total request time.

Usage fields are also overloaded. An API may report input, cached input, output, reasoning, or total tokens at different times. Converting a generic tokens number into cost or quality creates false precision.

Diagnosis

Capture one request at the application boundary and the raw provider event timestamps in a test environment. Compare them with span start/end, first-token metric, final-chunk event, and usage fields. Repeat for success, cancellation, timeout, retry, empty response, and error after partial stream.

Check whether:

• local queue time is attributed to the model provider;
• time to first token includes application admission;
• cancellation is mislabeled as successful completion;
• a retry creates a new attempt span under the same logical request;
• partial output and usage are retained consistently;
• model alias and resolved model/version are distinguishable.

Solution

Create one client span per provider attempt. Parent it under the logical model step and link retries where the tracing model supports it. Record standard GenAI attributes from the pinned convention version and a small set of namespaced extensions for missing phase timestamps.

Emit bounded events: request_sent, first_response, stream_completed, and usage_received. Derive queue duration, time to first response, stream duration, and total attempt duration in the telemetry pipeline. Record outcome as completed, cancelled, timed out, provider error, client error, or partial/unknown.

Keep prompt and completion content off by default. When targeted capture is approved, apply sampling, access, redaction, and retention policy separately from operational timing.

Artifact

The model-span profile includes:

Field group	Required data
Identity	Trace/span, logical request, attempt, provider, requested/resolved model, region
Versioning	Instrumentation library, adapter, and semantic-convention snapshot
Phases	Queue admitted, request sent, first response, final stream item, attempt end
Streaming	Chunk count, bytes if useful, cancellation point, and partial-output flag
Outcome	Status, error type, retry reason, and provider request ID where safe
Usage	Provider-reported categories, units, source event, and completeness
Derived metrics	Queue, first response, streaming, total duration, attempts per outcome
Privacy	Content-capture mode, sampling, redaction, access class, and retention

Common mistakes

• Calling span duration “model inference time” without separating client and provider phases.
• Creating one span or high-cardinality event per token.
• Ending the span at first token and losing completion and usage behavior.
• Overwriting the first attempt when a retry occurs.
• Treating client usage telemetry as settled billing or output quality.