Signal Studio field guide
Observability for AI Agents
Tracing Models, Tools, Memory, and Multi-Agent Workflows with OpenTelemetry
A vendor-neutral guide to observing agent runs across models, retrieval, tools, memory, handoffs, state, effects, evaluations, cost, privacy, and human review.
For: agent platform engineers, observability engineers, SRE and security teams
What this book helps you do
This book defines an agent run as a graph of model calls, retrieval, memory operations, tool proposals, authorization decisions, effects, handoffs, evaluations, and human review. It shows how to represent that graph with OpenTelemetry-aligned traces and governed content references while controlling privacy, cardinality, sampling, cost, and schema change.
Problems this book helps you solve
- The final answer is logged, but the system cannot explain how it was produced.
- Model calls are traced while retrieval, tool effects, memory, and approvals are invisible.
- Retries create duplicate spans or effects without a stable operation identity.
- Prompt and tool content is copied into telemetry without a privacy or retention decision.
- High-cardinality identifiers make metrics expensive and operationally unusable.
- Trace schemas change without a migration path, breaking dashboards and evaluations.
Decisions you will be able to make
- What constitutes a run, step, parent-child edge, link, event, and effect receipt.
- Which agent operations deserve spans and which details belong in attributes, events, or governed storage.
- How to correlate models, retrieval, memory, tools, handoffs, evaluations, and reviews.
- Which content can be recorded, redacted, hashed, referenced, sampled, or prohibited.
- How to keep metrics low-cardinality while preserving trace-level diagnostic detail.
- How to version and migrate a trace grammar as conventions and workflows evolve.
Who this book is for
- Teams that need to debug, evaluate, secure, or cost a tool-using AI workflow.
- OpenTelemetry practitioners extending an existing evidence plane to agent operations.
- Platform owners defining a vendor-neutral run envelope before selecting a backend.
Who this book is not for
- Teams that intend to store raw prompts and tool results without privacy review.
- Readers expecting an unstable semantic convention to eliminate local schema design.
Reading path
- Observe execution, not only answersDefine the run graph and the evidence needed to reconstruct important decisions and effects.
- Adopt a trace grammarRepresent runs, model operations, retrieval, tools, memory, handoffs, and approvals consistently.
- Instrument models and toolsCapture latency, usage, retries, proposals, authorization, and effect receipts without leaking content.
- Follow state across timeLink memory, workflow state, durable resumes, and multi-agent handoffs across process boundaries.
- Control telemetry riskApply privacy, security, cardinality, sampling, retention, and access boundaries.
- Operate the schemaVersion conventions, test instrumentation, replay runs, and connect telemetry to SLOs and evaluation.
The answer is one output of a distributed workflow
An agent can produce acceptable text after unsafe retrieval, a denied tool attempt, repeated work, stale memory, or human correction. Observability must preserve those differences so operators can diagnose the system rather than infer execution from prose.
Keep content and telemetry separate by design
Most operational questions can be answered with identifiers, types, versions, timings, reason codes, and references. Raw content should require a separate access, retention, and sampling decision.
Use it with
Adopt the public run-envelope schema as a discussion artifact, then remove every field that lacks a concrete operational question and add only locally governed extensions.
Evidence and method
OpenTelemetry specifications and semantic conventions establish telemetry concepts and field meanings, while the book's complete agent run envelope is an author-created integration pattern. Development-status conventions are labeled as such. Examples use synthetic identifiers and governed content references rather than real prompts, customer data, or secrets.
Read a sample
Signal Studio does not reproduce manuscript chapters on this site. Open the Amazon listing to use Read Sample or Kindle Instant Preview
Resources
- Agent run envelope schema (JSON Schema, v1.0.0)
Errata and related guidance
English editorial review: Codex native-English editorial review, .