How do I trace multi-agent and durable workflows?

Scope

Use this design for orchestrators that delegate to agents or workers, wait for external events, fan out, resume from checkpoints, and create artifacts or effects. It complements durable workflow state; telemetry must not become the only execution record.

Why it happens

Distributed tracing begins with a tree: one parent creates children. Durable workflows form a graph. A task is queued and picked up later, one artifact feeds several agents, a retry replaces an attempt, a human approval resumes work in a new process, or two branches join.

Forcing the graph into one tree creates misleading parentage and very long traces. Starting unrelated traces at every boundary loses the story. Stable domain identities and typed relationships bridge the two.

Diagnosis

Choose a workflow with fan-out and a pause. Attempt to answer:

• Which actor delegated each task, with what objective and authority?
• Which attempt is current and which attempts were superseded?
• Which messages and artifacts connected branches?
• Which checkpoint resumed this execution episode?
• Which tool effects belong to the workflow and which were compensated?
• Which spans were sampled out or expired?

If the answer depends on timestamps and service names, the model lacks durable relationships.

Solution

Create durable workflow and task IDs in the execution store. Each execution episode gets a trace; synchronous operations use parent-child spans. Emit links for enqueue/dequeue, delegation, retry, resume, artifact consumption, and join. Record relation type as governed metadata rather than overloading span names.

Delegation records should include delegator, delegate, objective, input artifacts, granted tools/authority, budget, deadline, and expected output. Message records need producer, consumer, schema, sequence or deduplication identity, and content classification. Checkpoints link prior and resumed episodes.

Build a workflow view from execution records plus trace references. It should remain useful when some traces are absent and should link to verified effects rather than infer them from final text.

Artifact

Use these nodes and relations:

Node	Key relations
Workflow	contains tasks, decisions, checkpoints, and effects
Task	delegated_by, assigned_to, depends_on, supersedes
Attempt/trace	executes task, retry_of, resumed_from checkpoint
Message	produced_by, consumed_by, deduplicates message
Artifact	generated_by, version_of, consumed_by, derived_from
Approval	authorizes proposal digest under conditions and expiry
Effect	attempted_by, committed_as, verified_by, compensated_by
Join	waits_for required branches and records missing/failed branches

Every relation carries time, source system, version, and confidence or status where applicable.

Common mistakes

• Keeping one unbounded trace open for an entire multi-day workflow.
• Making every async worker a child as though it ran synchronously.
• Logging delegation text without transferred authority, budget, or artifact IDs.
• Treating retry attempts as separate business tasks.
• Reconstructing effects from agent messages instead of target receipts.