How do I trace MCP and tool calls across a gateway?

Scope

Use this guide when an agent invokes a tool through MCP or a similar gateway and operators need to follow the action across process and trust boundaries. It concerns telemetry and correlation. Authorization remains enforced by identity, policy, gateway, and target systems.

Why it happens

Each layer sees a different name. The agent records send_email; MCP has a request ID; the gateway uses an audit ID; the service creates a job; the provider returns a message ID. A trace may stop at the gateway because the server lacks instrumentation or processes work asynchronously.

Teams often compensate by logging complete JSON payloads. This improves short-term search while exposing credentials, personal data, prompts, and untrusted content. It still may not prove which external effect occurred.

Diagnosis

Take one tool invocation and try to navigate from the agent trace to the target effect and back. Verify:

1. stable logical action ID across retries and protocol IDs per attempt;
2. tool name plus server and schema version;
3. gateway policy-decision reference and current actor/target identity;
4. downstream request and idempotency identity;
5. accepted, committed, failed, compensated, or unknown effect state;
6. retry and delegation relationships;
7. content capture and redaction policy at every hop.

If the only join is timestamp plus tool name, concurrent calls will collide.

Solution

Generate an action ID before invocation and preserve it as validated metadata. Create spans at each meaningful processing boundary. Propagate W3C Trace Context only across trusted, compatible channels; validate incoming context and prevent arbitrary baggage from becoming indexed attributes.

Use separate protocol request IDs for attempts. A retry belongs to the same logical action but receives a new attempt identity. Record the policy-decision ID, not an entire authorization token. Record argument schema, safe field names, and a digest of the canonical proposal where useful.

After the tool returns, query or consume an effect receipt when the operation is asynchronous. The final agent message should not be the effect oracle.

Artifact

The tool-action envelope contains:

action: {id: act-..., task_id: ..., attempt: 2, parent_decision: ...}
tool: {name: send_email, server: comms-mcp, schema_version: 7}
trace: {trace_id: ..., client_span: ..., gateway_span: ..., server_span: ..., links: [...]}
authority: {actor_ref: ..., decision_id: ..., policy_version: ..., target_ref: ...}
proposal: {digest: sha256:..., idempotency_key: ..., data_class: confidential}
protocol: {mcp_request_id: ..., operation: tools/call, transport: ...}
result: {protocol_status: ..., tool_status: ..., error_type: ..., receipt_ref: ...}
effect: {state: committed, target_effect_id: ..., verified_at: ..., verifier: ...}
privacy: {capture_mode: metadata, redaction_policy: ..., retention_class: ...}

Common mistakes

• Reusing a protocol request ID as the durable business action ID.
• Treating the gateway’s 200 response as proof of the target effect.
• Logging raw arguments and results by default.
• Accepting user-supplied trace IDs without validation.
• Losing relationships when the server queues work after returning.