What should an SRE agent do when telemetry is missing?

Scope

Apply this protocol when an incident agent expects a span, metric series, log event, resource attribute, change record, or topology edge and cannot find it. The objective is to preserve epistemic safety while still making progress. This page is about the agent’s behavior under incomplete observation, not about designing an organization-wide telemetry pipeline.

Why it happens

Missing telemetry has several causes with different implications. A request may be intentionally head-sampled, a collector may drop data under pressure, a library may not instrument an asynchronous boundary, a query may use the wrong resource identity, or retention may have expired. The event may also genuinely not have occurred.

Agents often collapse all of these cases into one of two bad responses: they ignore the gap and finish the story, or they refuse the entire investigation. Both lose useful information. A controlled system separates what was expected, what was observed, why the difference matters, and which safer source can reduce uncertainty.

Diagnosis

Classify the gap before changing the incident conclusion:

• Expected absence: policy or sampling predicts that some records will be missing.
• Collection loss: the signal should exist, but SDK, network, collector, export, or backend health is degraded.
• Instrumentation gap: the code path or external boundary never produced the signal.
• Identity/query mismatch: the record exists under another tenant, namespace, service, version, or time basis.
• Retention/access gap: the record existed but is expired, redacted, or outside current authority.
• Possible true absence: the observation process was healthy enough that non-occurrence is plausible.

Check telemetry health independently of application health. Collector queue metrics, export errors, sampling configuration, deployment history, and known coverage maps are evidence about the observation system itself.

Solution

Require the agent to emit an observation-gap event whenever missing data affects a material hypothesis. The event should reduce the affected claim level automatically. For example, a missing downstream span may allow localized near service A but block dependency B caused the incident.

Provide bounded fallbacks: load-balancer access logs, cloud audit events, deployment records, database health, synthetic probes, or a healthy control cohort. Each fallback needs its own identity and clock reconciliation. Never paste a low-trust source into the context without labeling its origin.

Set stop rules by consequence. If the next step is a reversible read-only query, the agent may continue. If the gap hides the target, actor, or blast radius of a production change, it should hand off before acting.

Artifact

Create one card for each consequential gap:

Field	Example question
Expected signal	Which span, log, metric, change, or topology record should exist?
Expectation basis	Which instrumentation version, policy, or coverage test creates that expectation?
Gap class	Sampling, loss, instrumentation, identity, retention/access, or possible true absence?
Affected claim	Which diagnosis becomes weaker or impossible?
Claim downgrade	What is the strongest statement still supported?
Fallback source	Which independent source can be queried, under what authority?
Reconciliation	How will identity, clock, scope, and semantics be aligned?
Owner and expiry	Who repairs the gap, and when must it be reviewed?

Common mistakes

• Writing “no evidence of X” when the collection path for X was unhealthy.
• Treating a sampled trace as a complete request population.
• Filling a missing field from model memory without marking it as inference.
• Joining fallback records by service display name rather than stable resource identity.
• Continuing to remediation when the missing evidence hides impact, ownership, or reversibility.