Which diagnostic tools should an SRE agent be allowed to call?

Scope

Use this method when an SRE platform team exposes logs, traces, metrics, topology, deployment history, configuration, feature flags, database diagnostics, or cloud APIs to an incident agent. The aim is a useful diagnostic surface with enforceable bounds. Remediation tools belong in a higher, separately reviewed authority tier.

Why it happens

Teams often build an allowlist from tool names: search_logs looks safe, restart_service looks dangerous. Names hide the real boundary. A log query may span every tenant, reveal secrets, or scan petabytes. A database “explain” call may be safe on one engine and execute work on another. A topology endpoint may disclose security-sensitive architecture.

The model also sees tool descriptions as instructions. A compromised server or ambiguous schema can steer it toward broader queries. Permission must therefore be enforced by infrastructure using verified identity, resource scope, parameter validation, and result policy.

Diagnosis

Inventory each tool as an operation, not a label. Record:

1. Target systems and tenants reachable from the tool.
2. Credential type, lifetime, audience, and delegation chain.
3. Parameters that control time, resource, row count, aggregation, and content fields.
4. Worst-case backend cost and concurrency under retries.
5. Direct, indirect, or hidden side effects.
6. Sensitive fields and untrusted content in results.
7. Audit events emitted before and after invocation.

Test failure cases. Send an unbounded time range, unknown tenant, wildcard resource, oversized result, repeated call, and malicious text inside a result. A permission system that relies on the agent declining these inputs is incomplete.

Solution

Create tiers. Tier 0 can inspect public runbooks and service metadata. Tier 1 can run bounded aggregate queries. Tier 2 can retrieve request-level details under incident-scoped approval. Tier 3 covers invasive diagnostics or any operation with meaningful side effects and normally requires a human.

Publish narrow schemas such as compare_error_rate(service_ids, start, end, cohort) instead of generic shell, SQL, or query-language access. Enforce maximum windows, mandatory resource IDs, result caps, redaction, rate limits, and cost budgets at the gateway. Verify authorization again at the target service.

Separate tool discovery from invocation. An agent may know that a privileged diagnostic exists without possessing authority to call it. A human-approved escalation can issue a short-lived, audience-bound grant rather than changing the global prompt.

Artifact

Maintain one row per tool and permission tier:

Field	Required decision
Diagnostic purpose	Which incident question can this tool answer?
Resource scope	Exact tenants, services, regions, environments, and time range
Identity	Workload, delegated actor, token audience, lifetime, and owner
Parameter guardrails	Required IDs, maximum window, filters, result and concurrency caps
Data class	Public, internal, confidential, regulated, or secret-bearing
Result treatment	Redaction, untrusted-content marking, storage, and retention
Cost/load	Backend budget, timeout, retry rule, and circuit breaker
Side effects	None, operational load, stateful diagnostic, or production mutation
Approval tier	Automatic, incident-scoped, or per-call human confirmation
Audit proof	Request, policy decision, parameters, target response, and denial reason

Common mistakes

• Giving the agent raw shell, SQL, or cloud-console access for convenience.
• Assuming a tool is safe because its JSON schema calls it read-only.
• Applying authorization at the model-facing gateway but not the target service.
• Returning entire logs when an aggregate or filtered evidence record answers the question.
• Letting retries multiply an already expensive or overloaded diagnostic query.