How do I control sensitive data and cardinality in agent telemetry?

Scope

Use this profile when instrumenting prompts, model calls, retrieval, memory, tools, workflows, and evaluations. It covers both sensitive content and high-cardinality identifiers that can overload telemetry systems. Security audit records may require different retention and integrity from performance traces.

Why it happens

Agent telemetry frameworks often capture input and output because it makes demos easy to debug. In production, those fields can contain personal data, source code, credentials, customer records, medical or financial information, and adversarial text. Copies spread to exporters, processors, vendors, dashboards, alerts, and support tickets.

High-cardinality fields create a parallel operational failure. Raw prompt, URL, user, document, tool argument, error string, and task ID become metric labels, exploding series counts and cost. Removing content later from the backend is harder than not collecting it.

Diagnosis

Inventory every field at the instrumentation source and follow it through collector, queue, storage, derived metrics, logs, dashboards, exports, backups, and support workflows. Classify data purpose, sensitivity, cardinality, owner, access, retention, and deletion path.

Run seeded-secret tests. Place unique canary values in prompts, retrieved documents, tool results, and errors, then verify whether they appear anywhere unexpected. Test cardinality with production-like task and user variety, not one demo session.

Solution

Define a default metadata profile using bounded operation names, provider/model versions, sizes, counts, durations, outcome enums, policy-decision IDs, opaque task/action IDs, and content-capture mode. Do not put high-cardinality IDs in metric labels; keep them in traces or protected event stores where justified.

Create a separate content-capture profile with explicit purpose, approval, sampling, target scope, redaction, encrypted storage, role-based access, audit, retention, and deletion. Prefer stable references to protected artifacts over copying content into spans.

Apply transformations as close to source as practical, then enforce again at the collector. Monitor redaction failures, dropped attributes, series counts, access, exports, and deletion lag.

Artifact

Complete one profile per environment and capture mode:

Field	Policy
Purpose and owner	Diagnostic question, decision, accountable owner, and expiry
Default metadata	Allowed bounded fields and naming/version standard
Prohibited content	Prompts, output, arguments, documents, secrets, and direct identifiers
Conditional capture	Trigger, sample, approval, scope, and user/tenant constraints
Transformation	Drop, allowlist, redact, tokenize, hash, aggregate, or reference
Cardinality	Label allowlist, limits, overflow behavior, and cost alert
Access/export	Roles, vendors, regions, dashboards, alerts, and download policy
Retention/deletion	TTL, legal hold, cache/backups, user rights, and verification
Testing	Seeded secrets, schema lint, volume load, and periodic access review
Incident mode	Time-bounded override, approver, extra controls, and automatic rollback

Common mistakes

• Capturing all prompts and outputs and promising to redact later.
• Using raw user, task, URL, or error text as metric labels.
• Assuming hashes are anonymous and harmless.
• Applying redaction only in the final backend after data crossed vendors.
• Creating an incident override that never expires.