Question-led guide · decision
What should an LLM application cache?
A decision framework for prompt prefixes, retrieval results, tool reads, embeddings, responses, and derived artifacts under staleness and privacy constraints.
Direct answer
Cache an object only when reuse is likely, identity is stable, freshness can be bounded, authorization is preserved, and a wrong reuse is recoverable. Good candidates often include immutable prompt prefixes, versioned embeddings, and idempotent public reads. Treat retrieved evidence, tool results, model responses, user memory, and anything tied to permissions or external effects as higher-risk objects with explicit invalidation and provenance.
Scope
This guide covers application-layer decisions for prompt prefixes, retrieval results, embeddings, tool reads, model responses, and derived artifacts. It includes provider-managed prompt caching as one mechanism but does not assume the application controls the provider’s implementation.
Why it happens
“Add a cache” sounds like one optimization. An LLM application actually contains objects with different identities and failure modes. A system instruction can be immutable for a release. A product price may change. A retrieved policy depends on tenant and effective date. A tool result may carry the permissions of the caller. A response can include stochastic interpretation and user data.
Hit rate encourages unsafe simplification. Teams use a normalized prompt as a key, one TTL for everything, or a shared namespace across tenants. The cache then saves calls while serving stale evidence, bypassing current authorization, or hiding that the model and policy version changed.
Diagnosis
List each candidate object and run five tests:
- Identity: can the exact semantic object be named and versioned?
- Equivalence: what must be equal before reuse is safe—text, tenant, policy, model, tools, locale, permissions, or task state?
- Freshness: what event makes the value wrong, and how quickly must invalidation propagate?
- Isolation: can another user infer existence, timing, or content from the cache?
- Consequence: what happens if the wrong value is reused, and can the effect be reversed?
Measure baseline repetition and object size. Do not build a high-risk cache for an object that rarely repeats.
Solution
Separate cache namespaces by object class and security boundary. Use explicit versions in keys for prompts, models, policies, schemas, and embedding models. For retrieved evidence, retain source identity and valid_at; revalidate authorization and freshness before inserting the evidence into a new task.
Prefer caching pure or idempotent reads over anything that represents an action. Never cache an approval or treat an earlier authorized tool result as permission for a later call. If a response can cause an effect, verify current state and idempotency at execution time.
Monitor hits by correctness-relevant slice. A high hit rate with rising stale-answer repairs is a regression. Keep a bypass path and make invalidation testable.
Artifact
Create a row for each object type:
| Field | Decision |
|---|---|
| Object | Prompt prefix, embedding, retrieval, tool read, response, state, or artifact |
| Reuse value | Frequency, saved latency, tokens, compute, and downstream work |
| Cache key | Tenant, identity, version, model, policy, locale, permissions, and task fields |
| Freshness | TTL plus the authoritative invalidation event |
| Provenance | Source, creation time, versions, and transformation chain |
| Isolation | Namespace, encryption, access checks, and side-channel review |
| Wrong-hit consequence | Harmless, repairable, sensitive, or irreversible |
| Validation | Check performed before hit is consumed |
| Metrics | Hit, miss, stale hit, invalidation lag, bypass, and outcome delta |
Common mistakes
- Using prompt similarity as proof of task equivalence.
- Omitting tenant, policy, model, or permission versions from the key.
- Treating TTL as the only invalidation mechanism.
- Caching free-form responses that embed user-specific or time-sensitive facts.
- Reporting savings without measuring stale hits, repairs, and downstream outcomes.
Evidence
Provider prompt caching can depend on exact prefix reuse and exposes cache-related usage fields, making prompt layout part of cache behavior.
OpenAI's prompt-caching guide documents automatic exact-prefix caching behavior, placement guidance, and cached-token telemetry.
Primary source · official-doc · checked Aug 26, 2026
Limit: Provider behavior, eligibility, retention, privacy terms, and prices are mutable and differ across services.
Prompt caching can create observable privacy and security behavior that deserves explicit threat analysis.
Auditing Prompt Caching studies timing behavior and potential side channels in model APIs across its evaluated setting.
Primary source · paper · checked Aug 26, 2026
Limit: The study is point-in-time and provider-specific; its results do not establish that every cache implementation leaks information.
Cache policy should be defined per object type, including key, tenant, version, expiry, invalidation, provenance, and failure consequence.
The cache-object matrix below prevents a single TTL and namespace from governing semantically different artifacts.
Signal Studio author framework · reviewed Aug 26, 2026
Limit: Correct values depend on local data contracts, provider terms, threat model, and workload measurements.
Limitations
The matrix does not replace provider data-processing terms or privacy review. Cache correctness depends on key construction, isolation, clocks, invalidation delivery, model and policy versions, and the consequences of serving stale or cross-scope content.
FAQ
- Should I cache complete model responses?
- Only for tightly bounded, versioned, non-personal tasks where semantic equivalence, policy, freshness, and authorization are checked. A similar prompt does not necessarily imply the same safe answer.
- Is a long TTL always more cost-efficient?
- No. A longer TTL increases hit opportunity but can increase stale or unauthorized reuse and invalidation cost. Optimize expected outcome cost, not hit rate alone.
Related guides
Continue within AI cost engineering, or use one of these adjacent diagnostics:
Editorial QA: automated native-English, structure, source-presence, and link checks completed . This record is not an independent expert endorsement. Review boundary.