Why can retries make an incident worse?

Scope

Use this guide for service, queue, tool, model, or workflow calls that may be retried after timeout, throttling, transport failure, or transient server error. It focuses on incident amplification, not on selecting one timeout value for every dependency.

Why it happens

A retry is locally rational: another attempt may succeed. At system scale, each caller makes the same decision while the dependency is already slow. If three layers each make three attempts, one user operation can expand into many downstream calls. Capped backoff can even synchronize clients at the cap without sufficient jitter.

Unknown outcomes create a second risk. The server may have committed before the client timed out. A repeated payment, message, deployment, or tool action can duplicate the effect. Even idempotent reads can add enough load to delay recovery.

Diagnosis

Trace one user operation through every layer and annotate timeout, retry count, backoff, queue, deadline, idempotency behavior, and circuit breaker. Compute the maximum attempt tree and compare it with dependency capacity during impairment.

Run a controlled failure exercise:

• add latency without returning errors;
• fail after commit but before response;
• return throttling with and without server retry guidance;
• keep a queued request alive past the user’s deadline;
• recover the dependency while stale retries remain in flight.

Observe total attempts, concurrent load, duplicate-effect prevention, useful successes after deadline, and recovery time.

Solution

Choose one layer to own the retry policy for each call path. Propagate an absolute deadline and remaining attempt budget. A downstream client cannot spend time the user no longer has. Use bounded exponential backoff with jitter for eligible transient failures and honor explicit server guidance.

Require idempotency identity for side-effecting operations and reconcile unknown outcomes before another attempt. Shed or queue work by priority when the dependency is impaired. Couple retry metrics to the original task ID so the team sees amplification per outcome.

Stop retrying when the failure is permanent, authority or preconditions changed, the deadline expired, the budget is spent, or added load threatens recovery.

Artifact

For each exercise, complete this record:

Field	Entry
User deadline and outcome
Call tree and retry-owning layer
Failure injected	Latency, throttle, transport, post-commit loss, or dependency outage
Attempt budget	Per task and per dependency
Backoff/jitter	Algorithm, cap, and server guidance
Idempotency	Key scope, lifetime, result replay, and conflict behavior
Unknown-outcome reconciliation	Query and authoritative source
Load result	Calls, concurrency, queue depth, and recovery time
Effect result	Duplicates prevented, residual effects, and late successes
Stop/rollback decision	Threshold, owner, and configuration change

Common mistakes

• Enabling default retries independently in every library and gateway.
• Retrying a timeout as if it proves the server did nothing.
• Using backoff without a total deadline, attempt limit, or jitter.
• Counting successful retry calls while ignoring user abandonment and recovery delay.
• Testing individual clients without measuring the shared dependency under a retry storm.