How should an agent platform separate its control plane from execution?

Configuration publication is a different operation from task execution

The control plane publishes revisions, changes routing, admits new tool definitions, and administers tenants. The execution plane performs work already accepted under a resolved revision. Separating those responsibilities lets a configuration outage freeze new changes without automatically stopping every eligible read-only task.

This separation is semantic before it is physical. Two deployments that both fetch mutable configuration from the same unavailable database on every step remain coupled. Conversely, one modest service can keep its interfaces distinct before separate scaling or failure domains are justified.

A cached release is not cached authority forever

Workers may retain the exact prompt and tool definition they already verified. That answers which behavior to execute. It does not answer whether an actor may still perform a consequential action. Fresh permission checks, revocation requirements, and expiry remain governing constraints.

An outage policy therefore names specific operations. Continuing a calculation over already authorized data is different from issuing a new write after an authorization service becomes unreachable. The latter must follow its established authorization contract. The detailed time boundary belongs in approval and authority expiry.

A configuration outage during twelve investigations

Imagine a fictional service with twelve active investigations when its release registry becomes unavailable. Nine are producing read-only summaries from already retrieved evidence. Two need a new tool revision that has not been resolved. One is about to request permission to publish a change.

The proposed policy lets the nine finish using verified local assets while recording registry unavailability. The two unresolved tasks wait with a visible reason. The pending publication waits because its required authority decision cannot be established. New releases freeze. This is useful partial service, provided the evidence store and user status path remain healthy.

Write a survival matrix that operators can execute

Use one row per failed dependency. Replace the illustrative decisions with the system’s actual permission, durability, and retention contracts.

Dependency lost	Permitted continuity	Stop condition	Recovery evidence
Release registry	Already resolved revisions only	Required asset missing or revoked	All required asset digests resolve
Authorization service	Previously authorized bounded computation	New authorization required or grant expired	Fresh decision path verified
Evidence export backend	Bounded local spool, if policy permits	Spool full or mandatory receipt unavailable	Backlog drained and gaps enumerated
Administration UI	Existing task processing within policy	Intervention cannot be applied safely	Independent stop control exercised
Task system of record	No new durable acceptance	A transition cannot be committed	Authoritative state and ownership reconciled

For every cell, record the deciding component and observable reason code. A dashboard banner saying “degraded” is too broad to tell an operator which action remains safe.

Test the path that restores the control plane

Draw dependencies for starting a replacement worker and restoring an administrator session. Include certificate issuance, image access, configuration decryption, DNS, and database credentials. A running worker may survive with warm connections while every replacement fails to start.

Cloudflare’s November 2023 postmortem shows why hidden dependencies deserve explicit drills. Its incident is not an agent case; the transferable question is whether a tested component outage actually removed all the dependencies of the broader failure domain.

Restore capacity gradually after the registry returns

Recovery can produce a second disturbance when waiting tasks, refreshing workers, and administrators reconnect together. Bound asset refresh and task admission separately. First establish that the intended revision and stop controls are trustworthy, then expand work within downstream headroom.

Google’s cascading-failure discussion supports assessing feedback and dependency pressure. The exact recovery ramp remains a local operating choice. Verify that recovery metrics distinguish refreshed assets from accepted tasks and completed work.

What a diagram cannot establish

Logical planes do not prove physical isolation, safe offline operation, or an acceptable revocation delay. The design needs dependency-loss tests and a named owner for each degraded mode. Continue with packaging a complete agent release and the integrated operating model in AI Agent Platform Engineering.