I Don't Trust an Idle Agent Until I Can Restore It Without Lying

Google published Agent Substrate on GKE. The headline is density: more sandboxes per node, freeze an idle agent in a few hundred milliseconds, wake it on a warm worker.

I stared at the numbers and wrote a rude note in the margin: so what?

Density is a capacity number. It tells me the pool is packed. It does not tell me the agent that comes back is the same agent that left. That is the question this post is for.

Why this post?

I have been building agent-shaped systems for a while now — NewsInsight had an AI layer, KEPLER had agents that search the live web. In both cases the “agent” was a process I could point at. Kill the process, the session is gone.

Substrate inverts that. The process is disposable. The session is supposed to survive.

If that inversion is real, I am no longer operating a fleet of containers. I am operating a checkpoint protocol for an open system that writes to the outside world. I wanted to write that down in language I would actually use at a whiteboard.

What issue does suspend/resume resolve?

Agents spend most of their life waiting. Waiting on a model. Waiting on a tool. Waiting on a human. If I reserve a full Pod of CPU and RAM for every quiet session, the bill is silly and the cluster fills up with ghosts.

So Substrate does the obvious-in-hindsight thing:

1. Keep a pool of warm workers.
2. When the agent is idle, snapshot it and give the worker back.
3. When the next turn arrives, restore the snapshot onto some free worker — maybe a different machine.

Kubernetes still owns machines. A small data plane owns the actor. The Pod lifecycle stays off the hot path, because scheduling a Pod for every tool call would add seconds to a turn that should take milliseconds.

That resolves idle compute. It creates a new problem: the worker is no longer the agent.

Why must the session outlive the worker?

Because the node is now a slot, not an identity.

If I keep thinking “the agent is whatever is running on gke-axion-12a,” I will do three dumb things:

• pin the session to a machine that can die
• leave credentials inside the guest, where the model can steal them
• lose work every time restore lands somewhere else

Substrate’s own architecture is explicit that actor identity is independent of the hardware. Pause keeps the snapshot on the node so the next wake is fast. Suspend uploads to object storage so the next wake can happen anywhere.

Locality is an optimization. Identity is not.

So the durable object is the session: who it is, what it remembers, which files it has, which credentials the gateway will honor, and which tool calls might already have changed the world.

What is an epoch, and why one instead of four?

A RAM dump is not a checkpoint of an agent.

An agent is an open system. It has guest memory, a workspace on disk, credentials injected outside the sandbox, and tool calls whose effects live in someone else’s database. If I freeze those four things at four different times, restore will mix yesterday’s brain with today’s disk. That is a split brain, just with better branding.

I wanted one name for “everything that must come back together.” I am calling it an epoch. Either the epoch commits, or restore is refused.

Four planes, said simply:

1. Guest state. Registers, file descriptors, the conversation in RAM. This is what gVisor checkpoint actually freezes. From the workload’s point of view there is no polite shutdown. The next syscall just blocks until resume.
2. Filesystem generation. The writable overlay, a durable directory, or a Filestore volume. Memory and disk must agree on a generation number. Restoring RAM from epoch 7 onto disk generation 8 is a silent lie.
3. Delegated credentials. Substrate’s gateway injects secrets outside the guest so the agent cannot steal them. Good. That also means the checkpoint cannot treat “whatever was in the environment” as the credential set. Resume with a rotated mint should be a rejection, not a successful wake.
4. Outstanding tool operations. Tickets, pushes, browser jobs. Their effects are not in the memory image in any useful sense. If I don’t name them, restore will guess.

The envelope around that freeze is a CheckpointManifest: where the bytes live, what they contain, which sandbox binary is pinned. I don’t need the full type system to use the idea. I need the invariant: partial freeze is not a freeze.

The window I kept circling

I asked a second rude question: what if the tool already succeeded?

Walk it slowly.

1. The agent calls create_ticket.
2. JIRA (or GitHub, or a payment API) applies the write. The HTTP 200 is in flight, or sitting in a socket, or already in userspace.
3. Suspend fires. Substrate is not graceful. There is no drain. In-flight TCP is reset. The checkpoint captures whatever the guest believed before the syscall completed.
4. Resume restores that belief. The model retries, because from its point of view the call never returned.

If the tool is not idempotent, I now have two tickets. The sandbox came back in 400ms. The world is wrong.

At-most-once is an acceptable description of a sandbox. It is an unacceptable description of an agent with write access to production.

The only honest mitigation I can see is to treat every tool call as a distributed write:

• mint a stable idempotency key before the call, from actor + epoch + tool + logical op
• persist a receipt: applied, failed, or unknown
• on restore, reconcile receipts before the model is allowed to speak

Applied means return the stored result. Do not replay. Unknown means probe. Missing key means refuse to resume, because I cannot tell “never sent” from “sent and lost the 200.”

That receipt is the only reason I kept a Java type called ToolCallReceipt. Not because I needed more records. Because I needed a place to hang the key.

Is CPU idle the same as safe to freeze?

No. And this is the part density dashboards will hide.

A session can look quiet from the sampler and still be live:

• Streaming. Tokens are on the wire. Freeze orphans the client. I cannot splice into a stream it already gave up on.
• Lease. A POSIX lock or Filestore grant is held by the actor, not the worker. Suspend without transferring it stalls every collaborator waiting on the volume.
• Heartbeat. Peers still expect a beat. Freeze without a “I am frozen” notice looks like a crash. They will spawn a duplicate.
• Async tool. A headless browser or webhook waiter is outstanding. The callback is aimed at a worker IP that will not host this actor after restore.

Safe suspension is a predicate, not a timeout. If that predicate is false, I do not have an idle agent. I have a live distributed transaction that happens not to be burning CPU.

If packing is a vanity metric, what would I actually measure?

Sandboxes per node answers “how dense is the pool?” It does not answer “how often does a wake redo a side effect?”

The five numbers I would put next to packing:

If 12% of wakes replay a write, I do not have a 10× win. I have a correctness incident with excellent bin-packing.

Why Java records?

I think in Java. Records are a good way to freeze an invariant so I cannot accidentally widen it.

The types are small because the invariant is small: one epoch, restorable, or not at all.

public record AgentEpoch(
        String actorId,
        long epochNumber,
        GuestState guest,
        FilesystemGeneration filesystem,
        DelegatedCredentialSet credentials,
        List<ToolCallReceipt> outstandingTools
) {
    public boolean generationsAligned() {
        return guest.generation() == filesystem.generation();
    }
}

public record CheckpointManifest(
        String snapshotUri,
        AgentEpoch epoch,
        Placement placement,   // local node vs object storage
        Scope scope            // full memory image vs data only
) {}

public record ToolCallReceipt(
        String idempotencyKey,
        String toolName,
        EffectStatus effectStatus,  // UNKNOWN, APPLIED, FAILED
        String resultDigest
) {}

public record WorkspaceLease(
        String actorId,
        String holderActorId,    // the session, never the worker
        boolean streamingLive,
        boolean freezeAdvertised,
        Set<String> asyncToolIds
) {}

RestorePolicy.evaluate is then a boring predicate. Refuse when generations skew, when the gateway mint does not match, when a stream or heartbeat is still live, when an async tool has no receipt, or when a tool call has no idempotency key. Applied receipts are returned, never replayed.

That is the whole design. I am not claiming Substrate implements this envelope today. I am claiming this is the minimum I would trust before I called sub-second resume “done.”

What I am still unsure about

I still don’t know how large a full guest snapshot is in practice once the workspace is on Filestore instead of the overlay. If the hot path is decompress plus receipt probe, resume p99 might be a storage story pretending to be a scheduler story.

I also don’t know how a harness should advertise “frozen” to peers without inventing a second control plane. That feels like the next post, not this one.

Until the epoch covers guest, filesystem, credentials, and tool receipts, a fast restore is just a fast way to be wrong.

References

More reading on the pieces I only sketched:

• Agent Substrate available on GKE — the announcement that started this.
• Agent Substrate architecture — workers as slots, actors as records, why Kubernetes stays off the hot path.
• Snapshot (Agent Substrate) — local pause vs external suspend, full vs data scope, manifest.json.
• Suspend and pause an actor — no drain, at-most-once for in-flight work, 502s for requests already inside the sandbox.
• gVisor checkpoint/restore — what guest freeze actually captures.
• Application checkpointing — the older name for “save this process, run it later, maybe elsewhere.”
• Idempotence and Stripe’s idempotent requests — why a stable key has to exist before the side effect.
• Java records (JEP 395) — the language feature behind AgentEpoch / CheckpointManifest / ToolCallReceipt / WorkspaceLease.
• Kubernetes Pods — the lifecycle Substrate is trying not to put on every tool call.
• Google Cloud Storage — where a suspend snapshot goes when the node should be fully free.
• Split-brain — the failure mode of restoring RAM and disk from different generations.

• ← Previous Log