Google AX: Suspend/Resume, Restart Costs and Self-Hosting

Back

Kevin Riedl

14 min read · 20 Sep 2026
Last reviewed October 15, 2026

Next

An agent spends $30. Its process disappears. You pay again for work it already did. That is a recovery problem. Paying for agents that spend most of their time waiting is an idle-capacity problem. Google AX is relevant to both, but the same headline does not solve both.

AX, also called Agent Executor, is Google’s Apache-2.0-licensed orchestrator for sandboxed agent tasks. It runs above Agent Substrate and uses Kubernetes-like YAML and commands. The useful idea is to manage agent execution as infrastructure, not to invent another reasoning framework. Read the pinned AX project overview and development warning.

Reviewed on 15 October 2026: AX 22954f64bb29 and Agent Substrate 6edd1fccfc9d. This is a source and architecture review, not a deployment test or throughput benchmark. The $30 story is an illustrative scenario, not an AX result we measured.

What does Google AX orchestrate now?

The current API has three core resources: Task, Workspace and Model. Older four-resource explanations include a Gateway that is no longer part of this reviewed AX schema. A resource defines configuration; it does not establish every operational guarantee around it.

A task may represent a whole agent job or one delegated operation. Multiple workspaces can be bound under spec.workspaces; the first becomes the command’s working directory. AX’s current resource and lifecycle concepts.

What changed since the September AX examples?

Do not deploy an old four-resource manifest unchanged. This refresh replaces the previous Gateway example and updates the execution contract. Pin the CLI, control plane and runner together before migrating configuration.

The schema is a better configuration reference than a lingering “four kinds” sentence in the manifests guide. It also no longer contains the old reserved Task-policy-field explanation from our September review. Inspect the current AX API schema.

Is AX really Kubernetes for AI agents?

It is a useful analogy, not a Kubernetes replacement. Your harness manages reasoning and tool decisions. AX manages task declarations and lifecycle. Agent Substrate supplies sandbox execution, while Kubernetes manages the underlying worker infrastructure.

The current AX server reconciles directly with Substrate under distributed locks and stores resource state in Redis. It does not create a Kubernetes custom resource for every short-lived task. Its design also explicitly describes tasks as immutable. Read the updated AX control-plane design.

The README’s billions-of-tasks language is a design target. It does not demonstrate billions of simultaneously active agents in your cluster. Size for active concurrency, state size, model latency and bursts. For model loops, tool handling and feedback, use our agent harness engineering guide.

What survives AX suspend and resume?

For the documented AX runner, workspace files survive; the command returns in a fresh process. Stop and suspend signal PID 1 with SIGTERM. The runner forwards shutdown to the command so it can flush state. This orderly path is not the same event as a machine disappearing mid-call. The pinned runner contract describes the persistence and shutdown boundary.

Make retries idempotent: give an operation a durable identity so repeating the request cannot silently repeat its effect. Also record an explicit success or failure result. Ready, WorkspaceReady and a healthy runner are not substitutes for an accepted deliverable.

There is an important layer distinction. Agent Substrate describes full-state snapshots that preserve RAM and filesystem state, and demonstrates approximately 250 actors multiplexed across eight pods. That does not override AX’s specific runner contract. Test the exact runner and backend pair you will deploy. Substrate’s snapshot capabilities, demonstration and deployment documentation.

Does sub-second resume mean zero cold start and zero idle cost?

No. Fast activation is a latency claim, not an end-to-end billing guarantee. Substrate’s README reports resumes below 500 ms and more than 500 suspend/resume activations per second. These are project-reported capabilities, not a Wavect benchmark or a latency promise for every AX workload.

A first launch, unavailable worker, large snapshot, dependency installation or slow model request can still delay useful work. Measure both resume-to-ready and resume-to-first-useful-result, including p95 latency, rather than treating infrastructure activation as the whole user experience.

Releasing CPU and RAM from one suspended actor can improve fleet utilization. It does not make worker pools, Redis, snapshot storage, networking or operations free. An already admitted model request may still be billable while the local agent waits. Infrastructure idle cost and token cost are different lines in the ledger.

Also distinguish explicit lifecycle commands from automation. AX documents ax suspend and ax resume; automatic idle detection and suspension still appear on its roadmap, alongside token/timeout budgets and approval policies. Do not assume an idle policy is active just because a command exists. Check which AX capabilities remain roadmap items.

How much could restart recovery actually save?

Measure the work repeated after an interruption, not the entire original job. A useful checkpoint can reduce replay. It cannot refund inference already consumed, and a suspend feature alone does not prove crash recovery.

Take an illustrative month with 1,000 jobs. Suppose 10% suffer exactly one interruption. If each affected job repeats $30 of model and tool work, replay costs $3,000. If tested recovery reduces repeated work to $3, replay falls to $300. That is $2,700 avoided replay, not $2,700 guaranteed savings from installing AX.

If the additional runtime, snapshot and operating costs total $600, the scenario leaves $2,100 before any other changed costs. These are illustrative USD assumptions, not provider prices or customer results. The improvement must be established under abrupt failure, not inferred from a successful pause.

For the complete method, including human review and accepted outcomes, use our AI agent cost-per-action guide. Use the same ledger to decide whether recovery improvements reduce the total cost of an accepted outcome.

How can you test AX file recovery without spending model tokens?

Start with a disposable workspace and a harmless sleeping process. This example follows the current manifest fields. It contains no model call, repository credential or goal-based bootstrap. It still consumes infrastructure resources. Compare the example with the current manifest reference.

Prerequisites are an installed, compatible AX/Substrate stack and CLI with authorized cluster access. Save this as task.yaml. debug: true is deliberate for this isolated diagnostic task because SSH exposes execution and file access. Do not use it as a public or production default.

apiVersion: ax.io/v1alpha1
kind: Workspace
metadata:
  name: recovery-lab
spec:
  files:
    - path: README.txt
      content: "Disposable recovery test. No model calls."
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata:
  name: recovery-lab-001
spec:
  workspaces:
    - name: recovery-lab
      path: /workspace/recovery-lab
  command: ["python3", "-c", "import time; time.sleep(3600)"]
  debug: true

Apply it and wait until Ready is true. Stop the streaming watch with Ctrl+C; this stops observation, not the task.

ax apply -f task.yaml
ax watch task recovery-lab-001

Write the marker after startup, then suspend and resume. The task command does not recreate this file, so reading it later checks persistence rather than repeated initialization.

ax ssh recovery-lab-001 -- sh -c 'printf "%s\n" "checkpoint-before-suspend" > /workspace/recovery-lab/recovery-marker.txt'
ax suspend task recovery-lab-001
ax resume task recovery-lab-001
ax watch task recovery-lab-001

Wait for Ready again, stop the watch with Ctrl+C and read the marker:

ax ssh recovery-lab-001 -- cat /workspace/recovery-lab/recovery-marker.txt

The expected output is checkpoint-before-suspend. Passing shows that this file survived an orderly suspend/resume cycle. It does not prove RAM restoration, abrupt-crash durability or safe retries of external actions. We have not run this cluster test here.

Copy any evidence you need, then delete only the disposable resources created above. These commands remove the test task and its workspace definition:

ax delete task recovery-lab-001
ax delete workspace recovery-lab

Where does network control live after AX Gateway removal?

Do not keep applying the old AX Gateway example. The reviewed Task schema has no gateway binding. Design outbound restrictions in the actual execution/network layer and verify they apply to the task’s actor.

Substrate’s current egress documentation describes policy-controlled HTTP(S), DNS restrictions and blocked WebSocket, forward-proxy CONNECT and other TCP traffic. It is a protocol contract to validate against your installed version, not an AX YAML shortcut. This matters when an SDK or MCP transport assumes a connection type that the runtime does not permit. Review Substrate’s egress traffic rules.

Test allowed model and tool endpoints, denied destinations, redirects, direct provider access and credential revocation. Keep request-level authorization on allowed servers: a permitted hostname is not permission to access every tenant or document behind it. These are proposed checks, not vulnerabilities we reproduced. The agent sandbox security checklist covers the wider boundary.

Does AX automatically stop a runaway token bill?

The reviewed Task schema does not declare a cumulative money or token cap. UsageStats and PendingApproval are not evidence of budget enforcement. Nor do CPU limits cap a remote model bill.

Use an independently controlled model-admission layer. Attribute setup, subtasks and retries to one durable job ID. Reserve enough budget before admitting concurrent calls, reconcile usage afterwards and stop new admissions at the limit. Already admitted requests can still incur charges, so define a bounded overshoot rather than promising an instantaneous zero-overrun stop.

The controller that can revoke credentials or block new requests should sit outside the agent’s writable workspace. A prompt asking an agent to respect its budget is not a spending control.

Can you self-host AX without Google Cloud?

A non-GCP development path exists, but production portability needs its own test. Substrate documents local kind and GKE quickstarts. AX needs a reachable Substrate control API, its own control plane and Redis, and suitable runner images. A local runtime can still use remote inference.

The current Substrate README describes a pre-1.0 project without backward-compatibility guarantees. AX warns of major breaking changes, and its roadmap still includes migration work against Substrate’s actor API. Select a tested version pair rather than assuming the newest revision of each project is compatible.

Is Antigravity required for every task?

No. The documented default runner invokes Antigravity for a workspace binding with a goal. That path requires GEMINI_API_KEY and has a default ten-minute bootstrap timeout controlled by AX_BOOTSTRAP_TIMEOUT. A prepared environment without a goal avoids that particular bootstrap path. The same documentation places metadata on loopback and tells the command to use AX_METADATA_URL. Read the current sandbox startup and metadata contract.

What about Pi or a custom agent harness?

Treat it as an integration to validate, not a ready-made AX adapter established by this review. A custom image must supply /usr/local/bin/ax-task-runner, read AX_TASK_YAML and AX_WORKSPACES_YAML, prepare durable workspaces, expose readiness and supervise shutdown. Verify the harness’s session location and resume behavior independently.

Owning the repository is only one part of owning your AI. You should be able to inspect execution, export state, choose inference providers, revoke authority and leave the deployment. Cloud location alone does not answer those questions.

What must an AX pilot prove before production?

Use synthetic tasks and disposable systems first. These are our proposed acceptance gates, not tests AX has passed in our environment.

Record the AX/Substrate revisions, runner digest, model route, policy configuration and outcome for every run. Choose AX when measured idle-capacity and recovery gains justify the platform work. A small fleet with cheap, short jobs may not need this extra infrastructure yet.

Wavect’s AI engineering services cover integration and production hardening. Our Twinsoft AI case study is separate delivery experience, not an AX reference deployment. Use the pre-launch QA checklist to structure acceptance evidence, or discuss your agent restart costs and runtime requirements.