Google Harness²: Setup, Benchmarks and the Real Cost

Back

Kevin Riedl

14 min read · 16 Oct 2026
Last reviewed October 16, 2026

Next

Your AI agent may not need a smarter model. It may need a better way to work.

That is the interesting idea behind Google Harness². Instead of only asking an agent to try again, the framework changes the instructions, tools and reusable routines around it, tests alternative approaches, and saves lessons for later tasks.

The opportunity is real. The social-media shorthand needs work. Harness² is not a ready-made no-code office assistant, its benchmark improvements are not guaranteed productivity gains, and an editable guardrail file is not a security boundary.

This guide separates those questions: what Google released, what the reported numbers mean, what recursion costs, and how to approach a first evaluation without handing an experimental agent your company's credentials.

Source review: 16 October 2026. The commands below were checked against the pinned source, not executed as model-backed experiments. Benchmark results are the authors' measurements, not Wavect measurements.

What is Google Harness²?

Google Harness², also written Harness2, is an Apache-2.0 research framework for improving an AI agent's execution harness on unfamiliar tasks while keeping model weights fixed. It combines task-specific experimentation with a persistent library that guides future improvements. The official repository supports OpenCode and Codex executors and three benchmark families. It also states that this is not an officially supported Google product.

A harness is the working system around the model. It supplies instructions, tools, memory and execution behavior. Harness² works on that system, not on the model's trained parameters. Its package metadata identifies the reviewed code as version 0.1.0, with Python 3.12 or 3.13 support.

“Open world” describes tasks not anticipated when the harness was designed. It does not mean unrestricted internet access is necessary, that every unfamiliar task can be solved, or that a correct answer is guaranteed.

For the broader architecture, use our AI agent harness guide. Here the focus is Google's recursive approach and the practical decision to evaluate it.

How recursive harnessing actually works

Harness² has two connected loops. The paper's method section describes a Propose, Probe, Compose step for the current task, followed by reflection that can help later tasks.

First, a reference execution shows how the initial harness behaves. The system proposes a general improvement, such as better tool use, and a task-specific improvement, such as a more appropriate checking procedure. It executes both candidates on the same task with the same executor model and environment configuration. It then composes a revised harness from the observed differences and executes that version.

Think of a hypothetical spreadsheet-reconciliation task. One candidate might improve how the agent plans the work. Another might add a reusable reconciliation script. The comparison asks which change actually helped the observed behavior, rather than accepting a persuasive explanation. This is an illustration, not a tested Harness² integration.

The second loop retains adopted harnesses and evidence-supported lessons, including an EXPERIENCE.md file. Those records guide future proposals. Importantly, each new task still starts from the common base execution harness. It does not automatically inherit every change made for the previous task. If reflection lacks sufficient evidence, it can add no new experience entry.

That distinction matters for buyers. This is learning through persistent artifacts and better configuration, not continuous model training. Nor is behavioral contrast an independent correctness check: multiple executions can share the same misunderstanding.

The eight editable components

Our deployment rule: let the system improve its method, but keep its authority outside the editable harness. A file saying “do not export customer data” should not be the only thing preventing an export.

What do the +11.5 and +4.5 results actually mean?

The headline gains are real entries in the authors' tables. They are maximum improvements from different model-and-harness configurations, not one universal uplift. The main results and experimental setup report means over three runs with shuffled task order. The following comparisons use one improvement step, k=1.

The WorkBuddy figure is not “every office task becomes 11.5% better.” With Gemini 3.5 Flash under OpenCode, that benchmark's reported average rises from 72.4 to 77.5, a 5.1-point gain. The relevant baseline matters.

There is another easy-to-miss detail: the main experiment's default harness editor is Claude Opus 4.8, even when the executor is Gemini or GLM-5. The same frozen executor is compared before and after improvement. That does not mean the entire improvement pipeline uses only that one model. The paper separately investigates configurations where the executor and editor are the same model.

The + Recursion rows describe additional changes with parallel k=5. Do not present them as the result of the basic one-step setup or combine maxima from different rows into a fictional best configuration.

The positive conclusion is narrower and more useful: the paper supplies evidence that changing how an existing model works can improve its results across several professional benchmarks. It does not establish your production acceptance rate, operating cost or required human-review effort.

What does Harness² cost to run?

The code license is not the operating bill. A recursive run pays for exploration as well as the delivered result.

For parallel recursion, the documented design has k reference executions, k general candidate probes, k domain-specific probes and k composed-harness executions. Adding those gives 4k executor rollouts: four at k=1, twenty at k=5. That is a count derived from the algorithm, not a measured fourfold or twentyfold invoice.

Editor, composer, reflection, any final selection, and separate grading add their own work. Token lengths, caching, model prices, failures and retries determine actual cost. Parallel execution can reduce elapsed time when capacity is available; it does not make those calls free.

The paper does not ignore this issue. Its compute analysis counts tokens used before delivery, including execution and improvement, and reports a favorable quality-versus-compute trade-off among the compared configurations. However, the execution-step column in the main tables is not a complete production-cost ledger, and benchmark token efficiency is not a euro-denominated business case.

Measure cost per accepted outcome

Use the same independently defined acceptance criteria for the fixed harness and Harness²:

Cost per accepted outcome = all executor, improvement-model, grading, infrastructure and human-review costs for the evaluated cohort / independently accepted deliverables.

Count each cost once. If no deliverable is accepted, do not report a finite unit cost.

Here the recursive option completes more work successfully but costs more per accepted result. It may still be worthwhile for higher-value outputs or faster turnaround. The point is to make that decision explicitly rather than confusing a benchmark-score increase with automatic savings.

For a broader operating-cost model, see AI agent cost per action. Keep the Harness² experiment focused on whether additional attempts buy enough usable work.

Is Harness² safe to connect to company systems?

Not on the strength of its default research setup alone. The experiment guide explicitly says the release does not restrict executor network access and that Codex runs with its sandbox disabled.

It also documents a validation run in which a Codex rollout downloaded its JobBench task's public grading rubric without being instructed to. That is a concrete warning for reproduction. It is not evidence that the paper's original headline results are contaminated.

Keep two different risks separate. A benchmark agent finding its answer key undermines evaluation. An agent reaching real credentials, unrelated files or external systems threatens the environment. Neither is solved by adding a sentence to guardrails.md.

Our recommended first-run boundary is a disposable isolated environment, synthetic or sanitized files, limited model credentials, externally enforced egress restrictions, and an execution budget that the agent cannot increase. Do not mount personal cloud profiles, production secrets, a sensitive home directory or an unrestricted host Docker socket into the agent's environment.

For benchmark evaluation, setup and execution need different network policies: installation downloads dependencies, while execution should not retrieve public answers or grading criteria. Test the restrictions rather than assuming that a configuration file enforces them.

Review persistent lessons before reuse, too. A domain-organized library is not automatically a customer-data boundary. Keep tenant separation, retention rules, artifact access and deletion controls outside the model's discretion. Treat generated scripts, plugins and deliverables as untrusted until reviewed.

How to try Google Harness² without starting a paid benchmark

The setup guide recommends Linux and Python 3.12 for benchmarks. Model-backed runs also require Node.js 20 or newer, the Google Cloud CLI and access to configured models. LAB adds document-processing dependencies; WorkBuddy adds Docker.

That is developer setup, not a no-code signup. A non-coder can help define the task and judge whether the outcome is useful. Someone still needs to own the runtime, credentials, isolation and failure handling.

Start with the credential-free demonstration

Inside a disposable Linux environment, the source-reviewed sequence is:

git clone https://github.com/google-research/harness2.git
cd harness2
git checkout 1cfe254e8d25b15f37048c3730134686abe60335
uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e '.[render]'
harness2-demo --output runs/demo --k 2

This pins the reviewed repository state. The demonstration uses deterministic editors and real Python executions without model credentials. It demonstrates the recursion mechanics, not the benchmark uplift of an LLM. Installing the environment and dependencies still requires network access.

Preview a narrowly scoped model-backed run

Only move on after configuring a limited cloud project, the desired executor, improver and judge models, and the external controls above. The defaults are not a promise to reproduce every paper configuration. Review .env.example; the documented Vertex route uses Application Default Credentials.

The benchmark guide separates JobBench, LAB and WorkBuddy setup. Their datasets and third-party code retain their own terms. In particular, WorkBuddy has an upstream custom license with a territorial limitation; review it before downloading that benchmark. Harness²'s Apache license does not override it.

The following LAB example selects one named domain and previews at most one task from it:

export HARNESS2_RUNS_DIR=runs/lab-first-k1
harness2-setup lab
harness2-check --bench lab
harness2 --bench lab --k 1 --mode parallel --seed 42 \
  --only-domains antitrust-competition --max-tasks 1 \
  --workers 1 --dry-run

harness2-check inspects dependencies, credentials and configuration without a model call. It does not prove that every selected model is available. --dry-run previews task selection without model calls.

The important limit is --max-tasks: it applies per domain, not across the entire run. Pair it with a single --only-domains value for this first experiment. --workers 1 limits domain concurrency, not the total API bill. Use an external spending limit as well.

Removing --dry-run starts paid model work. To make that boundary explicit, keep execution and grading separate:

harness2 --bench lab --k 1 --mode parallel --seed 42 \
  --only-domains antitrust-competition --max-tasks 1 --workers 1
harness2-judge --bench lab --k 1 --mode parallel --seed 42
harness2-report --bench lab --k 1 --mode parallel --seed 42

Use the same benchmark, executor, subset where applicable, recursion budget, mode and seed when judging and reporting. The seed shuffles tasks; it does not fix model sampling. Start a new output directory when changing models or task selection. To grade the reference executions, run the matching judge command with --bases; the standard report covers refined results.

Do not mistake this benchmark interface for a ready-made connector to your office workflows. A real business task still needs a suitable adapter, data access, acceptance checks and operational ownership.

A pilot that can justify adoption

Before increasing k, ask whether the improvement is useful outside the system's own reflection. Anthropic's agent-evaluation guidance distinguishes an execution transcript from the actual outcome in the environment. That is the right distinction here: a confident completion message is not a reconciled spreadsheet or a correctly updated record.

Our proposed acceptance matrix is deliberately stricter than “the demo ran”:

The release leaves failed evaluations unscored rather than counting them as zero. Report that coverage explicitly. A score based only on successfully graded tasks can hide operational failures.

Also evaluate when recursion should not run. A cheap deterministic script may already solve a stable transformation reliably. Saving exploratory model calls for ambiguous, higher-value tasks is a sensible pilot hypothesis, not a capability to assume is configured automatically.

Where Harness² fits in a Wavect project

Harness² is interesting when a recurring class of valuable tasks has enough variation to defeat one fixed procedure, but enough structure to judge the outputs. It is less compelling when the job is already deterministic, the latency budget is tiny, or every extra attempt creates unacceptable side effects.

Our AI engineering services start with the workflow and acceptance criteria, not a promise that a framework will make itself reliable. The Twinsoft AI case study is separate delivery context, not a claim that Twinsoft runs Harness². Use our pre-launch QA checklist to define the evidence a pilot must produce, then discuss a bounded pilot with Wavect.

The useful commercial question is not “How many agent attempts did we buy?” It is “Which independently accepted outcomes did those attempts make possible, at what total cost?”