AI MVP Scope Template: Acceptance Criteria, Eval Set, Launch Gate, and What Belongs in the SoW

Scoping an AI MVP is different from a normal software MVP because the system is probabilistic, not deterministic. "User can reset password" is a binary you can write as a pass-fail test. "The assistant answers correctly" is not, because the same question can produce different answers when the model version, temperature, or phrasing shifts. So you replace "it works" with four things that must be named in the statement of work before money changes hands: a versioned eval set with golden answers, a target metric and threshold on that set, a launch gate that decides go-live, and explicit handling of the uncertain case plus a rollback. If a vendor's scope says "build an AI assistant" with none of those, they scoped a demo, and you will pay for the gap during hardening. A copy-ready template is below.

This is the document founders send around internally before they buy. It is written from the build side, with the awkward questions made explicit. Regulatory dates are current as of mid-2026 and hedged where they are moving.

Why "it works" does not work for AI

In normal software, acceptance is binary: given input X, assert output Y, done. An LLM gives different output for the same input across runs, so "the chatbot answers accurately" has no test behind it. There is no number, no defined set, no floor, nothing to sign off and nothing to dispute when quality is poor. The fix is statistical, not binary. You accept a measured rate on a defined set, ideally averaged across several runs, and for reproducible regression checks you pin the temperature to zero where the use case allows. The second trap follows immediately: a fix for one case silently breaks another, so the eval set has to run as a gate on every prompt or model change, not once at the end. That single shift, from "it works" to "it clears this bar on this set, every time," is the whole point of scoping an AI build well.

What an AI MVP statement of work must pin down

Each section earns its place. The load-bearing ones are the eval set, the acceptance criteria, and the launch gate.

Acceptance criteria: wrong versus right

This is the section that decides whether you can hold a vendor to anything.

Wrong, because there is no number, no set, and no floor: "the chatbot answers customer questions correctly," "the assistant is accurate and helpful," "the model rarely hallucinates," "it works well in testing."

Right, on the agreed 100-case eval set with fixed golden answers:

• at least 90% faithful answers, grounded in retrieved context, scored by an LLM-as-judge on a binary pass-fail and calibrated against human labels;
• at least 85% answer relevance;
• under 2% harmful or policy-violating outputs, as a hard gate where any single harmful output blocks launch;
• refuses or escalates on at least 95% of the unanswerable test cases;
• p95 time-to-first-token under 2 seconds, p95 full response under 4 seconds;
• cost under EUR 0.05 per conversation at the agreed model;
• no regression below any floor on the eval run before a deploy.

Treat the exact thresholds as starting points to negotiate against your risk, not gospel. Vendors commonly cite faithfulness at or above 0.75 and hallucination under 5% (1% for high-stakes) as production starting points; set yours by how much a wrong answer costs you. The eval set you scope here is the same evidence an investor's technical diligence will ask for, which we cover in the technical due diligence checklist for AI MVPs, and the scoring methods are in when LLM evals are worth building.

The copy-ready scope template

Paste it, fill the brackets, delete what does not apply. Send it around before you take a single proposal.

"If the scope cannot tell you, in numbers, what good enough looks like and what happens when the model is wrong, it is not a scope. It is a wish. The eval set and the launch gate are the two lines that turn an AI demo into something you can actually buy."

A note on the AI Act

If your app interacts with users, the EU AI Act's Article 50 transparency duties, including telling users they are dealing with AI, apply from 2 August 2026 and are largely unaffected by the proposed changes. Most high-risk obligations were also due then. A provisional agreement in 2026 would push the standalone high-risk duties to late 2027, but as of mid-2026 that is not yet law and takes effect only on formal adoption. Do not scope your compliance plan around a delay that has not been enacted.