AI MVP Scope Template: Acceptance Criteria, Eval Set, Launch Gate, and What Belongs in the SoW

Scoping an AI MVP differs from scoping a fully deterministic workflow because model outputs can vary with model version, sampling settings, context, and phrasing. "User can reset password" may support an exact pass-fail test; "the assistant answers correctly" needs a defined task distribution and scoring rule. Our recommended statement of work therefore names four things: a versioned eval set with reference labels or rubrics, a target metric and threshold, a launch gate, and explicit handling of the uncertain case plus rollback. A copy-ready template is below.

This is a procurement starting point, not legal advice or a universal quality standard. We re-checked the regulatory dates and evaluation guidance on 2 September 2026.

Want this scope pressure-tested before you sign with an agency?

Book Free Consultation

Why "it works" does not work for AI

Conventional software also uses statistical, performance, and property-based tests, so acceptance is not always binary. The distinctive issue here is that generated answers may have many acceptable forms and can vary between runs. Measure a defined set, repeat cases where variance matters, and record the exact model and configuration. Temperature zero can reduce sampling variance but does not guarantee identical output from a hosted model. Run regression evals on material prompt, retrieval, model, policy, or tool changes, with frequency proportional to risk. This follows NIST's recommendation to document test sets, metrics, deployment-like conditions, uncertainty, and ongoing monitoring in its AI RMF Measure guidance.

What an AI MVP statement of work must pin down

Each section earns its place. The load-bearing ones are the eval set, the acceptance criteria, and the launch gate.

Section	What it must pin down
Problem + one outcome	One sentence. The single user job the MVP must do. Everything not serving it is out of scope.
In and out of scope	Two lists. The out-of-scope list is the load-bearing one: name the tempting things you are not building (multi-language, voice, fine-tuning, mobile) so they become change requests, not assumptions.
Functional + AI behavior spec	Normal requirements plus the AI behavior: task, tone, refusal behavior (when it must say "I do not know"), citation requirement, and fallback on low confidence or no retrieval hit. This is where you encode the uncertain case.
Acceptance criteria	Target metric plus threshold on a named eval set. Never "it works." Examples below.
The eval set	How cases are sampled from intended and foreseeable use, how sample size is justified, who owns reference labels or rubrics, and how cases are scored. Use deterministic checks where possible; calibrate model graders against human review and document disagreement.
Launch gate + rollback	The measurable bar to go live, agreed by the accountable product, engineering, risk, and quality owners before launch, plus monitoring, a rollback trigger, and a kill criterion.
Data	Sources, provenance, usage rights, personal-data handling, transfers, retention, and residency requirements. Determine whether each vendor is a controller, processor, or sub-processor using the EDPB role guidance, then put the required contract and instructions in place. No-training and retention settings are product-specific controls, not universal GDPR wording.
Non-functionals	User-perceived and end-to-end latency, time-to-first-token where streaming matters, cost per successful task, concurrency, availability, and privacy-aware telemetry. Check the selected provider's current input, output, cache, tool, and media prices instead of assuming one fixed ratio.
Security and compliance	Authentication, authorization, tenant isolation, threat model, incident response, applicable GDPR duties, and the system-specific EU AI Act classification and transparency duties.
Milestones, payment, IP, handover	Phased discovery, build, evaluation, hardening, and launch, with commercial terms, IP provisions, third-party-license treatment, and named handover artifacts agreed in the contract.

Acceptance criteria: wrong versus right

This is the section that decides whether you can hold a vendor to anything.

Wrong, because there is no number, no set, and no floor: "the chatbot answers customer questions correctly," "the assistant is accurate and helpful," "the model rarely hallucinates," "it works well in testing."

Better structured, using illustrative placeholders that must be replaced with evidence-based thresholds for the actual risk and workload:

faithfulness target [threshold], grounded in retrieved context, scored against a rubric and calibrated against human labels;
answer-relevance target [threshold];
critical safety failures [zero or an explicitly approved risk limit], with blocking categories named;
refusal or escalation on unanswerable cases [threshold];
p95 time-to-first-token and full-response budgets [where each matters];
cost per successful task [budget] on the agreed model and workflow;
no regression below any floor on the eval run before a deploy.

These fields are examples, not industry baselines. There is no defensible universal eval-set size, faithfulness threshold, harmful-output allowance, latency target, or cost ceiling. Derive each from use frequency, consequence severity, subgroup coverage, confidence intervals, and the fallback available when the system fails. NIST's Generative AI Profile is voluntary guidance, but it provides a useful risk-based frame. The scoring methods are discussed in when LLM evals are worth building.

The copy-ready scope template

Paste it, fill the brackets, delete what does not apply. Send it around before you take a single proposal.

AI MVP Scope / SoW, [Project Name]
Date [date] · Version [v0.1] · Owner [name]

1. Problem + one outcome. Problem: [one sentence]. The one outcome this MVP must deliver: [user] can [do X] so that [Y].

2. Scope. In scope: [feature 1], [feature 2]. Out of scope, change request only: [multi-language], [voice], [fine-tuning], [mobile].

3. Functional + AI behavior. Functional: [list]. AI behavior: task [exactly what], tone [concise, no speculation], citations [must or must not ground in sources], refusal [when out of scope or low confidence, say "I do not know" or escalate], fallback [secondary model, cached answer, or human handoff].

4. Acceptance criteria. On eval set [name/version]: [metric] >= [threshold with rationale and uncertainty]; critical safety failures [zero or defined risk limit]; correct refusal/escalation >= [threshold] on unanswerable cases; p95 [latency measure] < [budget]; cost per successful task < [budget]; no material regression below any agreed floor before deploy.

5. Eval set. Size [N, with sampling rationale] across [happy path], [edge], [unanswerable], [adversarial], and relevant subgroup cases. Reference owner: [client domain expert]. Scoring: deterministic checks for [objective fields], model grader with rubric for [quality], and blinded human review for calibration and disputed cases. Stored and versioned in [location].

6. Launch gate + rollback. Go live when all Section 4 floors are met, signed off by [product] + [engineering] + [QA]. Rollback: if [metric] drops below [floor] over a [window], auto-revert. Kill criterion: do not ship if [faithfulness < X% or any harmful output].

7. Data. Sources [list], provenance and rights [per source], personal data [what, purpose, lawful basis, retention], transfers/residency [requirements]. Vendor role [controller/processor/sub-processor], Article 28 terms where applicable, training use [terms], retention [terms/settings].

8. Non-functionals. Latency, availability, concurrency, and cost budgets as in Section 4. Observability: record the minimum metadata needed for quality, security, and cost; redact or avoid raw prompts and responses unless justified; define access and retention; alert at [threshold].

9. Security + compliance. Auth [method], permissions and tenant isolation [model], threat model and incident response, GDPR duties [roles, record where required, lawful basis, DPIA where likely high risk, processor terms], and applicable EU AI Act role, classification, Article 50 duty, and deadline.

10. Milestones + payment. Discovery, build, eval and hardening, launch, payment per phase. IP ownership, assignment or licence, third-party components, and effective date: [contract terms]. Handover: eval set and results, prompt and model registry, architecture and data-flow diagram, runbook, logs access, credentials. Sign-off: client [__] vendor [__] date [__].

"If the scope cannot tell you, in numbers, what good enough looks like and what happens when the model is wrong, it is not a scope. It is a wish. The eval set and the launch gate are the two lines that turn an AI demo into something you can actually buy."

A note on the AI Act

Article 50 is not a blanket duty for every app that contains AI. For systems intended to interact directly with people, providers generally must inform them that they are interacting with AI unless this is obvious to a reasonably well-informed, observant, and circumspect person, subject to specified exceptions. Other Article 50 duties cover synthetic-content marking and certain deployer disclosures. Most applicable Article 50 duties began on 2 August 2026; a transition until 2 December 2026 applies to Article 50(2) providers whose synthetic-content systems were already on the market. Following Regulation (EU) 2026/1744, Annex III high-risk rules apply from 2 December 2027 and Annex I product-embedded high-risk rules from 2 August 2028. Verify the current consolidated AI Act text and your role rather than applying one date to every system.

Frequently Asked Questions

How do you write acceptance criteria for an AI feature?

Not as "it works." Define a target metric and justified threshold on a named, versioned eval set, plus blocking safety categories, refusal or escalation behavior, latency and cost budgets, and a regression rule. Measure repeated runs where output variance matters and document uncertainty rather than copying universal numbers.

What is an eval set?

A versioned, owned sample of intended, edge, unanswerable, adversarial, and relevant subgroup cases with reference labels or rubrics. There is no universal minimum size. Justify coverage and uncertainty for the decision and risk, then expand the set with production failures.

Who should own the eval set?

A domain expert on your side, not the vendor alone. The person who knows what a correct answer looks like must define the golden answers, or you are letting the builder grade their own homework.

How are eval cases scored?

Combine deterministic code checks for objective fields such as dates, IDs, and JSON shape with rubric-based model grading and human review where judgment is needed. Calibrate model graders against blinded human labels, track disagreement, and route disputed or consequential cases to people.

What is a launch gate for an LLM app?

The measurable bar that decides go-live: the thresholds and blocking conditions the system must meet on the agreed eval set, signed off by the accountable product, engineering, risk, and quality owners. It also defines monitoring, a rollback trigger, and a kill criterion.

What belongs in an AI statement of work?

Problem and one outcome, in and out of scope, functional requirements plus an AI behavior spec (tone, refusal, citations, fallback), acceptance criteria as metric plus threshold on an eval set, the eval set itself, the launch gate and rollback, data rights and PII and residency, non-functionals (latency, cost-per-action, logging), security and compliance, and milestones, payment, IP, and handover artifacts.

Why can I not just write "the AI answers correctly" in the scope?

Because generated answers can have several acceptable forms and may vary between runs or model versions. You need a defined set, a scoring rule, and a floor, or there is nothing objective to sign off and nothing to dispute when quality is poor.

How do I handle the case where the AI is wrong or unsure?

Specify a refusal, escalation, or fallback path and test it with unanswerable and ambiguous cases. Set the threshold from consequence severity and available human review rather than copying a universal percentage.

What data clauses does an AI MVP SoW need?

Document provenance and usage rights, personal-data purposes and lawful basis, retention, transfers and residency requirements, vendor roles, processor terms where applicable, training use, security controls, deletion, and audit rights. A model provider may be a processor or sub-processor depending on the contracting chain.

Does the EU AI Act affect my AI MVP scope?

It can, depending on the system, role, and use case. Applicable Article 50 duties generally began on 2 August 2026, Annex III high-risk rules apply from 2 December 2027, and Annex I product-embedded high-risk rules from 2 August 2028. Classify the system and verify the consolidated law.

Final thoughts

An AI MVP scope depends on a defensible eval set and launch gate. Together they turn a vague "build us an assistant" request into evidence and contract terms that a buyer can assess.

Before you take a proposal, write the outcome, draw the out-of-scope line, define justified metrics and floors on a set your domain experts govern, decide what happens when the model is unsure, and name the bar for launch. Fill in the template first so proposals can be compared against the same requirements.