AI Pilot Kill-or-Scale Scorecard: 12 Metrics to Check After 30 Days

AI pilot success criteria should answer one decision: has this system produced enough reliable business value to justify the next euro of investment? After 30 measured operating days, evaluate 12 metrics across economics, automation quality, reliability, adoption and governance. Score each green, amber or red, but never let a high average hide an unacceptable safety, false-positive or auditability failure.

This is the measurement model, not the rollout plan. If you are deciding what to do in days 0 to 90, use our 30/60/90-day AI agent pilot plan. This scorecard starts after the workflow has accumulated 30 days of representative shadow or live usage. For a deeper operating-cost model, use AI agent cost per successful action.

What are good AI pilot success criteria?

Good AI pilot success criteria compare the new workflow with a measured baseline, count only outcomes that pass a business quality bar, include human correction and failure costs, and define hard governance gates. They are written before the pilot starts and produce an explicit scale, iterate or kill decision on a fixed date.

The clock matters. "After 30 days" means 30 days in which representative users and representative work could reach the system. A month spent waiting for permissions, test data or an integration is evidence about data readiness, not a valid sample of user adoption or task performance. Record the blocked days, fix the cause once, then start or extend the measurement window deliberately.

The framework below follows the measurement logic in the NIST AI Risk Management Framework Playbook: compare against a human or pre-deployment baseline, monitor real behavior, record overrides and escalations, and document accountable go or no-go decisions. Microsoft now separates agent evaluation into end-to-end task completion and process measures such as tool-call success, while Google Cloud evaluates immutable traces containing model inputs, responses and tool calls. That is why this scorecard measures the outcome and the path that produced it.

The 12-metric AI pilot scorecard

These bands are Wavect's recommended starting point for a low-risk internal operations workflow. They are not universal industry benchmarks. Replace them before kickoff when your economics, risk tolerance or service-level agreement demands a stricter bar.

Do not copy a threshold simply because it appears in a table. A 20% escalation rate can be healthy for a high-risk approval workflow and fatal for tier-one support deflection. A 95% success rate can be strong for drafting internal summaries and unacceptable for releasing payments. Write the business consequence first, then set the threshold.

How do you calculate each AI pilot KPI?

1. Baseline task cost

The baseline is the cost of today's process before AI. Time one representative sample, not the five easiest cases. Include fully loaded labour, software fees attributable to the task, review, rework and the expected cost of errors. Microsoft's agent metrics reference defines cost per transaction in the same fully loaded way: productive time, system cost and rework captured before the agent is built.

Baseline task cost = (median labour minutes × loaded hourly rate ÷ 60) + system cost per task + rework cost per task + expected error loss per task.

Use the median for the typical case, but retain P95 task time and cost. One blended average can hide an exception queue that consumes the entire saving.

2. Cost per successful action

A model response is not a business outcome. Define success before the run starts: an invoice posted with correct fields, a ticket resolved without reopening, or a lead accepted by sales. Keep model calls, retrieval, tools, platform overhead, retries, human review and failure handling in the numerator. Keep only actions that pass the agreed quality bar in the denominator.

Cost per successful action = (model + retrieval + tool + platform + retry + human correction + incident costs) ÷ successful actions.

3. Straight-through completion rate

Straight-through completion rate, also called touchless rate, is the cleanest automation metric. Microsoft defines touchless rate as the share of transactions completed end to end without human intervention. A run does not count as straight-through if a person corrected, approved, rescued or manually re-entered it.

Straight-through completion rate = successful actions with zero human intervention ÷ eligible attempted actions × 100.

4. Human correction time

Measure correction time across all eligible attempts, including zero-minute cases. Measuring only corrected outputs makes the average look worse as quality improves; measuring only AI runtime makes human work disappear. Record review and approval time separately if every action is intentionally supervised.

Human correction time per attempt = total minutes spent correcting AI work ÷ all eligible attempts.

5. Tool-call failure rate

Count timeouts, authentication failures, permission denials, schema validation errors, rate limits and 4xx or 5xx responses. Also report failures per tool because one unreliable CRM connector can be hidden inside a healthy overall average. Microsoft's agent evaluators explicitly separate end-to-end task completion from tool selection, input accuracy, output use and technical tool-call success.

Tool-call failure rate = failed technical tool calls ÷ all tool calls × 100.

6. False-positive cost

Accuracy treats every error as equal. The business does not. A false fraud alert that blocks a good customer, a false legal finding that creates review work, and a false sales lead that wastes ten minutes have different costs. Google's classification guidance recommends choosing metrics according to the relative cost of false positives and false negatives rather than relying on accuracy alone.

False-positive cost per attempt = sum of investigation, reversal, remediation, customer and compliance cost caused by false positives ÷ eligible attempts.

For capacity planning, also calculate monthly exposure: false-positive rate × monthly decisions × average cost per false positive.

7. P50 and P95 latency

P50 is the typical experience. P95 exposes the slow tail caused by retries, large context, slow tools and escalation loops. Measure from the business event entering the workflow to the usable outcome, not just the model API call. OpenTelemetry's generative AI conventions capture model-call duration, tokens and nested agent or tool spans, which lets a team trace a slow outcome to the component that caused it.

P50 latency is the duration at or below which 50% of actions finish. P95 latency is the duration at or below which 95% finish.

8. Escalation rate

An escalation is a deliberate transfer to a person because the system lacks confidence, authority, data or capability. Do not mix it with crashes or silent abandonment. Escalation can be a good control, but it still consumes capacity and limits automation value.

Escalation rate = actions deliberately handed to a human ÷ eligible attempts × 100.

9. User adoption

Invitations, accounts and one-time logins are not adoption. Define a meaningful use event tied to the workflow, then count repeat use among people who actually had eligible work. Microsoft's routine adoption definition uses four or more active days in a rolling four-week window. Use that when it fits, or set a task-frequency threshold before kickoff.

Routine user adoption = eligible users who meet the pre-agreed repeat-use threshold ÷ eligible invited users × 100.

10. Auditability

A run is auditable when an independent reviewer can reconstruct what entered the system, which model and prompt version ran, which tools were called with what result, where policy or human intervention occurred, and what business outcome followed. NIST recommends histories and audit logs that support error and vulnerability review, plus statistics on overrides and escalations.

Auditability rate = sampled runs with a complete reconstructible trace and linked business outcome ÷ all sampled runs × 100.

For EU high-risk AI systems, logging and human oversight are not optional scorecard polish. The EU AI Act requires technical logging capabilities across the system lifecycle and appropriate human oversight. Treat any missing high-impact trace as a hard stop while legal classification is confirmed.

11. Data-readiness failure rate

Tag failures caused by missing fields, stale records, duplicates, poor document quality, inaccessible sources, incorrect permissions and unrepresentative test data. Do not relabel them as model failures. AWS frames a serious generative AI proof of concept as validation of business value, data readiness, technical feasibility and risk mitigation, not as a demo of model output.

Data-readiness failure rate = attempts blocked or invalidated by unavailable, unsuitable, stale or inaccessible data ÷ eligible attempts × 100.

12. Payback period

Payback asks how long the post-pilot investment takes to recover through net monthly benefit. Include the remaining integration, security, data, training, monitoring and change-management spend. Net benefit must subtract ongoing AI operating cost, residual human work and expected failure loss.

Monthly net benefit = monthly baseline cost avoided − AI operating cost − residual human cost − expected failure loss.

Payback period in months = remaining production investment ÷ monthly net benefit.

If monthly net benefit is zero or negative, payback is not "long." It does not exist.

How does the kill-or-scale scoring model work?

Score each metric green = 2, amber = 1 or red = 0. The maximum is 24 points.

Three hard gates override the total:

• Unacceptable harm or false-positive event: kill or pause even when the economics are green.
• Missing audit trail for a high-impact action: do not increase autonomy until every such run is reconstructible.
• No defensible baseline or outcome definition: the pilot cannot prove value, so do not scale from its score.

This prevents safety and evidence from being averaged away. A pilot cannot compensate for an unauditable payment decision by loading faster.

Worked example: invoice operations AI pilot

An Austrian operations team pilots an agent that extracts invoice data, validates it against policy, checks the vendor record and routes the invoice. It runs for 30 measured operating days on 1,000 eligible invoices. A successful action passes field validation and reaches the correct queue. A straight-through action also requires no correction or escalation.

The result is 18 out of 24: iterate once, then rescore. The economics, adoption and auditability support another controlled step, but broad rollout would hide avoidable operational debt. The next 14-day cycle has three named jobs: fix the connector producing most tool failures, repair the vendor master data causing blocked invoices, and cut the P95 retry path below 60 seconds. If those metrics do not move, stop extending the pilot.

What should the day-30 decision meeting contain?

1. One page of definitions. Eligible attempt, successful action, straight-through action, escalation, false positive and data failure must mean one thing.
2. The baseline and sample. Show the pre-AI measurement period, case mix, exclusions and volume.
3. The 12-row scorecard. Include target, actual, band, owner and evidence link for every metric.
4. The expensive traces. Review the slowest, most corrected and highest-cost runs, not only the averages.
5. Hard-gate evidence. List incidents, false positives, missing traces, privacy issues and unapproved access attempts.
6. One signed decision. Scale one step, run one bounded iteration, pause for missing evidence or kill. Name the accountable sponsor and the next review date.

Google Cloud's current agent evaluation workflow follows the same loop: define cases, generate traces, compute task-success and safety metrics, analyze failures, then verify improvements. The scorecard adds the business and adoption layer required for a funding decision.

"A pilot is not successful because the model looked intelligent. It is successful when a representative workflow became cheaper or better, users chose to use it, failures stayed inside the risk budget, and another team can reconstruct what happened without asking the people who built it."

Where does this scorecard fit in the AI pilot toolkit?

Use this scorecard for the day-30 funding decision. Use the 30/60/90-day pilot plan for sequencing, permissions, shadow mode, limited production and handover. Use the cost-per-action model to investigate token, tool, retry, caching and model-routing economics. Together they answer whether to fund the next step, how to run it and where the operating cost comes from.

Sources and methodology

This is an original Wavect decision framework. The formulas are defined here so teams can reproduce the score rather than accept a black-box benchmark. The example thresholds are recommendations for a low-risk internal workflow, not claims of a universal industry average. Source concepts and metric definitions were checked on 16 July 2026 against:

• NIST AI RMF Playbook, Measure, for baselines, production monitoring, human overrides, audit logs and accountable go or no-go decisions.
• Microsoft's agent business-value metrics reference, for adoption, resolution, escalation, touchless rate, cycle-time percentiles and cost per transaction.
• Microsoft Foundry agent evaluators, for task completion and tool-call process evaluation.
• Google Cloud agent evaluation, for trace-based task-success, safety and failure analysis.
• Google's classification metrics guide, for matching false-positive and false-negative metrics to business cost.
• OpenTelemetry's generative AI observability guide, for model, token, tool and latency tracing.
• AWS Generative AI Lifecycle Operational Excellence, for validating business value, data readiness, technical feasibility and risk mitigation in a proof of concept.
• Regulation (EU) 2024/1689, the EU AI Act, for logging and human-oversight requirements applicable to high-risk AI systems.