External QA Benchmark: What We Find in the First 30 Days of a Software Product

There is no credible universal median for the number of defects an external software QA audit should find in 30 days. Product size, release frequency, test access, user roles and the definition of a defect vary too much. A raw count without those denominators is marketing, not a software defect benchmark.

What the research does support is more useful: external testing should concentrate on changed and high-churn areas, permissions, failure paths, configuration combinations and production escapes. Wavect's operational observation that a new QA team often raises issue discovery by 40–70% in the first month is consistent with those mechanisms. It is not a universal industry constant, and the studies below do not prove that exact range.

This page owns the observed-data question. Our software QA service explains the engagement; the software QA checklist before launch explains what a team should check. This benchmark explains which findings published research says an independent review is most likely to expose.

What is a realistic first-30-day external QA benchmark?

A defensible benchmark measures discovery lift, severity, defect class, prior awareness and production escape against the same product's own baseline. It does not compare a five-screen internal tool with a multi-tenant payments platform.

How many software defects should an external QA audit find?

The honest answer is: enough to change a decision, and no more than the evidence supports. Ten distinct authorization failures can matter more than 100 cosmetic inconsistencies. A benchmark based only on ticket count rewards splitting one cause into many reports and punishes teams that deduplicate well.

For the first 30 days, report both the median number of confirmed findings per tested release and a normalized rate:

• Confirmed findings per tester-day shows discovery productivity without pretending products are the same size.
• Discovery lift compares the external team's confirmed findings with the internal team's previous 30-day baseline.
• Net-new share excludes items already present in the backlog before external QA started.
• Defect escape ratio divides production defects by all confirmed defects for the same release window.
• Critical/high closure time measures whether the audit changed risk, not whether it created tickets.

Our own 40–70% first-month figure is a discovery-lift range: external QA findings compared with the product's preceding internal flow, after duplicates and non-defect requests are removed. It is not “40–70% of all bugs,” and it should not be used as a guarantee for an individual product.

Why authentication and permission defects appear early

Authentication proves who a user is. Authorization decides what that user may read or change. Internal teams usually test the intended role on the intended path. An external tester switches role, account, tenant and object identifiers, then checks whether the server, not merely the interface, enforces the boundary.

That emphasis matches the OWASP Top 10:2025 broken-access-control dataset: the category remained number one and recorded more than 1.8 million contributed occurrences. The number is not a prediction that every product has an access-control bug. It is evidence that permissions deserve an early, explicit test lane.

Why regression and high-churn code deserve disproportionate attention

A 2026 study covering more than 14,000 pre- and post-release defects found that escaped defects concentrate in older, frequently modified and high-churn components, and usually require more complex fixes than defects caught before release. Separately, research on OSS-Fuzz found that four out of five fuzzer-reported bugs had been introduced by recent code changes.

The practical conclusion is not that 80% of every product's bugs are regressions. The OSS-Fuzz sample is specific. The defensible conclusion is that the first month of external QA should weight recent changes and mature hot spots more heavily than untouched screens. See the 14k-defect pre/post-release study and the regression greybox fuzzing study.

What browser, device and configuration testing adds

Many failures are interactions: a role plus an old session, Safari plus a date input, a retry plus a partial write. NIST's combinatorial-testing program reports that most studied software failures were triggered by one or two parameters, and that multiple studies achieved exhaustive-equivalent fault detection with test sets 20 to 700 times smaller.

That does not promise a certain number of browser defects. It explains why a deliberate matrix beats repeating the happy path on Chrome. The first 30 days should select combinations from actual analytics, supported devices, user roles, data states and network conditions. The evidence and limitations are documented by NIST's combinatorial testing program.

How many defects have already reached production?

No public study gives a universal percentage for normal SaaS products. Production escape depends on release cadence, observability, reporting behavior and whether the product was already live when QA began. Publish the product's own escaped-defect ratio and split it by severity.

The strongest evidence for why this matters comes from a USENIX analysis of 198 user-reported failures in five distributed data systems. Seventy-four percent were deterministic, 77% could be reproduced by a unit test, and 58% of catastrophic failures could have been exposed through simple error-handling tests or statement coverage. Those results do not generalize numerically to every web app, but they show that “it only failed in production” often means the relevant failure path was never exercised. Read the USENIX production-failure study.

How should AI-specific failures be counted?

Do not mix a hallucinated answer, a prompt-injection path and a conventional permissions bug into one “AI defect” bucket. Record the product behavior that failed and the AI trigger separately. NIST's AI test, evaluation, validation and verification work stresses that metrics change with context. Its ARIA pilot evaluated seven AI applications at model-testing, red-teaming and field-testing levels, not through one generic accuracy score.

For an AI product, the 30-day report should therefore add eval pass rate, harmful-action rate, unsupported-answer rate, retrieval/permission leakage, cost and latency thresholds, and human-fallback success. Use NIST's AI TEVV guidance and the OWASP Top 10 for LLM applications as taxonomies, not prevalence claims.

What a buyer should receive after 30 days

• A deduplicated defect register with reproducible evidence and one severity rubric.
• A benchmark sheet showing raw count, tester-days, discovery lift, net-new share and escape ratio.
• A separate view of authentication/permissions, regression, browser/device, data-integrity and AI-specific findings.
• A critical/high retest result, not merely a list of open tickets.
• A short recommendation: continue external QA, build internal coverage, or stop because the marginal value is low.

That is the commercial test of an outsourced software testing engagement: after 30 days, can the buyer see which risks changed, what still reaches production and where the next euro of testing should go?

Sources and methodology boundary

This is a structured synthesis, not a pooled meta-analysis. The cited datasets cover different systems, dates and defect definitions, so their percentages must not be combined into one fake industry average. ISO/IEC 25010:2023 supplies a nine-characteristic product-quality model; OWASP supplies security taxonomies and contributed occurrence data; NIST supplies testing and evaluation methods; USENIX and empirical software-engineering research supply scoped defect observations.

Use the ISO/IEC 25010:2023 product quality model to define coverage, then publish your own cohort, dates, inclusion rules, denominators and uncertainty. That is how an external software QA audit becomes a benchmark that a buyer, search engine or LLM can cite without stripping away the caveat that makes the number true.