Skip to content
GuideGated resourceQA and Testing

Measuring QA coverage in a way that predicts escaped defects

Line coverage can be high while checkout is broken. Journey coverage is the number worth reporting.

June 24, 202611 min readTexas Solutions QA practice · Quality assurance

Most teams report code coverage because it is easy to generate. It is also weakly correlated with whether customers hit bugs, which makes it a poor thing to manage by.

Count journeys, not lines

List the user journeys that produce revenue or that would cause a support incident if they broke. Usually there are fifteen to forty. For each one record whether it is automated, manually tested or untested. That ratio is your coverage number, and it moves in step with escaped defects.

Rank by cost of failure

Automate in order of what failure costs, not in order of what is easy to automate. Checkout, signup, payment and permission boundaries come before settings pages, however tempting the easy wins look on a burndown.

Protect the suite from flakiness

A suite that fails randomly gets ignored, and an ignored suite is worse than none because it creates false confidence. Track flake rate per test, quarantine anything unreliable and fix it as a defect rather than rerunning until it passes.

Keep the feedback loop short

A regression pack that takes four hours will not run on every pull request. Parallelise, shard and keep the critical path under an hour. Speed is what makes coverage get used.

Report three numbers

MetricWhy it matters
Journey coverageShare of revenue-critical paths under automation
Escaped defects per releaseThe outcome measure that actually matters
Mean time to detectHow long a regression survives before anyone notices

If those three move in the right direction, the testing is working, whatever the line coverage percentage says.

QAAutomationMetrics

Turn the reading into a number

The calculator uses the same published ranges referenced in these articles. Seven questions and you have a figure in your own currency.