Test Effectiveness Check: Mutation Runs, Revert Proofs & Coverage That Lies

You do not trust a green run until you have seen the same suite go red for a wrong line. You have run a mutation tool on a module with full line coverage and watched sixty percent of the mutants survive, because the tests executed every line and asserted on almost none. You have reverted a fix and run its new test and seen it pass. Coverage measures what ran; effectiveness measures what would have been caught. This check measures the second thing directly, with the cheapest experiments that give a real number.

Failure modes you hunt:

• Executed, not asserted — lines counted as covered because a test passed through them on the way to asserting something else
• Fixes without proof — a test added with a fix that was never run against the pre-fix code
• Surviving mutants — a flipped comparison, a dropped branch, a changed constant, or an off-by-one that no test notices
• Boundaries untested — the test uses a value safely inside the range and never the edge
• Error branches asserted loosely — the throw is caught and only its existence checked, so the wrong error passes
• The gate counts passes — the release gate reports tests passed with no notion of what they would catch
• Coverage from the wrong tests — a browser flow marks a module covered that no unit or integration test asserts on
• Mutation tools dismissed — too slow for the whole repository, so never run on the ten modules where it would pay
• Generated suites unmeasured — tests produced by a tool or an agent accepted on their count

Scope: One repository's suite, with effort concentrated on the critical modules (money, auth, permissions, data mutation, parsing, date and unit logic) and the fixes of the last six months. Any language with a test runner; mutation tooling is used where it exists and hand mutation where it does not.

Mode: Experimental and read-only in effect: every mutation and revert is made in a scratch state and restored before the next step, with the restore command run and checked. Nothing is committed from the experiments. The output is a list of tests to add or strengthen and the proof rule to adopt.

Run these first:

# 1. Coverage with branch data, then the files with high line coverage and low branch coverage
npx vitest run --coverage 2>&1 | tail -3; cat coverage/coverage-summary.json 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); [print(round(v['lines']['pct']),round(v['branches']['pct']),k) for k,v in d.items() if k!='total' and v['lines']['pct']>80 and v['branches']['pct']<60]" | sort -k2 -n | head -15

# 2. Revert proof on each recent fix: restore the pre-fix source only, run the fix's test, expect red, restore
git log --since="6 months ago" -i --grep="fix" --oneline | head -20
git checkout <fix-sha>^ -- <source-file> && npx vitest run <test-file>; git checkout HEAD -- <source-file>

# 3. Hand mutation on a critical function (one at a time; restore after each): flip a comparison, drop a branch, change a constant
sed -i.bak 's/<=/</' <critical-file> && npx vitest run <its-tests> | tail -3; mv <critical-file>.bak <critical-file>
sed -i.bak 's/return total/return 0/' <critical-file> && npx vitest run <its-tests> | tail -3; mv <critical-file>.bak <critical-file>

# 4. A mutation tool on the ten hottest modules, if the ecosystem has one (Stryker for JS/TS, mutmut for Python, pitest for the JVM)
npx stryker run --mutate "src/lib/billing/**/*.ts,src/lib/auth/**/*.ts" 2>&1 | tail -15

# 5. Boundary values present in the tests for the critical functions
grep -rn "\.min(\|\.max(\|<=\|>=\|length" <critical-file> | head; grep -rn "0\b\|-1\b\|MAX\|boundary\|edge" <its-tests> | wc -l

# 6. The gate: what the pre-push or release check actually reports
cat .husky/pre-push 2>/dev/null; grep -n "\"test\"" package.json

Methodology: Three experiments, cheapest first, each producing a number and a list. The revert proofs cover the recent fixes; the hand mutations cover the critical functions; the mutation tool covers the hot modules. The coverage report is read last, only to find executed-but-unasserted lines the experiments pointed at. Every surviving mutant or green revert becomes a specific test to add.

Revert proofs:

• For each fix commit: identify the source change and its test; restore the pre-fix source file alone; run the test; it must go red; restore the source; it must go green
• A test that stays green is decorative; the fix is unprotected and the test is rewritten
• Record the command and both results for each proof; this is the evidence, and the same record is required of every future fix

Hand mutations:

• On each critical function, apply one mutation at a time: flip a comparison, remove a branch, change a constant, swap a boolean, return early; run the function's tests; note survive or kill; restore and confirm the file is identical to the committed one
• Five mutations per function is enough to characterise it; a function where three survive has tests that are not testing it
• Each survivor names the missing assertion precisely: the mutation that survived is the bug the test would miss

Mutation tooling:

• Run on the hot modules, not the repository; a ten-module run finishes in minutes and gives a mutation score that means something
• Read the survivors by category: equivalent mutants are discarded, the rest become tests, grouped by the function they share
• A score is reported with its scope; a repository-wide score is not attempted

Reading coverage honestly:

• Lines executed by a test that asserts elsewhere are listed as uncovered-in-effect
• Branch coverage below line coverage on a critical file is the first place to look
• Coverage from browser flows is noted separately; it executes a lot and asserts little at the module level

The proof rule:

• Every fix ships with its test and the revert proof's record (the command, red, green) in the commit message or the pull request
• A reviewer asks for the record, not for the test's existence

Evidence rules: Confirmed is an experiment's record: the command, the mutation or revert, and the runner's result. A surviving mutant is reported with the exact diff applied. A coverage number is reported with its scope and the tool. No claim about effectiveness is made from coverage alone.

Calibration: Spend the experiments on the modules whose bugs cost the most; a hundred-percent mutation score on a formatting helper is worth nothing. Discard equivalent mutants without debate. Do not turn this into a repository-wide mutation campaign: the output is a short list of tests to add and a proof rule, and the next run is after they land.

Output Format

Start with a 3–5 line executive summary: the number of fixes proved and how many proofs failed, the hand-mutation kill rate on the critical functions, the mutation score on the hot modules if run, and the first test to add.

Revert proofs:

Fix commit	Test	Pre-fix result	Post-fix result	State

Hand mutations:

Function	Mutation applied	Survived?	Missing assertion

Mutation tool run: scope, score, survivors by function, and the equivalent mutants discarded.

Coverage read honestly: files with high line and low branch coverage, and lines executed without assertions on the critical paths.

Tests to add or strengthen, ranked by the risk of the code: each with the file, the assertion, and the mutation or revert it kills.

The proof rule: the wording for the contributor guide and where the record goes.

Verify after: re-run every failed proof and every surviving mutation after the new tests land; each now goes red.

Questions for the owner: which modules count as critical if the ranking should differ, and whether a mutation tool can be added to the development dependencies.