Coverage tells you what ran, not what mattered

Every few months somebody points out that we could have built this on top of the coverage report, since the coverage tool is already recording which lines executed. It is a fair question and the answer is a useful way to explain what selection is.

Same instrumentation, opposite direction

A coverage report aggregates. It takes every test in the suite, unions what they touched, and hands you one number per file: this line was reached, that one was not. The aggregation is the point, because the question is “is any of this untested”.

A selection map refuses to aggregate. It keeps one entry per test, listing what that individual test loaded, and it never merges them. The question is the other way round: given this file, which tests can see it. You cannot recover that from a union. Once you have added the tests together you have thrown away the only thing selection needs.

You can build both from the same hooks. You cannot build one from the other.

The direction that bites

Suppose packages/core/money.ts is covered at 96 percent. Excellent. Now change it. Which tests do you run?

The coverage report cannot answer. It knows the file was well exercised. It does not know by whom. The best you can do is run everything, which is where you started.

Now invert it. Suppose billing/tax_region_spec.rb loaded eleven files on its last run and money.ts is not one of them. That is a fact about one test, recorded from one real execution, and it is enough to leave that test out of this commit with a reason you can print.

Two rules that keep it honest

The failure mode of selection is skipping something you should have run, and it is a quiet failure, which makes it the one worth engineering against. Two rules do most of the work:

• A stale entry is not an answer. If a test’s map entry is older than the test’s own file, run it. The map is a record of a past execution, and the test has changed since.
• An absent entry is not an answer either. A test that has never been mapped is always run. An unbuilt map and a map that says nothing is affected look identical, and the safe reading of both is the same.

Neither rule is clever. Both are the difference between a tool that skips tests and a tool you can leave switched on.

The case where it does nothing

Change the base image and every test in the repository is downstream of the change. The map is not wrong and the tool is not failing. The correct selection is the whole suite and the run takes as long as it used to.

We put that case on the front page rather than in the small print, because a selection tool that reports a saving on a commit like that is not measuring your repository. It is measuring how much you wanted to hear a number.