How it decides.

The map

On the first full run, Bellwether attaches to the process and records every file each test opened, imported or required. That is not a guess about your architecture, it is the list of files the test actually touched, and it is per test rather than per file.

The map is rebuilt continuously. Every full run that happens for any other reason, a nightly job or a release build, refreshes it. There is no step in your pipeline whose job is to keep Bellwether informed.

The selection

On a pull request Bellwether takes the diff, walks it against the map, and returns the set of tests that can reach a changed file. Everything else is reported as skipped, with the reason attached, so a skip is auditable rather than silent.

Two rules keep the set honest. A test whose map entry is older than its own file is always run. A test that has never been mapped is always run. Bellwether skips a test only when it can name the map entry that says it is safe to.

The flakes

A test that fails and then passes on an identical commit has told you nothing. When that happens Bellwether reruns it in isolation on the same machine image. If it disagrees with itself, it stops blocking the merge and moves onto a quarantine list with the two conflicting runs attached.

Quarantine is not deletion. The test keeps running on every commit and keeps reporting. It just stops being allowed to hold a branch closed on the strength of a coin toss.

When it selects everything

Change the base image, the lockfile or the CI definition itself and every test in the repository is downstream of the change. Bellwether selects the whole suite, and the run takes exactly as long as it used to.

That case is on the home page as one of the three panels, drawn rather than described: the third panel selects all forty five and the split it can show you is nothing at all. A tool that reported a saving there would be describing the commit rather than measuring it.

Where it runs

Bellwether is a binary and a step. It does not need a runner of its own, and it does not need your source. It has been run against these:

• GitHub Actions
• GitLab CI
• Buildkite
• CircleCI
• Jenkins
• Bazel
• Nx
• Earthly

Self hosted runners are the normal case rather than an enterprise upgrade. See the install notes for the two lines this adds to a pipeline file.

The same plate, twice

Left is the file this site ships. Right is the identical original with the grade never applied, published here on purpose so the claim can be checked by looking rather than taken on trust.