Bellwether

From $24 per committer / month

Shard balancing

The selected tests are split across your parallel machines by each test’s recorded duration, so every shard finishes within seconds of the others. No shard lists to maintain.

Balanced by
Recorded duration, last 20 runs
Typical spread
Shards finish within 40 seconds
Machines
2 to 256 per run
Setup
One flag: --shard 3/8
Rows of server blades with green status lights, shot from below
$ bellwether select --shards 4
shard 1/4   103 tests   1m 38s
shard 2/4   104 tests   1m 41s
shard 3/4   102 tests   1m 36s
shard 4/4   103 tests   1m 39s

Splitting a suite across machines is the usual fix for a slow pipeline, and it works until the shard lists drift. Someone adds forty slow integration tests to shard two, shard two becomes the one everybody waits for, and a person ends up owning a spreadsheet of test files.

Splitting by time, not by count

Bellwether already knows how long every test took on its last twenty runs. When it writes the selection it splits it across however many machines you asked for, so that each shard carries about the same total duration. Pass --shard 3/8 on machine three and it reads its own share.

Selection first, then balance

The balancing happens after selection, which is the order that matters. A four-way split of the whole suite still makes every machine run tests the change cannot reach. A four-way split of the selected tests finishes in a fraction of the time and bills a fraction of the minutes.

When a shard is slow anyway

If one shard runs more than twice as long as the median, the run summary names the test that caused it and the time it took against its recorded average. That is usually a test waiting on a network call or a lock, and it is worth a look whether or not you use Bellwether.

The other three