Hand-balanced shards drift, and nobody notices until Thursday
Splitting a suite across machines is the right first fix for a slow pipeline. Here is why the shard lists decay, and what to do before one person ends up owning them.
Ruth Okonkwo-Vance · · 6 min read
When a pipeline gets slow, the first fix is almost always the right one: run the suite on more machines at once. Four machines, a quarter of the wait. It works on the day you set it up, and it starts decaying the day after.
Why the lists drift
A shard list is a snapshot of how long each test took on the day someone balanced it. Tests do not hold still. A team adds forty integration tests to the checkout directory, which happens to live on shard two. A fixture gets slower. Someone splits one large test file into three and they land on the same machine.
None of this is visible from the pipeline summary. The run still passes, it just waits on shard two for eight minutes after the others have finished. Everyone notices on the Thursday before a release, when the queue is long and eight minutes times forty merges is an afternoon.
Balance by time, not by file count
The fix everybody reaches for next is to split by file count, which is better than a hand-written list and still wrong. Test files are not the same size. What you want is each shard carrying the same total duration, and for that you need recorded durations for every test, refreshed every run.
That is a small piece of bookkeeping, and it is worth doing whether or not you ever buy a tool for it: write each test’s duration to a file on every full run, and split by the last twenty runs’ median.
Selection before balance
Sharding shortens the wait. It does not reduce the work. Four machines running the whole suite bill the same minutes as one machine running it, plus four lots of setup.
If most of the suite cannot fail on most commits, the bigger saving is to stop running those tests before you split what is left. A four-way split of the selected tests is often faster than a sixteen-way split of the whole suite, and it bills a fraction of the minutes. We built shard balancing to run in that order, and it is the order we would recommend with any tooling.
A checklist for this week
- Look at your last fifty pipeline runs and note the spread between the fastest and slowest shard. More than a minute is drift.
- Find who last edited the shard list. If it is the same person every time, that is a job nobody agreed to give them.
- Record durations per test on every full run, even if you do nothing else with them yet.
