What a red build is worth once nobody believes it
A failing test is a claim about your code. A test that fails at random is a claim about nothing, and it costs more than the minutes it burns.
There is a number every engineering team knows and almost none of them write down: the share of red builds that turn out to mean nothing. Ask around a standup and you will get answers between a fifth and two thirds, delivered with total confidence and no data behind them at all.
The number matters more than the suite runtime, and it is the one nobody instruments.
The arithmetic of not believing a test
Say a quarter of your reds are noise. A developer who sees red now has a decision to make, and the cheapest correct move is to press retry. That costs a full suite run, so a quarter of your reds cost double. Fine. That is the visible cost and it is the small one.
The invisible cost is what happens to the other three quarters. A person who has learned that red sometimes means nothing does not read a red carefully. They skim it, they retry it, and when it comes back red a second time they start reading. You have inserted a full suite runtime between a real failure and the first human who looks at it, on every real failure, forever.
That is the actual price of a flaky test. Not the minutes. The habit.
Quarantine is not deletion
The usual objection to quarantining a flaky test is that it stops protecting you. It does not, and the confusion is worth being precise about.
A quarantined test in Bellwether keeps running on every commit. It keeps reporting. It appears on a list with both of its conflicting runs attached, so somebody can go and find out what is actually non deterministic about it. The only thing that changes is that it is no longer allowed to hold a branch closed on the strength of a coin toss.
A test that fails half the time was never protecting you. It was producing a signal you had already learned to ignore, which is worse than no signal, because it also taught you to ignore the ones next to it.
What to instrument first
Before buying anything, including this, measure the thing:
- Take your last two hundred failed builds.
- For each one, find whether the very next run on the same commit passed.
- Divide.
That fraction is your flake rate and it will be higher than the guess in the standup. It is also the only number that tells you whether test selection is your problem at all. If your suite is slow but honest, you need selection. If it is fast and lying to you, selection will make it lie faster.
Most teams have both, in which case the order matters: fix the lying first. A suite you believe and wait for is a nuisance. A suite you do not believe is a suite that has stopped doing its job, and it will keep not doing it at any speed.