Free on public repositories

Flaky test quarantine

A test that fails and then passes on the same commit is rerun in isolation. If it disagrees with itself, it stops blocking merges and moves to a quarantine list with both runs attached.

Start free Quickstart

Isolation reruns: Up to 3, same machine image
Quarantined tests: Still run on every commit
Evidence kept: Both conflicting runs, logs and seed
Release: Automatic after 50 clean runs

Close-up of dark keyboard keys D, F and G with dust on the caps

$ bellwether flakes
quarantined  3 tests
  web/checkout_flow_spec.ts   4 of 50 runs red
  jobs/scheduler_spec.rb      2 of 50 runs red
  search/ranking_spec.rb      1 of 50 runs red
blocking     none

A red build is a claim about your code. A test that goes red at random is a claim about nothing, and it teaches everyone who sees it to press retry before they read. That habit costs more than the minutes: it puts a full rerun between every real failure and the first person who looks at it.

What happens when a test flips

When a test fails and then passes on an identical commit, Bellwether reruns it in isolation on the same machine image, up to three times. If it disagrees with itself, it is quarantined: it keeps running on every commit and keeps reporting, but it can no longer hold a branch closed on the strength of a coin toss.

Quarantine is not deletion

Every quarantined test sits on a list with both of its conflicting runs attached: the logs, the random seed, the machine it ran on and how long it took. That is usually enough for someone to find what is non-deterministic about it, whether that is a clock, an ordering assumption or a shared fixture.

Getting out of quarantine

A quarantined test that passes 50 runs in a row is released automatically and starts blocking merges again. You can release or pin one by hand from the dashboard or with bellwether flakes release <test>.

The other three

Hands typing on a dark laptop keyboard in a dim room

Free for open source

Test selection

On every pull request, Bellwether walks the diff against a recorded map and runs only the tests that can reach a changed file. Median suites drop to 13 percent of their size.

How it works

Two monitors with code on a desk at night, a hand on the keyboard in the background

Free for open source

The skip report

Every test Bellwether leaves out is listed with the reason it was safe to leave out. Ask about any test on any commit and get one of eight plain answers.

How it works

Rows of server blades with green status lights, shot from below

Team, from $24 a seat

Shard balancing

The selected tests are split across your parallel machines by each test’s recorded duration, so every shard finishes within seconds of the others. No shard lists to maintain.

How it works