Bellwether
Start a trial

Run the tests that can tell you something.

Bellwether picks the tests a change can actually break, and quarantines the ones that fail at random.

Start a trial See how it works
A switchgear aisle in a machine hall, cabinets running away to a lit end wall.

Selection is a partition, not a cut.

A suite of 45 tests still has 45 tests after Bellwether decides which of them a change can reach. Nothing is deleted. The set is split, and the half that cannot fail is the half you stop paying for. Scroll a panel and the split it can express is the whole payload: the third one can express none.

packages/billing/proration.ts

8 / 45 selected

Two modules import it. Nothing in search, jobs or web can see the change.

  1. api/auth/session_spec.rb
  2. api/auth/token_refresh_spec.rb
  3. api/accounts/create_spec.rb
  4. api/accounts/merge_spec.rb
  5. api/accounts/suspend_spec.rb
  6. api/webhooks/delivery_spec.rb
  7. api/webhooks/retry_spec.rb
  8. api/webhooks/signature_spec.rb
  9. api/rate_limit/bucket_spec.rb
  10. api/rate_limit/headers_spec.rb
  11. billing/proration_spec.rb
  12. billing/invoice_render_spec.rb
  13. billing/invoice_totals_spec.rb
  14. billing/coupon_stacking_spec.rb
  15. billing/tax_region_spec.rb
  16. billing/dunning_spec.rb
  17. billing/refund_partial_spec.rb
  18. billing/seat_count_spec.rb
  19. search/indexer_spec.rb
  20. search/tokenizer_spec.rb
  21. search/ranking_spec.rb
  22. search/facet_counts_spec.rb
  23. search/synonym_map_spec.rb
  24. jobs/scheduler_spec.rb
  25. jobs/dead_letter_spec.rb
  26. jobs/backoff_spec.rb
  27. jobs/idempotency_spec.rb
  28. web/checkout_flow_spec.ts
  29. web/checkout_errors_spec.ts
  30. web/settings_form_spec.ts
  31. web/settings_roles_spec.ts
  32. web/table_paging_spec.ts
  33. web/table_sorting_spec.ts
  34. web/upload_resume_spec.ts
  35. web/nav_keyboard_spec.ts
  36. core/result_spec.ts
  37. core/money_spec.ts
  38. core/clock_spec.ts
  39. core/retry_spec.ts
  40. core/uuid_spec.ts
  41. infra/migration_lock_spec.rb
  42. infra/read_replica_spec.rb
  43. infra/cache_warm_spec.rb
  44. infra/feature_flag_spec.rb
  45. infra/health_probe_spec.rb

packages/core/result.ts

20 / 45 selected

Widely imported. The selection grows because the blast radius grew, not because the run was cautious.

  1. api/auth/session_spec.rb
  2. api/auth/token_refresh_spec.rb
  3. api/accounts/create_spec.rb
  4. api/accounts/merge_spec.rb
  5. api/accounts/suspend_spec.rb
  6. api/webhooks/delivery_spec.rb
  7. api/webhooks/retry_spec.rb
  8. api/webhooks/signature_spec.rb
  9. api/rate_limit/bucket_spec.rb
  10. api/rate_limit/headers_spec.rb
  11. billing/proration_spec.rb
  12. billing/invoice_render_spec.rb
  13. billing/invoice_totals_spec.rb
  14. billing/coupon_stacking_spec.rb
  15. billing/tax_region_spec.rb
  16. billing/dunning_spec.rb
  17. billing/refund_partial_spec.rb
  18. billing/seat_count_spec.rb
  19. search/indexer_spec.rb
  20. search/tokenizer_spec.rb
  21. search/ranking_spec.rb
  22. search/facet_counts_spec.rb
  23. search/synonym_map_spec.rb
  24. jobs/scheduler_spec.rb
  25. jobs/dead_letter_spec.rb
  26. jobs/backoff_spec.rb
  27. jobs/idempotency_spec.rb
  28. web/checkout_flow_spec.ts
  29. web/checkout_errors_spec.ts
  30. web/settings_form_spec.ts
  31. web/settings_roles_spec.ts
  32. web/table_paging_spec.ts
  33. web/table_sorting_spec.ts
  34. web/upload_resume_spec.ts
  35. web/nav_keyboard_spec.ts
  36. core/result_spec.ts
  37. core/money_spec.ts
  38. core/clock_spec.ts
  39. core/retry_spec.ts
  40. core/uuid_spec.ts
  41. infra/migration_lock_spec.rb
  42. infra/read_replica_spec.rb
  43. infra/cache_warm_spec.rb
  44. infra/feature_flag_spec.rb
  45. infra/health_probe_spec.rb

Dockerfile.base

45 / 45 selected

Every test in the repo runs on this image, so there is nothing downstream to exclude. The selection is the suite.

  1. api/auth/session_spec.rb
  2. api/auth/token_refresh_spec.rb
  3. api/accounts/create_spec.rb
  4. api/accounts/merge_spec.rb
  5. api/accounts/suspend_spec.rb
  6. api/webhooks/delivery_spec.rb
  7. api/webhooks/retry_spec.rb
  8. api/webhooks/signature_spec.rb
  9. api/rate_limit/bucket_spec.rb
  10. api/rate_limit/headers_spec.rb
  11. billing/proration_spec.rb
  12. billing/invoice_render_spec.rb
  13. billing/invoice_totals_spec.rb
  14. billing/coupon_stacking_spec.rb
  15. billing/tax_region_spec.rb
  16. billing/dunning_spec.rb
  17. billing/refund_partial_spec.rb
  18. billing/seat_count_spec.rb
  19. search/indexer_spec.rb
  20. search/tokenizer_spec.rb
  21. search/ranking_spec.rb
  22. search/facet_counts_spec.rb
  23. search/synonym_map_spec.rb
  24. jobs/scheduler_spec.rb
  25. jobs/dead_letter_spec.rb
  26. jobs/backoff_spec.rb
  27. jobs/idempotency_spec.rb
  28. web/checkout_flow_spec.ts
  29. web/checkout_errors_spec.ts
  30. web/settings_form_spec.ts
  31. web/settings_roles_spec.ts
  32. web/table_paging_spec.ts
  33. web/table_sorting_spec.ts
  34. web/upload_resume_spec.ts
  35. web/nav_keyboard_spec.ts
  36. core/result_spec.ts
  37. core/money_spec.ts
  38. core/clock_spec.ts
  39. core/retry_spec.ts
  40. core/uuid_spec.ts
  41. infra/migration_lock_spec.rb
  42. infra/read_replica_spec.rb
  43. infra/cache_warm_spec.rb
  44. infra/feature_flag_spec.rb
  45. infra/health_probe_spec.rb

Three things, and it stops there.

  1. Record

    Bellwether watches one full suite run and records which files each test actually loaded. No annotations, no test tagging, no config file listing your modules.

  2. Select

    On the next commit it intersects the diff with that map. Tests that cannot reach a changed file are not run. Tests that can are run first.

  3. Quarantine

    A test that passes and fails on the same commit is not a signal. Bellwether reruns it in isolation, and if it disagrees with itself, it stops blocking the merge and starts appearing on a list.

What that came to on three commits.

The demo repository below is a sample, not a customer. It carries 45 test files and the three commits are the three on this page.

Sample repository, three consecutive commits
Commit Changed Selected Skipped Share of suite
a4f19c2 packages/billing/proration.ts 8 37 18%
7be0d31 packages/core/result.ts 20 25 44%
e02c8aa Dockerfile.base 45 0 100%
Across the three 73 62 54%

The third row is the one worth reading. When a change is upstream of everything, Bellwether selects everything, and a tool that claimed a saving there would be lying about the commit rather than measuring it.

We had already accepted a 40 minute suite. What we had not accepted was that half of the reds were noise, and that is the part the team had quietly stopped reading.

Priya Raghunathan Platform lead, Hartsmere Logistics

$40

per developer, per month

Billed yearly. No seat minimum. One plan. Everything Bellwether does, it does on that plan, because a selection tool that withholds selection from the cheaper tier is not selling you anything.

Start a trial

Bring us your worst suite. We will tell you what we can skip before you pay for anything.

020 7946 0347

[email protected]

Or send the details and we will read them before we reply.

Rows of cabinets in a plant room, each door carrying its own labelled switchgear.