Reading the skip report

The skip report is attached to every pull request as a check. It lists every test that was left out and the reason, and it is the page to read before you trust selection with anything that matters.

Asking about one test

$ bellwether why api/webhooks/retry_spec.rb
skipped  api/webhooks/retry_spec.rb
  map recorded  2 hours ago, run 8841
  loads         12 files, none of them in this diff
  nearest       packages/core/result.ts  (not loaded by this test)

The verdicts

Selected: reaches (the map puts it on a changed file), stale (its entry is older than the test), unmapped (never recorded) and pinned (an always-run rule).

Skipped: unreachable (no path from the diff), quarantined (on the flake list), excluded (your configuration) and other shard (another machine runs it).

Before you gate a release on it

• Look at a week of nightly comparisons in the dashboard. Zero stale entries is normal; one or two a week means a file the recorder cannot see, which belongs in global.
• Check that your smoke tests are in always_run.
• Keep the release branch on the full suite for the first month. Most teams switch it over after that.