How to split tests across CI shards evenly

1. Get per-test timings as JUnit XML

Every mainstream runner can write JUnit XML with a time on each test case. Run the whole suite once, on one machine, with it switched on:

pytest --junitxml=junit.xml
JEST_JUNIT_ADD_FILE_ATTRIBUTE=true jest --reporters=default --reporters=jest-junit
gotestsum --junitfile junit.xml ./...
mvn test            # reports in target/surefire-reports/TEST-*.xml
gradle test         # reports in build/test-results/test/*.xml
bundle exec rspec --format RspecJunitFormatter --out junit.xml

For Jest, the file attribute matters: without it the report has test names but no file path, and a shard command needs file paths. Option names are from each project's own documentation as best understood when this page was written; check them against your versions.

2. Balance by duration, not by count

Splitting 24 files into four groups of six looks fair, but if one file takes ninety seconds and most take five, the group holding it is slow. Putting tests on shards so the slowest shard is as short as possible is the multiway number-partitioning problem. It is hard to solve exactly in general, but a simple rule gets close: take the longest file, put it on the least-loaded shard, repeat. That is the LPT rule, and it is never worse than about a third above the best possible split. A pass of moving and swapping files between shards then evens out what is left, and for small suites the best split can be found exactly.

The CI Test Shard Balancer does all of that from the XML in your browser, and prints the file list for each shard.

3. Look for the test that cannot be split

Total time divided by the number of shards is the ideal shard. If one file takes longer than that, it sets the floor: no split can finish before it does, however many shards you add. The tool flags these. If the file holds many tests, group by single test instead; if it is one long test, that test is the thing to speed up or split.

4. Pick the number of shards from setup time

Every shard job pays a fixed cost before its first test: checkout, dependency restore, build. With 60 seconds of setup and a 10-minute suite, going from 4 to 8 shards cuts the test part from 2.5 minutes to 1.25 but adds four more setups. Wall-clock time is the slowest shard plus setup, and billed time is roughly shards times that, so there is a shard count past which you pay more and wait about the same. Enter your setup seconds and an optional wall-clock target and the tool prices every count in a table.

5. Put it in CI, and keep it fresh

Paste the emitted commands, or use the GitHub Actions matrix JSON with fromJSON in a job's strategy.matrix. Timings drift as tests are added, so recompute every few weeks, or let CI tell you:

node cli.mjs split junit.xml --shards 4 --max-imbalance-pct 20

That exits 1 when the slowest shard is more than 20 percent over an even share. Timings also vary from run to run, so treat the result as a good starting point rather than a promise. If you want a split that adapts during every run, a paid service does that; see the comparison.

Flaky tests distort timings because reruns inflate them. Rule those out with the Flaky Test Finder first.

Common questions

Why is one of my CI shards always the slowest?

Most simple splits give each shard the same number of test files in alphabetical or hashed order. Test files differ a lot in duration, so shards with the slow files finish last and the others sit idle. Balancing by measured duration fixes that part; a single very slow file can still set the floor.

Do more shards always make CI faster?

No. Each shard pays a fixed setup cost, and wall-clock time can never go below the longest single test or file plus setup. Past that point more shards only add billed minutes.

How often should I recompute the split?

Whenever the suite changes a lot, and otherwise every few weeks. A CI step that runs the CLI with --max-imbalance-pct tells you when the split has drifted.