mix stampede.compare (stampede v1.4.0)

Copy Markdown View Source

Compares the timings of two runs, from files written by mix stampede.run --out.

$ mix stampede.compare main.json branch.json
$ mix stampede.compare main.json branch.json --max-regression 10

For every scenario and timing in both files, prints the p50, p95 and p99 of each run and how much they changed.

Options

  • --max-regression - how many percent slower a percentile may get. The task exits with status 1 when one gets slower than that, so it can gate CI.
  • --format - text, the default, or markdown, to post the comparison as a comment on a pull request, such as with gh pr comment --body-file

Keep in mind that timings vary from one run to the next, even with no change to the code, so leave some room. Runs of a minute or more on the same machine compare best.