Load tests in CI

mix stampede.run exits with status 1 when a threshold fails, so a load test can fail a build like any other test. Two more things make it useful in CI: keeping the HTML report of every run, and comparing each change with the branch it goes into.

Comparing with the base branch

Timings depend on the machine, and CI machines vary from one job to the next. Comparing a branch with numbers recorded elsewhere would mostly measure that. Running both branches one after the other in the same job compares them on the same machine.

A script that starts the app, waits for it, runs the load test and stops the app again keeps the workflow short:

#!/bin/sh
# loadtest/ci.sh - runs the load test against a fresh server, writing the
# results to the file given first, with any other options given to the task.
set -e

out=$1
shift

mix deps.get
mix ecto.migrate
mix phx.server > server.log 2>&1 &
server=$!
trap 'kill $server' EXIT

until curl -sf -o /dev/null http://localhost:4000/; do sleep 1; done
mix stampede.run loadtest/checkout.exs --out "$out" "$@"

Then, in .github/workflows/load.yml:

name: Load test

on:
  pull_request:

jobs:
  load:
    runs-on: ubuntu-24.04

    # To comment on the pull request with the comparison.
    permissions:
      contents: read
      pull-requests: write

    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_PASSWORD: postgres
        ports: ["5432:5432"]
        options: >-
          --health-cmd pg_isready --health-interval 5s
          --health-timeout 5s --health-retries 10

    steps:
      - uses: actions/checkout@v5
        with:
          fetch-depth: 0

      - uses: erlef/setup-beam@v1
        with:
          elixir-version: "1.19"
          otp-version: "28"

      - run: mix deps.get && mix ecto.create

      # A threshold that already fails on the base branch should not stop
      # the comparison.
      - name: Load test the base branch
        run: |
          git checkout ${{ github.event.pull_request.base.sha }}
          sh loadtest/ci.sh base.json || true

      - name: Load test this branch
        run: |
          git checkout ${{ github.event.pull_request.head.sha }}
          sh loadtest/ci.sh head.json --html report.html

      - name: Compare with the base branch
        if: hashFiles('base.json') != ''
        run: mix stampede.compare base.json head.json --max-regression 20 --format markdown > comparison.md

      - name: Comment on the pull request
        if: always() && hashFiles('comparison.md') != ''
        run: gh pr comment ${{ github.event.pull_request.number }} --body-file comparison.md
        env:
          GH_TOKEN: ${{ github.token }}

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: load-test-report
          path: report.html

The job fails when a threshold fails on the branch, or when a p50, p95 or p99 got more than 20% slower than on the base branch. Either way, the comparison shows up as a comment on the pull request, with the slower percentiles in bold, and the report is kept with the run, to see where the time went. Pull requests from forks get a read-only token, so there the comment step fails and the comparison is only in the job's log.

Keeping the numbers steady

• Run for a minute or more. Short runs are mostly the first requests, which are slower while the server warms up.
• Keep the load modest. The app and the load test share the CI machine, so a load test that takes all of its CPU slows down the app it measures.
• Leave room in --max-regression. Even on one machine, timings move a few percent from one run to the next with no change to the code. 20% is a forgiving place to start; tighten it once you know how much your runs vary.
• The script runs the app in the dev environment, which is slower than prod but slower on both branches alike. Run it with MIX_ENV=prod when your CI can build a release-like environment, to get numbers closer to production.

With ExUnit

Load tests written with Stampede.ExUnit fail the build the same way. Run them with mix test --only load while the script's server is up. The server still runs on its own, so the load tests measure the app rather than the test sandbox, and the test environment needs its database, as for your other tests. mix stampede.compare reads the files that mix stampede.run --out writes, so keep mix stampede.run for comparing branches.

The first load test

The first pull request that adds a load test has nothing to compare with, since the base branch doesn't have it yet. No base.json is written then, and the comparison is skipped. Likewise, mix stampede.compare skips the scenarios that only one of the files has.