Stampede (stampede v0.2.0)

Copy Markdown View Source

Load testing for Elixir and Phoenix applications.

A load test is a module that implements Stampede.Scenario. Stampede starts virtual users (VUs), each one a process that calls the scenario's Stampede.Scenario.run/1 over and over, moves the number of running VUs through the stages you configured, and records how long things take.

defmodule MyApp.Load.Lobby do
  use Stampede.Scenario

  @impl true
  def options do
    [
      stages: [{[second: 30], 1_000}, {[minute: 1], 1_000}, {[second: 10], 0}],
      thresholds: [{"channel.join", p95: 250}, {"iteration.errors", count: 0}]
    ]
  end

  @impl true
  def run(_vu) do
    {:ok, socket} = Stampede.Channels.connect("ws://localhost:4000/socket")
    {:ok, _reply} = Stampede.Channels.join(socket, "room:lobby")
    {:ok, _reply} = Stampede.Channels.push(socket, "room:lobby", "ping")
    Process.sleep(1_000)
  end
end

Run it with mix stampede.run, or from code:

result = Stampede.run(MyApp.Load.Lobby)
IO.puts(Stampede.Reporter.format(result))

Metrics

Every metric has a name and is a timing, a counter or a check. Stampede records these on its own:

  • "iteration" - how long each successful Stampede.Scenario.run/1 took
  • "iteration.errors" - iterations that raised, threw or exited
  • "iterations.dropped" - iterations a :rate run could not start because all of its :max_vus were busy
  • "vu.crashes" - VUs that crashed and had to be replaced
  • the request metrics listed in Stampede.HTTP
  • the connection and channel metrics listed in Stampede.Channels
  • the page, join and event metrics listed in Stampede.LiveView

Record your own with measure/2, record/3, increment/2 and check/2. Timings, counters and checks share one set of names.

Timings are counted in buckets that grow by 2% each, so every percentile in a Stampede.Result is within 1% of the exact value, however many samples the run collected.

Telemetry

Stampede emits these :telemetry events:

  • [:stampede, :run, :start] - with the :system_time as measurement, and the :scenario and its :options as metadata
  • [:stampede, :run, :interval] - every second, with the number of busy :vus as measurement, and the :scenario and the :interval, a point of Stampede.Result's timeline, as metadata
  • [:stampede, :run, :stop] - with the :duration as measurement, and the :scenario and its :result as metadata
  • [:stampede, :timing] - for every timing recorded, with its :duration as measurement and its :name as metadata
  • [:stampede, :counter] - for every counter incremented, with the :count added as measurement and its :name as metadata
  • [:stampede, :check] - for every check, with a :count of 1 as measurement, and its :name and whether it :passed? as metadata

Durations are in native time units, as usual for telemetry. The run events are emitted where run/2 was called, the others on the node that recorded them.

Summary

Functions

check(name, condition)

Records whether condition holds as the check name, and returns it as a boolean.

increment(name, amount \\ 1)

Adds amount to the counter name.

measure(name, fun)

Calls fun and records how long it took as the timing name.

record(name, duration, unit \\ :millisecond)

Records duration, given in unit, as the timing name.

run(scenario, opts \\ [])

Runs scenario and returns a Stampede.Result.

Functions

check(name, condition)

@spec check(String.t(), as_boolean(term())) :: boolean()

Records whether condition holds as the check name, and returns it as a boolean.

Unlike a failed match, a failed check does not end the iteration. The summary shows how often each check passed, and thresholds can limit how many fail:

Stampede.check("status is 200", response.status == 200)

thresholds: [{"status is 200", failed_rate: 0.01}]

increment(name, amount \\ 1)

@spec increment(String.t(), pos_integer()) :: :ok

Adds amount to the counter name.

measure(name, fun)

@spec measure(String.t(), (-> result)) :: result when result: var

Calls fun and records how long it took as the timing name.

Returns what fun returns. Nothing is recorded if fun raises, or when it is called outside of a run.

Stampede.measure("checkout", fn -> MyApp.Client.checkout(cart) end)

record(name, duration, unit \\ :millisecond)

@spec record(String.t(), non_neg_integer(), System.time_unit()) :: :ok

Records duration, given in unit, as the timing name.

Useful when the duration comes from somewhere else, such as a timestamp sent back by the server.

run(scenario, opts \\ [])

@spec run(module(), keyword()) :: Stampede.Result.t()

Runs scenario and returns a Stampede.Result.

Options given here take precedence over the ones returned by the scenario's Stampede.Scenario.options/0.

Options

  • :stages - a list of {duration, target} tuples. Each stage moves the number of running VUs in a straight line from the previous target, starting at zero, to target over duration. A stage with no duration jumps to its target right away.

  • :vus and :duration - run a constant number of VUs, 1 by default, for duration instead of going through stages. vus: 100, duration: [minute: 1] is the same as stages: [{0, 100}, {[minute: 1], 100}].

  • :rate and :duration - start rate iterations per second for duration, however long they take, instead of keeping a number of VUs busy. Unlike VUs that wait for their last iteration to finish, a rate keeps the pressure on a server that slows down, which is how real traffic behaves. Rates can be fractions, such as 0.5.

  • :rate_stages - like :stages, but each target is a rate.

  • :max_vus - how many iterations a :rate run may have in progress at once. VUs are started as needed and reused once they finish an iteration. When all of them are busy, iterations are dropped and counted as "iterations.dropped". Defaults to 1_000.

  • :thresholds - a list of {metric, limits} tuples, such as {"channel.join", p95: 250, max: 1_000}. Each limit is the highest value the statistic may have for the run to pass. Timings have :min, :mean, :p50, :p90, :p95, :p99 and :max, in milliseconds. Checks have :failed and :failed_rate, from 0 to 1. Every metric has :count and :rate, per second. Thresholds on a timing with no samples fail, while counters and checks with no samples are zero.

  • :nodes - the nodes to run the VUs on, [node()] by default. VUs and rates are split evenly between them, and their metrics are merged. Every node must be connected and able to run Stampede and the scenario's code, for example because it was started from the same project. A scenario compiled to a .beam file is loaded on the nodes that miss it.

  • :progress - prints the number of VUs, iterations and errors every second. Defaults to false.

Durations are milliseconds or anything else Kernel.to_timeout/1 accepts, such as [second: 30] or Duration.new!(minute: 1).