Seeing inside your app

A load test measures your app from the outside. It can tell you that joins got slow at 40,000 users, but not whether that was the CPU, the database or one overloaded process. With --probe, Stampede watches the app from the inside during the test, and puts both views in the same report.

Your app doesn't need any changes or dependencies. All it takes is a running BEAM node that Stampede can connect to.

Connecting to your app

Start the app with a node name and a cookie, then run the test from a node with the same cookie:

$ elixir --name app@127.0.0.1 --cookie secret -S mix phx.server

$ elixir --name loader@127.0.0.1 --cookie secret -S mix stampede.run loadtest/lobby.exs --probe app@127.0.0.1

From code, it's an option, with one node or a list of them:

Stampede.run(MyApp.Load.Lobby, probe: :"app@127.0.0.1")

A release gets its name and cookie from the environment:

$ RELEASE_DISTRIBUTION=name RELEASE_NODE=app@10.0.0.5 RELEASE_COOKIE=secret bin/my_app start

Nodes find each other through epmd on port 4369, then talk on a port the VM picks at random. Behind a firewall or in a container, fix those ports and open them to the load test, as in the guide to spreading the load.

A node that knows the cookie can run any code on your app, so only do this on a private network, with a long random cookie. If you're probing production, be sure that's what you want.

What it does to your app

While the run lasts, a small agent runs on the app's node:

• Stampede loads the agent's code onto the node, and removes it afterwards.
• It turns on the VM's scheduler statistics, and attaches telemetry handlers for Phoenix, LiveView and any Ecto repos it finds.
• Every second, it reads the node's numbers. Every 10 seconds, it looks for long message queues.

If the load test crashes or the network goes away, the agent stops by itself.

Watching costs a little. Against one Phoenix server holding hundreds of thousands of connections, the server's CPU was about 6 points higher with the probe on. If you need the exact maximum, run it once without.

What you get

The summary gets a server section with the node at its busiest, and the dashboard and the HTML report chart it second by second:

  server
    app@10.0.0.5 at its busiest: schedulers 87% busy, 27.85 GB of memory, 1,200,484 processes, run queue 717

And there are new metrics, timed by the app itself:

They work like any other metric, so you can set thresholds on them:

thresholds: [{"server.ecto.queue", p95: 50}]

Your database

When your app has an Ecto repo, the probe also sends how long queries waited for a connection from the pool, every second. When that repo uses Postgres, it reads Postgres's own statistics every second too: commits and rows written per second, rows read, the cache hit rate, connections, and what active queries are waiting on.

You don't give it a database URL. The agent uses your repo's own settings to open one connection of its own, outside your app's pool, so it doesn't take a connection your app needs. It's closed when the run ends.

The summary gets two more lines, and the dashboard and the HTML report chart the pool wait, commits and writes per second, and active queries:

    database at its busiest: 2,454 commits/s, 1,156 rows written/s, 34,000 rows read/s, 11 connections
    database pool of 10: queries waited up to 300.00ms for a connection

When a --find-limit step fails, its verdict says what the database was doing, from what it measured:

  limit
    Held 250 VUs. At 350 VUs, iteration.errors count was 64, over 0. Queries waited up to
    171.94ms for one of the database pool's 2 connections. Postgres was mostly busy running
    queries. The app's schedulers were only 16% busy, so the database was the limit, not the app.

It names the wait Postgres spent most of its time on, such as WALWrite, writing its log to disk before each commit, or DataFileRead, reading table data from disk, or locks held by other queries. It only says so when queries were really waiting on it, and pool waits under 50ms aren't mentioned.

The probe connects as your app's database user, so it sees what your app's queries are waiting on. Sessions of other database users only show their waits to a superuser or a member of pg_read_all_stats. Databases other than Postgres only get the pool wait for now.

In LiveDashboard

If your app has Phoenix LiveDashboard, it can show the run too. Add Stampede to the app's dependencies, in the environments where you load test it, and its page to the dashboard's route:

{:stampede, "~> 1.4", only: [:dev, :test]}

live_dashboard "/dashboard",
  additional_pages: [stampede: Stampede.LiveDashboard]

A run that probes the app shows in the dashboard's Stampede tab while it runs, with the same charts and server numbers as Stampede's own dashboard. When the run is over, it stays there with its full results, even after the load test's node goes away. The tab keeps the last 20 runs.

That puts the load next to everything else the dashboard shows. Watch the Processes page sorted by message queue while the users climb, and you can see which process falls behind first.

Finding the bottleneck

When a --find-limit step fails, its verdict says if the server's schedulers ran out, if the server went down, or if the database pool or Postgres was the limit, and warns you if the load test was the one too busy to trust.

Past 20,000 processes, the agent only checks the named processes for long queues, plus a random sample of a thousand others. Asking a process for its queue wakes it, and waking every connection's process would change what you're measuring. Most real bottlenecks are named anyway, such as a PubSub or a pool.

If your app can't accept a distribution connection at all, --probe won't work for now. That's on the roadmap.