Load testing AI chat APIs

Everyone shipping a chatbot ends up asking the same thing: how many people can talk to it at once before it feels slow? Stampede.LLM answers that for any server with an OpenAI-style chat API, such as Ollama, vLLM, llama.cpp or LM Studio. The server can be written in anything. Only the load test is in Elixir, and it fits in one file. I've run it against Ollama so far, so if you try it on something else, I'd like to hear how it went.

Stampede.LLM is new in 1.3, so it may still change a little.

A first test

You don't need a project, just Elixir. Save this as chat.exs, and change the URL and the model to yours:

Mix.install([{:stampede, "~> 1.4"}])

defmodule Chat do
  use Stampede.Scenario

  @impl true
  def options do
    [
      find_limit: [step: 2, up_to: 40, hold: [second: 60]],
      thresholds: [
        {"llm.ttft", p95: 2_000},
        {"llm.itl", p95: 200},
        {"iteration.errors", count: 0}
      ]
    ]
  end

  @impl true
  def run(_vu) do
    {:ok, _reply} =
      Stampede.LLM.chat("http://localhost:11434/v1",
        model: "qwen2.5:0.5b",
        messages: [%{role: "user", content: "Why is the sky blue?"}],
        max_tokens: 128
      )

    # Reads the answer and thinks, as a person would.
    Process.sleep(Enum.random(5_000..10_000))
  end
end

result = Stampede.run(Chat, progress: true)
IO.puts(Stampede.Reporter.format(result))

Then run it:

$ elixir chat.exs

It adds two users at a time, holds each step for a minute, and stops at the first step that breaks a threshold. Here, that's the first token taking over 2 seconds, tokens more than 200ms apart, or any error. The summary tells you the most users that held.

In a Mix project, mix stampede.gen.scenario Chat --llm writes a scenario to start from, and everything else in Stampede works too: the live dashboard, HTML reports, thresholds that fail your CI, and spreading the load over machines.

What it measures

Stampede.LLM measures these the same way as vllm bench serve, so numbers from the two compare directly:

Use realistic prompts

Model servers reuse the work of a prompt they've just seen, so sending the same question over and over makes them look much faster than they are. In my first try, a repeated question got its first token in 12ms instead of about a second. Real people don't all send the same words, so make prompts differ at the start:

question = Enum.random(["Why is the sky blue?", "How do bridges stay up?"])
messages = [%{role: "user", content: "Ticket #{:rand.uniform(1_000_000)}: #{question}"}]

To send real conversations from a benchmark dataset, read them from a file once, in Stampede.Scenario.setup/1. Each line of the file can have "messages", ShareGPT's "conversations" or a "prompt":

def setup(_opts), do: Stampede.LLM.prompts_from_jsonl("prompts.jsonl", limit: 1_000)

def run(vu) do
  {:ok, _reply} = Stampede.LLM.chat(@url, model: @model, messages: Enum.random(vu.data))
end

When what matters is how long the prompt is, not what it says, Stampede.LLM.synthetic_prompt(500) makes one of 500 words.

A user that keeps talking sends the whole conversation each time, which is how prompts get long in real chats:

def run(_vu) do
  first = [%{role: "user", content: "Tell me about Lisbon."}]
  {:ok, reply} = Stampede.LLM.chat(@url, model: @model, messages: first)

  Process.sleep(5_000)

  next = first ++ [%{role: "assistant", content: reply.content}, %{role: "user", content: "Where should I eat?"}]
  {:ok, _reply} = Stampede.LLM.chat(@url, model: @model, messages: next)
end

Some servers can be told to generate a fixed number of tokens, which makes runs easier to compare. On vLLM, send extra: %{ignore_eos: true} along with max_tokens. For hosted APIs, :api_key sends a bearer token. Careful there: a load test spends real tokens, and real money.

Reading the result

The two timings tell you different things:

• The first token gets slower, but the gaps between tokens don't. Requests are queuing. Once one starts, it runs at full speed, so the server has too few slots, or too few replicas.
• The gaps between tokens get slower too. The machine itself is running out of compute, because the requests in progress are sharing it.

Watch the load test as well. The summary warns you when the load test itself is too busy to trust. With AI servers that's rare, since nearly all the work is on the model's side.

A real run

Here's how that looked on AWS: Ollama 0.40 on an 8-core Graviton machine with 16 GB and no GPU, serving the 397 MB qwen2.5:0.5b model, with Stampede on a second machine. Each user asked a different question, read up to 128 tokens and thought for 5 to 10 seconds. A step passed with the first token under 2 seconds and tokens under 200ms apart at p95, with no errors.

With Ollama's default settings:

Tokens stayed 8ms apart the whole way, while the first token kept getting later. That's the queue from the last section, and Ollama's log explained it: OLLAMA_NUM_PARALLEL is 1 by default, so it answers one person at a time and everyone else waits. So I changed that one setting to 4 and ran it again:

Twice the users. Each reply streams a bit slower, since four share the CPU, but 20ms a token is still far faster than anyone reads. Total throughput went from about 70 to 125 tokens a second, and at 14 users the CPU was at 98%, so that's the hardware's real limit.

A small model on CPU says little about a big model on GPUs, so don't read these as numbers for your setup. Run it against yours. Everything from these two runs is in bench/million/results, and the scripts that set it up are ollama.sh and run-llm.sh in bench/million/aws.