A client for AI chat APIs in OpenAI's format, such as Ollama, vLLM, llama.cpp, LM Studio or OpenAI itself, to find out how many people your model server can chat with at once.
{:ok, reply} =
Stampede.LLM.chat("http://localhost:11434/v1",
model: "qwen2.5:0.5b",
messages: [%{role: "user", content: "Why is the sky blue?"}],
max_tokens: 100
)
reply.contentThe server can be anything that speaks this format. Scenarios are in Elixir, but the server doesn't have to be.
Metrics
Inside a run, chats record the same measures as vllm bench serve, so
numbers from the two compare directly:
"llm.ttft"- time to first token, from sending the request to the first chunk with content"llm.itl"- inter-token latency, every gap between two chunks with content"llm.tpot"- time per output token, once per request with more than one: the time after the first token over the tokens after the first"llm.request"- the whole request, from start to end"llm.request.errors"- requests that failed: a status of 400 or more, a stream that broke or reported an error, or no tokens at all"llm.tokens.output"and"llm.tokens.input"- tokens generated and read. Their:rateis the tokens per second of the whole run.
Token counts are what the server reports, when it does. Otherwise, the output is counted as one token per chunk, which is close for most servers, and the input isn't counted.
Requests without streaming only record "llm.request", its errors and the
token counts, as there's no first token to time.
Summary
Functions
Sends a chat to the API at url and reads the whole reply.
Reads conversations to send from a file with one JSON object on each line,
returning a list of :messages for chat/2.
Returns a prompt of random words, words long, for load tests where the
size of the prompt matters and what it says doesn't.
Types
@type reply() :: %{ content: String.t(), usage: %{input: non_neg_integer() | nil, output: non_neg_integer()}, ttft: float() | nil, finish_reason: String.t() | nil }
Functions
Sends a chat to the API at url and reads the whole reply.
url is the API's base, such as "http://localhost:8000/v1", or the full
URL ending in /chat/completions.
Returns {:ok, reply}, with the :content, the tokens in :usage, the
:ttft in milliseconds and the :finish_reason. Otherwise, returns
{:error, reason}: {:status, status, body} for a status of 400 or more,
{:stream, error} for an error sent in the stream, :no_tokens for a reply
without any, or a Mint.TransportError when the connection failed.
Options
:model- the model's name, required:messages- the conversation, required, as a list of maps with a:roleand:content:max_tokens- the most tokens to generate:temperature- the sampling temperature:api_key- sent as a bearer token:headers- extra headers:stream- whether to stream the reply. Defaults totrue.:timeout- how long the whole reply may take, in milliseconds. Defaults to120_000.:extra- a map of other fields to send, such as vLLM's%{ignore_eos: true}for replies of a fixed length:transport_opts- options for:gen_tcpor:ssl, given toMint.HTTP.connect/4
Reads conversations to send from a file with one JSON object on each line,
returning a list of :messages for chat/2.
Each line can have its conversation as "messages", in OpenAI's format, as
"conversations", in ShareGPT's format (with "from" and "value"), or as
a single "prompt". With "conversations", the conversation stops before
the last reply from the model, so that it's the server's turn to answer.
Options
:limit- the most conversations to read
@spec synthetic_prompt(pos_integer()) :: String.t()
Returns a prompt of random words, words long, for load tests where the
size of the prompt matters and what it says doesn't.