# `Stampede.LLM`
[🔗](https://github.com/mandar1045/stampede/blob/v1.4.0/lib/stampede/llm.ex#L1)

A client for AI chat APIs in OpenAI's format, such as Ollama, vLLM,
llama.cpp, LM Studio or OpenAI itself, to find out how many people your
model server can chat with at once.

    {:ok, reply} =
      Stampede.LLM.chat("http://localhost:11434/v1",
        model: "qwen2.5:0.5b",
        messages: [%{role: "user", content: "Why is the sky blue?"}],
        max_tokens: 100
      )

    reply.content

The server can be anything that speaks this format. Scenarios are in
Elixir, but the server doesn't have to be.

## Metrics

Inside a run, chats record the same measures as `vllm bench serve`, so
numbers from the two compare directly:

  * `"llm.ttft"` - time to first token, from sending the request to the
    first chunk with content
  * `"llm.itl"` - inter-token latency, every gap between two chunks with
    content
  * `"llm.tpot"` - time per output token, once per request with more than
    one: the time after the first token over the tokens after the first
  * `"llm.request"` - the whole request, from start to end
  * `"llm.request.errors"` - requests that failed: a status of 400 or more,
    a stream that broke or reported an error, or no tokens at all
  * `"llm.tokens.output"` and `"llm.tokens.input"` - tokens generated and
    read. Their `:rate` is the tokens per second of the whole run.

Token counts are what the server reports, when it does. Otherwise, the
output is counted as one token per chunk, which is close for most servers,
and the input isn't counted.

Requests without streaming only record `"llm.request"`, its errors and the
token counts, as there's no first token to time.

# `reply`

```elixir
@type reply() :: %{
  content: String.t(),
  usage: %{input: non_neg_integer() | nil, output: non_neg_integer()},
  ttft: float() | nil,
  finish_reason: String.t() | nil
}
```

# `chat`

```elixir
@spec chat(String.t(), keyword()) :: {:ok, reply()} | {:error, term()}
```

Sends a chat to the API at `url` and reads the whole reply.

`url` is the API's base, such as `"http://localhost:8000/v1"`, or the full
URL ending in `/chat/completions`.

Returns `{:ok, reply}`, with the `:content`, the tokens in `:usage`, the
`:ttft` in milliseconds and the `:finish_reason`. Otherwise, returns
`{:error, reason}`: `{:status, status, body}` for a status of 400 or more,
`{:stream, error}` for an error sent in the stream, `:no_tokens` for a reply
without any, or a `Mint.TransportError` when the connection failed.

## Options

  * `:model` - the model's name, required
  * `:messages` - the conversation, required, as a list of maps with a
    `:role` and `:content`
  * `:max_tokens` - the most tokens to generate
  * `:temperature` - the sampling temperature
  * `:api_key` - sent as a bearer token
  * `:headers` - extra headers
  * `:stream` - whether to stream the reply. Defaults to `true`.
  * `:timeout` - how long the whole reply may take, in milliseconds.
    Defaults to `120_000`.
  * `:extra` - a map of other fields to send, such as vLLM's
    `%{ignore_eos: true}` for replies of a fixed length
  * `:transport_opts` - options for `:gen_tcp` or `:ssl`, given to
    `Mint.HTTP.connect/4`

# `prompts_from_jsonl`

```elixir
@spec prompts_from_jsonl(Path.t(), keyword()) :: [[map()]]
```

Reads conversations to send from a file with one JSON object on each line,
returning a list of `:messages` for `chat/2`.

Each line can have its conversation as `"messages"`, in OpenAI's format, as
`"conversations"`, in ShareGPT's format (with `"from"` and `"value"`), or as
a single `"prompt"`. With `"conversations"`, the conversation stops before
the last reply from the model, so that it's the server's turn to answer.

## Options

  * `:limit` - the most conversations to read

# `synthetic_prompt`

```elixir
@spec synthetic_prompt(pos_integer()) :: String.t()
```

Returns a prompt of random words, `words` long, for load tests where the
size of the prompt matters and what it says doesn't.

---

*Consult [api-reference.md](api-reference.md) for complete listing*