Charts · 31

Where an LLM call spends its time

A call to a language model is two waits with different shapes. First nothing: the request arrives, waits its turn, and the model reads the whole prompt before it can produce a word. Then the answer, one token at a time. Measured on my own machine, a short question had its first token in 165 ms and its last after 4.6 seconds. Behind a spinner the user waits the whole 4.6 seconds and it looks broken; streamed, they are reading within a fifth of a second.

The call

First token after 165 ms, the whole answer after 4.6 s. Behind a spinner, that is 4.6 s of nothing.

A one-line question, about 50 prompt tokens. 11 calls; this is the median one by time to first token. 115 tokens out at 25.6 a second.

The whole call, 4.6 sThe wait before the first token, 165 ms, zoomed

Hover a phase

Everything before the marker is waiting; everything after it is the answer arriving, one tick per streamed chunk at the moment it reached the client. Switch to the long document and the wait grows, because the model has to read it all first. Send it again and the wait nearly disappears.

0.0 s

Behind a spinner

Press replay.

Streamed

Press replay.
Medians per condition, milliseconds
CallCallsPromptQueueLoadPrefillFirst tokenStreamingTotalTokens/s
A short question11501191421654,4404,60423.8
A long document83,2973373,4283,4723,0206,48835.0
The same document again53,2982815501073,4133,50632.8
A cold start33,2974239,5353,36739,6993,20144,97834.4

Notes

Measured on 19 October 2026 with scripts/labs/llm_latency.py: 28 streamed chat calls to qwen3.5:2b (Qwen 3.5, 2 billion parameters) through Ollama on a Mac mini with an Apple M4, 16 GB, client and server on the same machine, temperature 0. The client timestamps the connection, the request, and every streamed chunk as it arrives; Ollama’s final message adds the server’s own split into model load, prompt evaluation (prefill) and generation, and queue is what is left of the server’s total after those three. Each short and long prompt began with a different request number so no call could reuse another’s cached prompt; the cached condition sent one long prompt again on purpose, and its prefill fell from 3.4 seconds to 50 ms. Medians are shown because the machine is shared: other jobs were using it during the run, decode speed varied from 18 to 39 tokens a second, and three calls found the model pushed out of memory, two of them waiting about 40 seconds for it to load back. That is where the cold start comes from. One streamed chunk is one token; the final end-of-answer token carries no text. The long document is my post on the 31 Vercel crons behind Lunary.

A hosted API adds what this local setup leaves out: DNS, TCP and TLS before the request (see How a URL becomes a page), the network in both directions, and a queue shared with everyone else, so its first token is usually later and less predictable, and the case for streaming is stronger. The thresholds are Nielsen’s: about 0.1 seconds feels instantaneous, about 1 second keeps a train of thought, and past about 10 seconds attention goes elsewhere (Nielsen, Usability Engineering, 1993; Response Times: The 3 Important Limits). Prefill reads the whole prompt in parallel and decoding makes one token per pass through the model, which is why the wait grows with the prompt and the stream grows with the answer (Pope et al., Efficiently Scaling Transformer Inference, 2022).