Charts · 33

How text becomes tokens

A language model never sees letters. Before your words reach it, a tokeniser cuts them into pieces from a fixed vocabulary and swaps each piece for a number, and the numbers are all the model gets. Common words are one piece, rare ones are several, and a character can be split across pieces. Here are the same sentences through four tokenisers in real use, every chip one token exactly as each returned it, and the reason the strawberry question trips models up.

How many r's are in strawberry? 31 characters · 31 bytes

The question is eight or nine tokens and ' strawberry', with its space, is one of them. The model is asked about letters inside a token it receives as a single number.

cl100k_baseGPT-4, GPT-3.5 Turbo · 100,277 token ids8 tokens
  1. How
  2. many
  3. r
  4. 's
  5. are
  6. in
  7. strawberry
  8. ?
o200k_baseGPT-4o, GPT-4.1, o3, GPT-5 · 200,019 token ids8 tokens
  1. How
  2. many
  3. r
  4. 's
  5. are
  6. in
  7. strawberry
  8. ?
Llama 3Llama 3 and 3.1, read from Hermes 3 8B · 128,256 token ids8 tokens
  1. How
  2. many
  3. r
  4. 's
  5. are
  6. in
  7. strawberry
  8. ?
Gemma 3Gemma 3, SentencePiece · 262,208 token ids9 tokens
  1. How
  2. many
  3. r
  4. '
  5. s
  6. are
  7. in
  8. strawberry
  9. ?

Hover a chip for its id. · is a space, ↵ a newline; a chip in hex is a token that is only part of a character.

What GPT-4o actually receives[5299, 1991, 428, 885, 553, 306, 101830, 30]

Why it cannot count the r’s in strawberry

To GPT-4o, strawberry is [302, 1618, 19772]: “st”, “raw”, “berry”. None of those is the letter r. Gemma 3 has the whole word as one piece, which hides the letters completely. Whatever a model knows about how its pieces are spelt, it had to learn as a fact about numbers: it cannot look. Spell the word out with spaces and every letter becomes its own token, and the count is suddenly easy.

Inputcl100k_baseo200k_baseLlama 3Gemma 3
strawberry
  1. str
  2. aw
  3. berry
  1. st
  2. raw
  3. berry
  1. str
  2. aw
  3. berry
  1. strawberry
strawberry
  1. strawberry
  1. strawberry
  1. strawberry
  1. strawberry
Strawberry
  1. Str
  2. aw
  3. berry
  1. Str
  2. aw
  3. berry
  1. Str
  2. aw
  3. berry
  1. Strawberry
STRAWBERRY
  1. ST
  2. RAW
  3. B
  4. ERRY
  1. ST
  2. RAW
  3. B
  4. ERRY
  1. ST
  2. RAW
  3. B
  4. ERRY
  1. STRA
  2. W
  3. BER
  4. RY
strawberries
  1. str
  2. aw
  3. berries
  1. st
  2. raw
  3. berries
  1. str
  2. aw
  3. berries
  1. straw
  2. berries
strawberry
  1. s
  2. t
  3. r
  4. a
  5. w
  6. b
  7. e
  8. r
  9. r
  10. y
  1. s
  2. t
  3. r
  4. a
  5. w
  6. b
  7. e
  8. r
  9. r
  10. y
  1. s
  2. t
  3. r
  4. a
  5. w
  6. b
  7. e
  8. r
  9. r
  10. y
  1. s
  2. t
  3. r
  4. a
  5. w
  6. b
  7. e
  8. r
  9. r
  10. y

Notes

Every token on this page was produced offline by scripts/labs/tokens.py on 19 October 2026; nothing is tokenised in the browser and nothing is approximated. cl100k_base and o200k_base are OpenAI’s tiktoken 0.12.0, with the BPE files checked against the SHA-256 hashes pinned in tiktoken itself. Llama 3 is tiktoken running the 128,000 ordinary tokens read out of a Llama 3.1 8B model file (Hermes 3, through Ollama) with the split pattern from Meta’s llama3 tokenizer. Gemma 3 is SentencePiece’s BPE, score-ordered merges with byte fallback, over the 262,208-piece vocabulary in Ollama’s gemma3:4b-it-qat. Both of those were checked against the models themselves: for ten test sentences, including the emoji, the Japanese and the whitespace, the number of prompt tokens Ollama reported was ours plus one start-of-text token, every time.

Which models use which: o200k_base is GPT-4o, GPT-4.1, o1, o3, o4-mini and GPT-5, and cl100k_base is GPT-4 and GPT-3.5 Turbo, per the model table in tiktoken. Token ids count every id in the vocabulary, special tokens included. Byte-level tokenisers (the GPT ones and Llama 3) can always fall back to single bytes, which is why a Japanese character can arrive in three tokens that are not characters at all; SentencePiece does the same with its 256 byte pieces. The strawberry problem is the well-known failure of models to count letters: the letters are not in the input, only the ids of the pieces, so counting them means recalling how a piece is spelt. Sources: Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units (2016); Kudo and Richardson, SentencePiece (2018); OpenAI, tiktoken; Meta, llama3 tokenizer.py.