What Is a Token? Why a Language Model Sees No Text

What is a token?

A token is the smallest unit a language model splits text into before processing it. Usually a word fragment, not a letter and not a whole word. A program, the tokenizer, turns text into a list of numbers. The model only ever sees those numbers, never the letters. Tokens drive cost, context limits, and a whole class of errors.

Ask a language model how many “r”s are in the word “strawberry”, and it often gets it wrong. There are only three. Not a hard task.

The model is not a bad counter. It just does not see the word the way a human does. A picture helps, and it carries this whole series: the model is the brilliant, fast, occasionally-lying intern. Strong at phrasing. Blind in a surprising spot. That blindness starts with this word.

A Model Reads in Building Blocks, Not Letters

Before the model computes anything, a separate program runs first. It cuts the text into fixed building blocks from a fixed box. The model never gets the individual letters. Only these ready-made pieces.

Like a mosaic made of whole tiles, not single dots of color. Counting the red dots would mean knowing by heart what sits on each tile. That is the model’s situation when counting letters.

This picture has a name: token. The program that does the cutting is the tokenizer.

A token is one building block in the series. Here is where it sits:

Under the Hood

The building blocks come from BPE (Byte-Pair Encoding)Byte-Pair EncodingThe common tokenizer method. It starts from individual bytes and repeatedly merges the most frequent adjacent pair into a new token. Frequent words become one token, rare ones shatter into many.. Before training, a procedure runs over huge amounts of text and repeatedly merges the most frequent pair of characters into a new token. Frequent words become one token, rare ones shatter. The vocabulary ends up around 100,000 to 200,000 tokens.

Concretely, with the GPT-4 tokenizer:

"strawberry"  →  str · aw · berry  →  [496, 675, 15717]

The model gets [496, 675, 15717], nothing more. The three “r”s sit spread across str and berry. Rule of thumb for English: one token is about four characters or 0.75 words. Each model family has its own tokenizer, so a different model splits the same word differently.

And the cost side, with numbers: German text splits into more tokens than English of the same meaning. Research calls this surcharge the token premium (Petrov et al., 2023). It persists even when content and vocabulary size are controlled, so it is not a mere length effect. For German it runs roughly 1.3 to 1.5 times, depending on the tokenizer.

Where It Breaks

Every task at the letter level becomes unreliable. Counting letters. Spelling a word. Reversing it. Digit-by-digit arithmetic. The model does not have that level in front of it. It sometimes succeeds anyway, but it cannot be relied on. The fix: hand such steps to code that actually counts.

The expensive break sits elsewhere, and it hits German-speaking businesses directly. This token surcharge on German lands straight on the bill: billing is per token, so the same task costs noticeably more in German. Estimating a German AI project from English test data underestimates the bill. The fix: measure cost on real German data, not on English.

The model sees no text. It sees numbers for fixed word building blocks. Holding that in mind makes a whole class of AI errors obvious at a glance. From the miscounted “strawberry” to the higher bill for German text.

Next term: the Context Window. How many of these building blocks a model can hold in view at once. And why it treats the middle of long texts worse than the start and end.

Frequently Asked Questions

Is a token the same as a word? No. A token is usually a word fragment. Frequent words are one token, rare ones split into several. Rule of thumb for English: one token is about four characters or 0.75 words.

Why does a language model count letters wrong? Because it does not see the word as a sequence of letters, but as a few token fragments with number IDs. The individual letters are locked inside those fragments and not directly accessible.

Why does AI cost more in German than in English? Because German text splits into more tokens than English of the same meaning, roughly 1.3 to 1.5 times depending on the tokenizer. Billing is per token, so cost rises accordingly.

Should I write AI prompts in German or English? English saves tokens, so it saves cost and room in the context window. But it barely improves answer quality on current frontier models. The quality edge held mainly for older and smaller models and has largely closed for well-resourced languages like German. In short: English is a cost lever, not a quality lever.

Part of the AI Engineering, explained series. Related: Why ChatGPT Does Not Know Your Company Knowledge and What Does an AI Project Cost in the Mittelstand.