Tokenizer
The component that splits text into tokens — usually sub-word units, not whole words — that a language model can process. Different models use different tokenizers, so token counts vary.
Language models don't read characters or whole words; they read tokens. A tokenizer is the component that turns text into a sequence of token IDs and back. Modern tokenizers use sub-word units: common words are single tokens, less-common words split into pieces, and unusual character sequences (rare names, code, punctuation) may take several tokens each.
Different models use different tokenizers. GPT-4 and Claude use different vocabularies. A tokenizer with better coverage for a specific language will produce fewer tokens for the same sentence in that language, which means faster + cheaper processing.
Rough English rule of thumb: 1 token ≈ 4 characters ≈ 0.75 words. So a 100-word paragraph is roughly 130 tokens. Code, non-English text, and unusual formatting can push this ratio around meaningfully.
Two practical implications: (1) context window limits are in tokens, not words, so a "200K context" is roughly 150K English words, and (2) API pricing is per-token, so a more verbose prompt costs proportionally more.
Anthropic switched tokenizers in the Claude 4.7+ generation, which produces ~30% more tokens for the same text but with improved model behavior on the same character-count input. Claude Sonnet 4.6 uses the previous tokenizer.