What is a token?

Language models don't read whole words or individual characters. They read tokens — sub-word units chosen during training to balance vocabulary coverage against sequence length.

The rough rule

For English text: **1 token ≈ 4 characters ≈ 0.75 words**. So a 100-word paragraph is roughly 130 tokens. A 500-word essay is roughly 650 tokens. A 200K-token context window is roughly 150K words.

Why it's not exact

The tokenizer treats common words as single tokens ("the" is one token) but breaks less-common ones into pieces ("photosynthesis" might be 4-5 tokens). Non-English text uses more tokens per word — Chinese, Japanese, and Arabic can hit 2-3× English density. Code often tokenizes densely too (a punctuation-heavy line might be all short tokens).

Unusual character sequences, rare names, and formatting can each take multiple tokens where you'd expect one.

Why token counts matter

Two practical implications:

**Context window limits are in tokens.** A "200K context window" (Claude Sonnet) means the model can process ~200K tokens per request — so a 500-page dense PDF is at the edge. Chat history + system prompt + attached files all count against that budget.

**API pricing is per-token.** More tokens = more cost. A verbose prompt costs more than a tight one for the same task. LADLE bakes this into the metering: every send's cost is computed from the actual token counts, not from any per-message flat fee.

Different models, different tokenizers

Claude Sonnet 4.6 uses one tokenizer. Claude 4.7+ generation uses a newer tokenizer that produces ~30% more tokens for the same text (with improved model behavior on the same character-count input). GPT models use OpenAI's tokenizer.

The takeaway: token counts aren't universal. The same text sent to two different models will have two different token counts, and comparing costs across models requires normalizing for this.

Estimating token counts

If you're eyeballing whether something fits in context: count characters, divide by 4, add ~15% headroom for non-English or code-dense content. Good enough for planning.

Real token count is available programmatically via Anthropic's count-tokens endpoint if you need exact numbers. For most decisions, the rule of thumb is enough.