The problem it solves
A model does not read text the way people do, as letters and words. It needs its input as numbers, in consistent, manageable pieces. Splitting text into whole words would create an impossibly large vocabulary and choke on anything unusual; splitting into single letters would throw away too much meaning. Tokens are the practical middle ground.
By breaking text into subword chunks, a model gets a fixed, workable vocabulary that can still represent any text, including words it has never seen, by assembling them from smaller pieces. Tokenization is the quiet step that makes everything else possible.
How it works
When you send text to a model, it is first split into tokens, and each token is mapped to a number the model can process. Common words often become a single token; longer or rarer words get split into several.
A rough rule of thumb. For ordinary English, one token is about four characters, or roughly three-quarters of a word. So 100 words is around 130 tokens. It is only an approximation, and it shifts with language and content: the same meaning in some other languages, or in code, can take noticeably more tokens.
Tokens are the unit of cost. Most AI providers bill by the token, counting both what you send (input) and what the model generates (output), with output usually costing more. This is why the length and phrasing of prompts and responses translate directly into money at scale, and why concise is often cheaper.
Tokens are the unit of capacity. The amount a model can consider at once, its context window, is measured in tokens, not words or pages. Everything competing for that space, your instructions, the conversation, any documents, is counted in tokens. Run out, and the earliest material falls away.
You do not need to count tokens by hand. But knowing that text becomes tokens, and that tokens drive both cost and capacity, is what makes the behavior of AI systems predictable instead of mysterious.
A concrete example
Take the sentence “Tokenization is useful.”
To you, that is three words. To a model, it might be five or six tokens: “Token,” “ization,” “ is,” “ useful,” and the period, because a longer word like “tokenization” gets split into familiar sub-pieces while common short words stay whole.
Now scale that up. A 10,000-word report is roughly 13,000 tokens. If you send it to a model on every request, you pay for those 13,000 input tokens each time, and they take up a large share of the context window. Seeing the report as tokens, rather than pages, is what lets you reason about what it will cost and whether it will even fit.
How it connects
Tokens are how an LLM actually ingests text, so they sit underneath everything a model does. The size of a context window is counted in them, and they are the direct driver of cost and latency, the practical constraints every AI system runs into.
For the building roles this is daily arithmetic: an AI/LLM Developer watches token counts for cost and limits, and an AI Solutions Architect designs systems around them. Even non-technical roles benefit from knowing that, to a model, everything is tokens.