Token
The small unit of text an AI language model actually reads and writes. Rather than whole words, models break text into tokens — often a word, part of a word, or a punctuation mark — and process one at a time. Token counts are how usage is measured and priced.
When you type a sentence into a language model, it does not see letters or tidy words. It first chops the text into tokens: chunks that are usually a short word, a fragment of a longer word, or a mark of punctuation. The word “understanding” might become two or three tokens, while “cat” is just one. This step is how the model turns messy human text into a sequence of numbers it can actually work with.
How tokenisation works
Tokenisers are built by scanning huge amounts of text and finding the most common chunks, so frequent words get their own token while rarer words are stitched together from smaller pieces. That is why “cat” is one token but “antidisestablishmentarianism” is many: common things are cheap to represent, unusual things cost more. Each token is then mapped to a number, and it is those numbers — not the letters — that the model reads and predicts. Spaces, capital letters and punctuation all affect how text splits, which is why two sentences that look almost identical can have different token counts.
Why tokens matter
Tokens are not just a technical nicety; they govern the practical limits of using a model. A model generates its answer one token at a time, each one predicted from everything that came before. And everything it can hold in mind at once — your prompt plus its reply so far — is measured in tokens, not words. That ceiling is called the context window, and running past it is how a model “forgets” the start of a long conversation.
How tokens are priced
When an AI service quotes a price or a usage cap, it is almost always counting tokens — usually both the tokens you send (input) and the tokens it generates (output), often at different rates. A rough rule of thumb for English is that a token is about three-quarters of a word, so a thousand tokens is roughly 750 words. It is a small idea, but it sits underneath almost everything a language model does — how fast it responds, how much it can consider at once, and what it costs to run.