Full explanation
A token is a unit of text, often a word or part of a word, that an AI language model processes as its basic building block.
A token is not quite the same as a word. Common short words are often a single token; longer or less common words can be split into two or three token pieces. As a rough rule of thumb, a token is close to four characters of English text, so a page of a few hundred words might run to several hundred tokens.
This is the unit both the context window and typical API pricing are measured in. It also explains a common overclaim: a system that says it has "read your whole site" is bounded by how many tokens it can actually hold or was fed at once, so a very large site is more often summarized, sampled, or retrieved from selectively than read in full.
Why this matters
It is the unit behind context limits and pricing, and useful to know whenever anyone claims a system has read an entire website.
Example
A short sentence describing what a company does might run to seven or eight tokens, not seven or eight words, once uncommon words get split into smaller pieces. Multiply that across a whole page and it becomes clear why context windows, measured in tokens, run out faster than a word count would suggest.
Where this sits
Token belongs to AI & Search Infrastructure, one of the seven layers of the Aingworth Glossary. The technical vocabulary behind the interfaces. Useful for understanding how the systems work, without becoming the center of the conversation.
Related terms
Ready to see where your brand stands?
Start with the free diagnostic to see which condition of the Recognition Triangle is leaking, or read the frameworks these terms belong to.