What is a token in AI? How models read your text

If you have used an AI assistant for any length of time, you have met the word "token": in usage limits, in pricing, in an error message saying a prompt is too long. Tokens are the basic unit a language model works in, and knowing what they are explains several behaviors that otherwise look strange.

What a token is

A language model does not read letters or whole words. Before the model sees your message, a program called a tokenizer splits the text into pieces called tokens and turns each piece into a number. The model reads and writes sequences of those numbers, and the app converts them back into text for you.

A token can be a short common word such as "the", part of a longer word (a word like "unbelievable" may be split into several pieces), a fragment of a number, a punctuation mark, or a space attached to the start of a word. Where the splits fall depends on the tokenizer, and different model families use different ones.

How tokenizers decide where to split

Most tokenizers build their vocabulary from a large body of text. A common method, byte-pair encoding, starts from very small pieces and repeatedly merges the pairs that appear together most often, until the vocabulary reaches a chosen size, usually tens of thousands of entries or more. The result is that frequent words become a single token, while rare words, names, and unusual spellings break into several.

That is why the same sentence can have a different token count in two different models. The text is identical; the vocabularies are not.

Rules of thumb

These are estimates, not guarantees, and the exact count always comes from the specific tokenizer:

• English prose averages roughly four characters per token, or about three-quarters of a word. A hundred tokens is a little under a hundred words.
• Code, URLs, and numbers often need more tokens per visible character, because they contain many symbols and uncommon strings.
• Other languages frequently need more tokens to say the same thing, especially languages written in non-Latin scripts, because many vocabularies were built from text that was mostly English. How large the gap is depends on the tokenizer.
• Formatting counts. Tables, JSON, HTML tags, and long runs of whitespace all add tokens that carry little meaning for you.

Why tokens matter

• Limits. A model's context window is measured in tokens, and it has to hold your prompt, the earlier conversation, any attached text, and the reply being written. For what that limit means in practice, see what an AI context window is.
• Cost. Many providers charge by the token, with separate prices for input and output. Because a chat app usually resends the earlier conversation with every new message, a long conversation uses more tokens with each turn.
• Speed. A reply is produced one token at a time, so a long answer takes longer than a short one.
• Cut-off replies. If an app sets a maximum number of output tokens and the answer needs more, the reply simply stops, sometimes in the middle of a sentence.

Why models stumble on letters and arithmetic

Because a model works with tokens rather than letters, tasks that depend on spelling can be harder than they look. A question such as "how many times does the letter r appear in this word?" asks about something the model does not directly see, since the word may arrive as one or two tokens. Reversing a word, counting characters, and long exact arithmetic are similar. Models are often right, but they get these wrong in ways a person would not, and it is more reliable to have the assistant use a tool, such as code or a calculator, when an exact answer matters.

Practical ways to spend fewer tokens

• Cut boilerplate. Remove repeated headers, signatures, and copied text that has nothing to do with your question.
• Do not paste the same text twice. If a document is already in the conversation, refer to it instead of pasting it again.
• Start a new conversation for a new task. Old, unrelated history is resent and paid for even when it no longer helps.
• Ask for the length you need. If you want three bullet points, say so, and the answer will be shorter and faster.
• Use the provider's token counter. Word counts only approximate token counts, and the difference grows for code and for non-English text.

Quick answers

How many tokens is a page of text? By the usual rule of thumb, a page of about 500 words of English prose is roughly 650 to 700 tokens. Code, tables, and text in other languages often come out higher.

Do spaces and punctuation count? Yes. Spaces, punctuation, and line breaks are part of the text, and they are included in the token count, often attached to a neighboring word. Long runs of formatting can add up.

Is a token the same in every AI model? No. Each model family has its own tokenizer, so the same text can have different counts in different models. Use the provider's own counter when the exact number matters.

Why is my prompt longer than I typed? The count usually includes more than your message: earlier turns of the conversation, attached files, and hidden instructions from the app.

Where NAMELESS stands

NAMELESS is an AI model that is still in training, built on our own architecture so that you do not pay to reread a conversation. It does not reprocess the whole history to continue it, so long conversations cost pennies compared with today's leading assistants and never run into a context ceiling. Final pricing will be announced before launch.

To follow progress, you can join the early-access waitlist. For how remembering across conversations differs from fitting more text into one, read AI memory vs. context.