Tokens: How Text Becomes Numbers

When you type a message into ChatGPT, Claude, or Gemini, the model never sees your words the way you do. It sees numbers. Before any "thinking" happens, your text is chopped into small pieces called tokens, and each token is swapped for an ID number the model can do math on. Understanding tokens is the single most useful idea for demystifying large language models, because almost everything else (pricing, context limits, why models miscount letters) traces back to it.

This lesson gives you a clear mental model of tokenization without any code. If you have used AI tools daily and wondered why they charge "per token" or why they stumble on simple spelling questions, this is the answer.

What You'll Learn

• What a token is and how it differs from a word or a letter
• Why models break text into subword chunks instead of whole words
• Why the classic "how many R's in strawberry" question trips models up
• Why AI pricing and limits are measured in tokens, not words
• How token IDs differ from embeddings, the step where meaning actually lives

A token is a chunk of text, not a word

A token is a common chunk of characters. Sometimes a token is a whole word, sometimes it is part of a word, sometimes it is just punctuation or a space. The model has a fixed vocabulary of these chunks (often tens of thousands of them), and every token in that vocabulary has a number.

Here is the rough idea, using made-up IDs for illustration only:

• "cat" might be one token
• "unbelievable" might split into "un", "belie", "vable"
• A space usually attaches to the word after it, so " dog" is different from "dog"

A helpful rule of thumb in English: one token is roughly four characters, and 100 tokens is roughly 75 words. This ratio shifts by language and content, so treat it as a ballpark, not a law.

Why not just use whole words?

You might ask why models do not simply keep one number per word. Two reasons.

First, language is endless. People invent words, mash them together, make typos, and write in many languages. A whole-word vocabulary would need to be impossibly large and would still miss things. Subword tokens let the model represent any text by combining familiar pieces, even a word it has never seen.

Second, subword tokens capture structure. The chunk "ing" shows up in thousands of words. Learning it once and reusing it is far more efficient than memorizing every -ing word separately. This is why tokenizers are built by scanning huge amounts of text and merging the most frequent character pairs into reusable chunks.

Why "strawberry" breaks models

A famous party trick is to ask a model how many times the letter R appears in "strawberry." Many models have historically answered "two" when the answer is three. People take this as proof the model is dumb. It is actually proof of tokenization.

The model does not see s-t-r-a-w-b-e-r-r-y as ten separate letters. It sees a couple of tokens, maybe "straw" and "berry", each stored as a single number. The individual letters inside a token are not directly visible to the model any more than the individual pixels of a printed word are visible to you when you read it at a glance. Counting letters means reasoning about something the model was never handed cleanly. Newer models handle this better by reasoning step by step, but the underlying reason it was ever hard is tokens, not intelligence.

The practical takeaway: tasks that depend on exact characters (counting letters, reversing strings, strict spelling) are the ones where models are weakest, because those operate below the token level.

Why you are billed per token

AI providers charge by the token, not the word, and they usually charge separately for input tokens (what you send, including the whole conversation so far) and output tokens (what the model writes back). This matters for anyone using AI at work:

• A long document you paste in is counted as input tokens every time you send it.
• A rambling system prompt or a huge chat history quietly inflates your input token count on every turn.
• Asking for a shorter answer genuinely costs less, because output tokens are real money.

You do not need to count tokens by hand. Just internalize the model: more text in or out means more tokens, and tokens are the unit of cost and the unit of limits. Trimming what you send and being specific about the length you want are the two easiest ways to keep usage efficient.

Non-English text often uses more tokens per word, because the vocabulary was built mostly from English-heavy data. The same sentence in some languages can cost noticeably more tokens than its English version. This is worth knowing if you work across languages.

From token IDs to meaning: embeddings

One clarification before we move on, because tokenization and embedding often get blurred together. Turning text into token IDs is only bookkeeping: the ID for "cat" is just a lookup number, and nothing about it says that cats are more like dogs than like carburetors. The step that adds meaning is the embedding: inside the model, each token ID is converted into a long list of numbers, and those lists work like coordinates. Tokens used in similar ways end up near each other, and "near" is something a computer can actually measure.

So the full journey is text, then tokens, then IDs, then embeddings. Tokenization decides how text is chopped up and billed; embeddings are where "cat" and "kitten" end up close together. You will meet embeddings again in AI search and "chat with your documents" tools, because measuring which stored text sits closest to your question is exactly how those systems find relevant passages. If you want to go deeper, we cover them in What Are AI Embeddings? and put them to work in the Local RAG for Beginners micro course.

A quick way to build intuition

Most AI providers offer a free "tokenizer" web page where you paste text and watch it light up in colored chunks. Spending two minutes there does more for your intuition than any explanation. Try pasting a URL, a code snippet, an emoji, and a sentence in another language, and notice how differently they split. You will quickly see that "characters," "words," and "tokens" are three different things.

If you want to go deeper on the mechanics of tokenization after this course, the FreeAcademy blog post "What Is LLM Tokenization?" walks through more examples, and the broader AI Essentials course sets the big-picture context.

Key Takeaways

• A token is a common chunk of text (a word, part of a word, or punctuation), and each token maps to an ID number the model computes on.
• Models use subword tokens so they can represent any text, including new or misspelled words, from reusable pieces.
• Models struggle with letter-level tasks like counting R's in "strawberry" because they see tokens, not individual letters.
• Pricing and limits are measured in tokens, split into input and output, so shorter prompts and answers cost less.
• A rough English guide is ~4 characters per token and ~75 words per 100 tokens, but this varies by language and content.