What Are Tokens? The Small Pieces AI Reads

Before an AI processes text, image, or audio data, or any content online it breaks the information down into smaller pieces through a process called tokenization.

When you send text to an AI, it does not read it the way you do. It splits your text into small pieces called tokens. A token can be a single character, part of a word, a whole word, or a punctuation mark.

As a rough rule for English text, one token is about four characters, or about three quarters of a word. So 100 tokens works out to roughly 75 words. These are estimates, not exact counts, and other languages divide up differently.

Two reasons. First, memory. A model's context window, the total amount of text it can consider in one request, is measured in tokens. Second, cost. Most AI services charge by the token, with separate prices for input tokens and output tokens.