What Is a Token in AI? Tokens vs Words vs Characters

The short answer

Language models do not read letters or words. A tokenizer first cuts your text into tokens and turns each one into a number, and the model works with those numbers. OpenAI describes a token as a character, part of a word, a whole word or punctuation, and adds that spaces also affect how text is divided.

For English, OpenAI gives three rough equivalents: 1 token is about 4 characters, about three quarters of a word, and 100 tokens are about 75 words. Google's Gemini documentation also says about 4 characters per token, and 100 tokens for about 60 to 80 English words. Both present them as approximations. Our own counts with OpenAI's current tokenizer landed close: 5.0 characters per token for plain English and 4.4 for text with many numbers, and 800 to 900 words per 1,000 tokens.

How text becomes tokens

OpenAI's tokenizers use byte pair encoding, or BPE. The tiktoken project describes four properties: it is reversible and lossless, so tokens turn back into the exact original text; it works on any text, even text the tokenizer never saw in training; it compresses text, because there are fewer tokens than bytes; and it lets the model see common subwords, such as "ing" in English. In practice, frequent words get a token of their own, while rare words are built from smaller pieces.

Here is how the o200k_base tokenizer, which OpenAI lists for GPT-4o, splits a few strings. Each pair of brackets holds one token.

Three things stand out. Long numbers are cut into groups of up to three digits, so 1,000,000 costs five tokens while "one million" costs two. Dates and prices break at every separator. And a web address is expensive: the address above is 42 characters and 11 tokens, about 3.8 characters per token, against about 5 for plain English.

Spaces and capital letters matter too. OpenAI's cookbook notes that spaces are usually grouped with the start of a word, so " red" with its leading space is a different token from "red", and "Red" is a third one. In our test all three are single tokens, each with its own ID number. That is why the same word can count differently depending on where it stands in a sentence.

Tokens vs words vs characters, measured

To see how far the rule of thumb carries, we counted five samples with two of OpenAI's encodings: o200k_base, which OpenAI's cookbook lists for GPT-4o and GPT-4o mini, and the older cl100k_base, listed for GPT-4, GPT-4 Turbo and GPT-3.5 Turbo. Words are counted the way the word counter counts them and characters include spaces, as in the token counter.

Plain English beat the rule of thumb: 1.11 tokens per word, so about 900 words per 1,000 tokens instead of 750. The guide prose, full of prices, percentages and formulas, came in at 1.26 tokens per word, 800 words per 1,000 tokens. Code was 3.26 characters per token, about 14 tokens per line, and the JSON file only 2.45 characters per token, because quotes, braces and colons each cost a token.

The method has limits. These are five samples, not a corpus, and the words and characters are ours to count; the providers bill the tokens only. Your own text can land on either side. Paste it into the token counter to get the exact figure for OpenAI models.

Why German and other languages need more tokens

OpenAI notes that other languages can have different relationships between characters, words and tokens. Our German translation shows how. It has almost the same number of words as the English original, but needed 283 tokens instead of 245, 16% more for the same content. With the older cl100k_base encoding the gap was 41%. One sentence makes it visible:

In the older encoding, "Frühjahr" fell apart into four pieces and "Büro" into two; the newer one keeps each as a single token. The newer vocabulary is about twice as large, with roughly 200,000 entries against roughly 100,000 in the library we used, which leaves room for more non-English words. Long compound words still split: "Kundenanfragen" (customer questions) is four tokens, while "customer questions" is two. Accented letters, scripts other than Latin and emoji add more. A Christmas tree emoji is two tokens in o200k_base and three in cl100k_base.

Why every model counts differently

Each provider trains its own tokenizer, so a token on one model is not the same size as a token on another. Anthropic states that Claude Opus 4.7 and later models use a newer tokenizer that turns the same text into about 30 percent more tokens than earlier Claude models, depending on the content, and tells developers to recount prompts for the model they plan to use. Google gives its own rule for Gemini, quoted above.

For exact numbers, use the provider's own counting:

• OpenAI: the tiktoken library for plain text, and an input token counting API for complete requests, which also counts the formatting tokens around messages, tools and files.
• Anthropic: a token counting endpoint that is free to use within rate limits. Anthropic calls its result an estimate that can differ from the real request by a small amount.
• Google: token counting in the Gemini API, which also covers images, video and audio, each with its own rate.

The token counter on this site uses OpenAI's own encodings, so for text it matches what OpenAI counts with the same encoding. For Claude, Gemini and other models, treat its result as an estimate. A whole chat request also adds a few tokens per message for roles and boundaries, which OpenAI's cookbook shows in its counting example.

Context windows in tokens and words

A context window is the most a model can work with in one request. Anthropic defines it as all the text the model can reference when it writes a response, including the response itself. Three consequences follow from the providers' documentation:

• The answer counts. Input and output share the window. For reasoning models, OpenAI says the hidden reasoning tokens count as output tokens and are billed as output, so leave room for them as well as the visible answer.
• History counts. In a conversation, each new request carries the previous turns as input, so a long chat fills the window turn by turn.
• Everything counts. System prompts, tool definitions, documents and images all take up tokens in the same window.

Models also have a separate cap on output. Google's documentation describes some Gemini models where input, thinking and output share one window, and others with separate input and output budgets, so check the model page before you plan a long document. As a rough guide, here is how many words or lines of code fit in common window sizes, at the ratios we measured. Real capacity is lower once you subtract the answer.

How pricing per million tokens works

API providers bill per token, with separate prices for input and output, usually quoted per million tokens. OpenAI adds that cached input and output can have their own prices too. The cost of one request is:

Take a made-up model that charges $1 per million input tokens and $4 per million output tokens. You ask it to summarize a 3,000-word English report in 400 words, with 200 tokens of instructions. At our plain English ratio that is about 3,541 input tokens and 445 output tokens, so $0.0035 for the input and $0.0018 for the output: $0.0053 per report, about $5.32 for 1,000 reports. The same job in German, at 16% more tokens for the same content, costs about $0.0061 per report.

Two lessons from that arithmetic. In the price list behind our calculators, checked October 9, 2026 on the providers' own pricing pages, output costs more than input for 93 of 96 models, and the typical model charges about 5 times as much for output. Long answers can drive the bill more than long prompts. And as OpenAI points out, a lower price per million tokens does not guarantee a lower total cost, because models tokenize the same text differently and write different amounts of output. Current list prices change often, so we keep them in two tools with the date they were checked: the AI token cost calculator prices one request on the models you pick, and the AI cost comparison runs the same job on every model in the list, cheapest first.

How to use fewer tokens

• Cut what repeats. OpenAI's own advice for oversized input is to shorten the prompt, remove unnecessary or repeated context, split large inputs and summarize before sending.
• Strip the markup. The HTML of one of our guide pages needed 46% more tokens than its plain text, and JSON packs fewer characters into each token than prose. Paste plain text when the formatting does not matter.
• Watch long numbers and links. IDs, tracking links and long figures cost a token per few characters. Leave out the ones the model does not need.
• Ask for shorter answers. Set a length in the prompt and the output limit in the API. For most models, output is the expensive side.
• Count before you send. Paste a draft into the token counter to see how it splits, and check the long parts first.

Frequently asked questions

Sources

• OpenAI Help Center: Understanding and counting tokens (rules of thumb for English, input, output, cached and reasoning tokens, context window, pricing per token)
• OpenAI Cookbook: How to count tokens with tiktoken (encodings per model, spaces grouped with the start of words, extra tokens per chat message)
• OpenAI tiktoken on GitHub: "What is BPE anyway?" (reversible and lossless, works on any text, compresses text, common subwords)
• Anthropic, Claude API docs: Token counting (free endpoint, the count is an estimate, newer tokenizer with about 30 percent more tokens)
• Anthropic, Claude API docs: Context windows (input and output count toward the window, conversation history accumulates)
• Google AI for Developers: Understand and count tokens (about 4 characters per token, 100 tokens about 60 to 80 English words, token limits, billing)

Written by Tobias Kurth, designer. Updated October 20, 2026.