AI Token Counter

A token is the atomic unit a language model reads and writes — typically 3–4 characters of English, but sometimes a whole common word, sometimes a single byte of an emoji. Models use byte-pair encoding (BPE): the tokenizer starts from individual bytes and repeatedly merges the most frequent adjacent pairs learned from training data, producing a fixed vocabulary (about 200,000 entries for GPT-4o's o200k_base). Counting words would be simpler for humans but useless for billing and context limits, because the API charges and truncates by tokens. The rule of thumb — 1 token ≈ ¾ of an English word, 100 tokens ≈ 75 words — is only an average; code, non-English text and special characters tokenize far less efficiently.

Each model family ships its own tokenizer with its own vocabulary and merge rules. GPT-4o uses o200k_base (~200k tokens), GPT-4 and GPT-3.5 use cl100k_base (~100k tokens), Claude models use Anthropic's proprietary tokenizer, and Gemini uses Google's SentencePiece vocabulary. A rarer word in one vocabulary splits into more pieces in another. This tool runs OpenAI's exact encodings for the GPT family via js-tiktoken (the same BPE the API bills against), and uses cl100k_base as a close, clearly-labeled approximation for Claude, Gemini and Llama — typically within a few percent, but not billable-exact.

Providers price input tokens (your prompt) and output tokens (the completion) separately, quoted per one million tokens. The estimate multiplies your measured input tokens by the model's list input price and your expected output length by its output price. Output tokens cost more because generating them is the expensive part of inference — GPT-4o, for example, lists $2.50 per million input tokens but $10.00 per million output tokens. Prices change, so every figure here is labeled an estimate; check the provider's pricing page before budgeting production workloads.

The API does not error — it silently truncates. Models read the most recent tokens that fit the window, so overflowing input drops content from the start of your prompt (often the system instructions, which is the worst part to lose). The context-window bar turns red past 100% to make this visible before you pay for a degraded call. Fixes: shorten the prompt, summarize earlier conversation turns, switch to a larger-window model (Claude 3.5 Sonnet offers 200K, Gemini 1.5 Pro up to 2M), or use retrieval to pass only relevant chunks.

No. Tokenization runs entirely in your browser using js-tiktoken's BPE tables compiled to JavaScript — the same merge ranks the API uses, executed locally. Prompts frequently contain the most sensitive material a developer handles (proprietary code, customer data, unreleased copy), which is exactly why pasting them into random online counters is risky. You can confirm the offline behavior in the Network tab: nothing carrying your text ever leaves the page.

BPE vocabularies are learned from predominantly English training data, so the merge table is rich in English fragments and poor in everything else. A common English word is often one token; the same concept in Thai, Arabic or emoji may split into many byte-level pieces. As an extreme case, a single emoji can cost several tokens because it encodes as multiple UTF-8 bytes with no learned merge covering them. If you work multilingually, budget roughly 2× the tokens per word compared to English, and measure with this tool rather than guessing.