Every major AI provider handles token counting, billing, and context windows differently. This reference covers the exact mechanics, edge cases, and implementation details.

The Complete Guide to LLM Tokenization and Token Economics

Tokenization is the first and last step of every LLM interaction. It determines how much a request costs, whether it fits in the context window, how the model interprets input, and why the same text costs different amounts across providers. Despite this, most developers treat tokenization as a black box — sending strings in, getting strings out, and hoping the bill makes sense.

The mechanics underneath matter. A tokenizer that produces 30% more tokens for the same text changes cost calculations, context window planning, and rate limit headroom. Pricing tiers that shift based on prompt length can double a bill without any change in the actual content. Understanding these systems at the byte level turns vague cost estimates into precise engineering decisions.

How Tokenization Actually Works
BPE, SentencePiece, and the Tokenizer Zoo
Token Counts Vary Across Providers — By a Lot
Counting Tokens Before You Send Them
Context Windows and Max Output: What the Numbers Mean
How Billing Actually Works
Prompt Length Pricing Tiers
Cached and Prefilled Tokens
Hidden Token Costs: System Prompts, Tool Definitions, and Thinking
Cross-Provider Token Cost Comparison
Practical Token Optimization Patterns
Tokenizer Edge Cases That Break Things
Summary
Further Reading

How Tokenization Actually Works {#how-tokenization-actually-works}

LLMs operate on integer sequences, not text. A tokenizer converts a string of characters into a sequence of integer token IDs, each mapping to an entry in a fixed vocabulary. The model processes these IDs through its embedding layer, attention mechanism, and output head. On the way out, the sampled token IDs get decoded back into text.

The vocabulary size varies by model family. GPT-6 Astra uses a vocabulary derived from OpenAI’s o200k_base encoding (~200,000 tokens). Anthropic’s Claude models from the 4.7 generation onward use a tokenizer that produces roughly 30% more tokens for the same English text compared to the previous Claude tokenizer. Gemini models use SentencePiece with a 256K vocabulary. These differences are not academic — they directly affect cost and context utilization.

The tokenization pipeline: raw text becomes byte pairs, then integer IDs, then dense vectors the model consumes.

A single English word typically maps to 1-3 tokens. Common words like “the” or “and” are single tokens. Less common words get split: “tokenization” might become [“token”, “ization”] or [“tok”, “en”, “ization”] depending on the vocabulary. Code, non-Latin scripts, and emoji tend to consume more tokens per semantic unit.

BPE, SentencePiece, and the Tokenizer Zoo {#bpe-sentencepiece-and-the-tokenizer-zoo}

Almost all current LLM tokenizers are variants of Byte Pair Encoding (BPE). The algorithm starts with individual bytes (or characters) and iteratively merges the most frequent adjacent pair into a new token, repeating until the vocabulary reaches a target size. The merge order defines the tokenizer’s behavior — the same text will tokenize differently depending on which merges were learned during training.

OpenAI uses tiktoken, a Rust-backed BPE implementation. The current encoding for GPT-6 family models is o200k_base (200,019 tokens). Earlier GPT-4-era models used cl100k_base (~100K tokens). The larger vocabulary means more common subwords get their own token, generally producing fewer tokens for the same text.

Anthropic uses a custom BPE tokenizer. The tokenizer changed at the Claude 4.7 generation boundary. Models from Claude 4.7 onward (including all current models: Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5) emit approximately 30% more tokens for the same English input than Claude Sonnet 4.6 and earlier models. This has a direct cost impact: even though per-token prices might look competitive, the effective per-character cost is higher than a naive comparison suggests.

Google uses SentencePiece, which operates on raw Unicode text (no pre-tokenization step) and supports both BPE and Unigram algorithms. Gemini models use a 256K vocabulary. SentencePiece handles multilingual text more gracefully than byte-level BPE in some cases, since it doesn’t require language-specific pre-tokenization rules.

Open-source models vary. Llama 4 uses a SentencePiece BPE tokenizer. Qwen3.8 uses a BPE tokenizer with a ~150K vocabulary. Mistral models use SentencePiece. DeepSeek-V4.1-Flash uses a custom BPE tokenizer.

The four major tokenizer families in production today, each producing different token counts for identical text.

Token Counts Vary Across Providers — By a Lot {#token-counts-vary-across-providers}

The same 1,000-word English document produces meaningfully different token counts depending on which provider processes it. This makes cross-provider cost comparisons unreliable if done only on per-token price.

Here are approximate token counts for a representative 1,000-word English technical document:

Provider / Tokenizer	Approximate Tokens	Relative to OpenAI
OpenAI (o200k_base)	~1,300	1.0x
Anthropic (Claude 4.7+)	~1,690	~1.3x
Google (SentencePiece 256K)	~1,250	~0.96x
Llama 4 (SentencePiece)	~1,350	~1.04x
Qwen3.8	~1,280	~0.98x

These ratios shift with content type. For Chinese text, Qwen’s tokenizer is substantially more efficient than OpenAI’s. For code, tokenizers with larger vocabularies tend to produce fewer tokens because common programming constructs (function signatures, common library names) get dedicated tokens. For JSON-heavy payloads, the overhead of structural characters ({, }, ", :) is roughly similar across tokenizers, but key names and string values diverge.

The 30% inflation from Anthropic’s post-4.7 tokenizer deserves emphasis. If an application sends 100K tokens per request on OpenAI, the equivalent text sent to Claude Opus 5.5 will be billed as roughly 130K tokens. At Opus 5.5’s $4/$20 per million pricing, that 30% matters:

OpenAI GPT-6.1 Sol at $2/$10 per M: 100K input tokens = $0.20
Claude Opus 5.5 at $4/$20 per M: ~130K input tokens = $0.52

The per-token prices look like a 2x difference. The effective cost for the same text is closer to 2.6x.

Counting Tokens Before You Send Them {#counting-tokens-before-you-send-them}

Accurate pre-flight token counting prevents context overflow errors and enables precise cost estimation. Each provider offers different tooling.

OpenAI — tiktoken:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
tokens = enc.encode("Your text here")
print(len(tokens))  # exact token count for GPT-6 family

tiktoken is fast (Rust-backed), runs locally, and gives exact counts for OpenAI models. It’s the gold standard for pre-flight estimation.

Anthropic — no official local tokenizer:

Anthropic does not publish its tokenizer as a standalone library. The API returns usage.input_tokens and usage.output_tokens in every response, but there’s no way to get an exact count before sending. The /messages/count_tokens endpoint exists for pre-flight counting but requires a network round trip. For estimation, multiplying the tiktoken count by 1.3 gets within 5-10% for English text.

import anthropic

client = anthropic.Anthropic()
response = client.messages.count_tokens(
    model="claude-opus-5-5",
    messages=[{"role": "user", "content": "Your text here"}]
)
print(response.input_tokens)

Google — tokenCountText API or local approximation:

Google provides a count_tokens method in the Gemini API client libraries. For local estimation, the sentencepiece Python package can load Google’s published tokenizer models.

import google.generativeai as genai

model = genai.GenerativeModel("gemini-4-argon")
count = model.count_tokens("Your text here")
print(count.total_tokens)

Open-source models — use the model’s tokenizer directly:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.8-2.4T-A95B")
tokens = tokenizer.encode("Your text here")
print(len(tokens))

Three approaches to pre-flight token counting: local exact count, API endpoint, or ratio-based estimation.

Context Windows and Max Output: What the Numbers Mean {#context-windows-and-max-output}

The “context window” number in a model’s spec sheet is the total token budget for a single request — input tokens plus output tokens must fit within it. This is frequently misunderstood as the input limit alone.

Model	Context Window	Max Output	Effective Max Input
GPT-6 Astra	1,050,000	128,000	922,000
Claude Fable 5.1	1,000,000	128,000	872,000
Claude Opus 5.5	1,000,000	128,000	872,000
Gemini 4 Argon	1,000,000	65,536 default / 1,048,576 long decode	934,464 - varies
DeepSeek-V4.1-Flash	1,000,000	384,000	616,000
Qwen3.8-Max	1,000,000	varies	varies
Grok 4.7	500,000	no documented limit	~500,000

Gemini 4 Argon is an outlier with its “long decode” capability — up to 1M output tokens, which means the context window effectively becomes shared between input and output in a flexible way. The default 65,536 max output is conservative; long decode must be explicitly enabled.

GPT-6 Astra’s 1,050,000 context window is the largest among the commercial flagships, but prompts exceeding 272K input tokens trigger a pricing surcharge: 2x on input/cached tokens and 1.5x on output for the entire request. This creates a soft boundary that’s invisible in the model spec but very visible on invoices.

The context window is a shared budget. Input and output tokens compete for space within it.

How Billing Actually Works {#how-billing-actually-works}

LLM billing is per-token, but the devil is in the token definition. Providers bill on their own tokenizer’s output, not on some standard unit. This means:

Input tokens are billed at the input rate. These include the system prompt, user message, conversation history, tool definitions, and any injected context.
Output tokens are billed at the output rate, which is typically 2.5–5x higher than input. This reflects the higher computational cost of autoregressive generation versus the parallel processing of input.
Thinking/reasoning tokens — models with adaptive thinking (Claude Opus 5.5, Claude Fable 5.1) or configurable reasoning (Grok 4.7) generate internal reasoning tokens that may or may not be billed. Anthropic bills thinking tokens at the output rate. OpenAI includes reasoning tokens in the output token count.

The asymmetry between input and output pricing is the single most important factor in cost optimization. A prompt-heavy, short-response workload (classification, extraction) costs far less per request than a generation-heavy workload (long-form writing, code generation) even at identical input sizes.

Billing flow for a single API request. Cache hits reduce the input portion; output tokens always cost full price.

Prompt Length Pricing Tiers {#prompt-length-pricing-tiers}

Several providers now implement tiered pricing where the per-token rate changes based on prompt size. This is a relatively recent development and catches teams off guard.

GPT-6 Astra: Prompts over 272,000 input tokens trigger a multiplier on the entire request — 2x on input and cached tokens, 1.5x on output. A request with 300K input tokens and 10K output tokens costs substantially more per token than an identical-content request split into smaller chunks.

Prompt Size	Input Rate	Output Rate
≤ 272K tokens	$10/M	$50/M
> 272K tokens	$20/M	$75/M

Claude Haiku 5.5: Pricing is split at the 100,000-token boundary for the prompt.

Prompt Size	Input Rate	Output Rate
≤ 100K tokens	$0.10/M	$0.50/M
> 100K tokens	$0.50/M	$2.50/M

This 5x jump means Haiku 5.5 stops being the obvious budget choice once prompts exceed 100K tokens. A 150K-token prompt on Haiku 5.5 costs $0.075 input — comparable to Sonnet 5.5 at $0.30 for the same 150K. The output rate differential ($2.50 vs $10) still favors Haiku, but the gap narrows dramatically.

Gemini 4 Argon: Has a two-tier structure with introductory pricing of $2/$10 per M on the lower tier, rising to $4/$20 on a larger-prompt tier. The exact token threshold for the tier change has not been confirmed by Google, so precision here is impossible. The introductory window runs through November 30, 2026, and rates double on January 1, 2027.

Grok 4.7 and 4.6: Both use a 200K prompt threshold. Below 200K: $2/$6 input/output per M. Above 200K: $4/$12.

Provider / Model	Tier Boundary	Below	Above
GPT-6 Astra	272K input	$10/$50	$20/$75
Claude Haiku 5.5	100K prompt	$0.10/$0.50	$0.50/$2.50
Gemini 4 Argon	unconfirmed	$2/$10	$4/$20
Grok 4.7	200K prompt	$2/$6	$4/$12

The architectural takeaway: applications that routinely send large contexts should model these tiers explicitly. A RAG system stuffing 300K tokens of retrieved context into GPT-6 Astra pays double compared to one that manages to stay under 272K.

Tiered pricing creates a step function in cost. Staying below the threshold can halve the effective rate.

Cached and Prefilled Tokens {#cached-and-prefilled-tokens}

Prompt caching is the single largest cost lever for applications with repetitive prompt prefixes — system prompts, tool definitions, few-shot examples, or shared document context.

OpenAI: Automatic prompt caching matches the longest prefix of a new request against recently cached prompts. Cached input tokens are billed at roughly 10% of the standard input rate. For GPT-6.1 Sol at $2/M input, cached tokens cost $0.10/M. For GPT-6 Luna at $0.10/M, cached tokens cost $0.01/M. Cache hits require exact prefix matching — any change in the prefix invalidates the cache.

Anthropic: Explicit cache control via cache_control blocks in the message. Cached input reads cost a fraction of full input. For Opus 5.5: cache reads are $0.20/M versus $4/M full price (95% discount). For Sonnet 5.5: $0.10/M versus $2/M. For Fable 5.1: $0.25/M versus $10/M. There’s a cache write premium on the first request that creates the cache entry.

Google Gemini: Cached input is 95% off for Gemini 4 Argon during the introductory period. Context caching is explicit and requires creating a cached content object that persists for a configurable TTL.

DeepSeek: Automatic caching with off-peak pricing already at $0.15/M input (vs $0.30/M at peak), meaning cached tokens at off-peak rates can go as low as $0.003/M — extremely cheap for million-token contexts.

The economics of caching are stark. An agentic system that sends a 50K-token system prompt with every request and makes 1,000 requests per day:

Without caching (Opus 5.5): 50K × 1,000 × $4/M = $200/day on system prompt alone
With caching (Opus 5.5): 50K × 1,000 × $0.20/M = $10/day

That’s a 20x reduction on the static portion of the prompt. For high-volume applications, implementing caching is not an optimization — it’s a prerequisite for economic viability.

Hidden Token Costs: System Prompts, Tool Definitions, and Thinking {#hidden-token-costs}

Several token costs are invisible in a naive cost estimate because they don’t appear in the user’s message content.

System prompts are billed as input tokens on every request. A 2,000-token system prompt sent with 500 requests per hour adds 1M input tokens per hour. At Opus 5.5 rates without caching, that’s $4/hour or ~$2,880/month for the system prompt alone.

Tool/function definitions are serialized into the prompt by the provider. Each tool definition with its parameter schema, description, and examples can consume 200-500 tokens. An agent with 20 tools might add 5,000-10,000 tokens to every request. These are billed as input tokens.

Conversation history grows linearly with turn count. A multi-turn conversation carries all previous messages as input context. After 20 turns of substantive conversation, the accumulated history can easily reach 30K-50K tokens — all billed as input on the final request.

Thinking/reasoning tokens are the most surprising cost. Adaptive thinking models (Claude Opus 5.5, Fable 5.1) generate internal reasoning that’s billed at the output token rate. A request that returns 500 visible output tokens might generate 3,000 thinking tokens, making the effective output 3,500 tokens. At Fable 5.1’s $50/M output rate, those invisible thinking tokens cost $0.15 per request.

The visible part of a request is often the minority of total billed tokens. Hidden costs accumulate across system prompts, tools, history, and thinking.

Estimating hidden cost for a typical agent request:

Component	Tokens	Billed at
User message	500	input rate
System prompt	3,000	input rate
Tool definitions	5,000	input rate
Conversation history	20,000	input rate
Total input	28,500
Thinking	4,000	output rate
Visible response	800	output rate
Total output	4,800

At Sonnet 5.5 ($2/$10 per M): input cost = $0.057, output cost = $0.048. Total = $0.105. The user’s 500-token message accounts for $0.001 of that — less than 1%.

Cross-Provider Token Cost Comparison {#cross-provider-token-cost-comparison}

Comparing effective cost requires normalizing for tokenizer differences. The table below shows cost per 1,000 words of English text (approximately 1,300 OpenAI tokens), factoring in each provider’s tokenizer inflation.

Model	Input / Output per M	Est. Tokens per 1K Words	Input Cost per 1K Words	Output Cost per 1K Words
GPT-6 Astra	$10 / $50	~1,300	$0.013	$0.065
GPT-6.1 Sol	$2 / $10	~1,300	$0.0026	$0.013
GPT-6 Luna	$0.10 / $0.50	~1,300	$0.00013	$0.00065
Claude Fable 5.1	$10 / $50	~1,690	$0.0169	$0.0845
Claude Opus 5.5	$4 / $20	~1,690	$0.00676	$0.0338
Claude Sonnet 5.5	$2 / $10	~1,690	$0.00338	$0.0169
Claude Haiku 5.5 (≤100K)	$0.10 / $0.50	~1,690	$0.000169	$0.000845
Gemini 4 Argon (intro)	$2 / $10	~1,250	$0.0025	$0.0125
DeepSeek-V4.1-Flash (peak)	$0.30 / $1.20	~1,300	$0.00039	$0.00156

Claude Haiku 5.5 (under 100K tokens) and GPT-6 Luna compete for the cheapest option. Haiku 5.5 at $0.10/$0.50 with 1.3x tokenizer inflation effectively costs slightly more per word than Luna at $0.10/$0.50 with no inflation. DeepSeek-V4.1-Flash at off-peak rates ($0.15/$0.60) sits just above both on per-token price.

For output-heavy workloads, the 5x output/input ratio common across providers means that a 1,000-word generated response costs 5x what a 1,000-word input costs. This is why classification and extraction tasks (short output) are dramatically cheaper than generation tasks (long output).

Practical Token Optimization Patterns {#practical-token-optimization-patterns}

1. Prompt compression. Remove redundant instructions, consolidate examples, and use terse phrasing. A system prompt rewritten from conversational English to telegraphic style often saves 30-50% of tokens with no quality degradation:

# Before (847 tokens)
You are a helpful assistant that helps users with their questions about
our product. You should always be polite and professional. When users
ask about pricing, refer them to the pricing page. When users ask
about features, provide detailed explanations...

# After (312 tokens)
Role: product support agent.
Rules:
- Pricing questions → link to /pricing
- Feature questions → detailed explanation
- Tone: professional, concise
...

2. Strategic model routing. Use cheap models for simple tasks and expensive models only when needed. A classifier running on GPT-6 Luna ($0.10/$0.50) can triage requests, routing only complex ones to Opus 5.5 ($4/$20):

3. Context window management. For multi-turn conversations, summarize older turns instead of carrying full history. A 50-turn conversation might carry 100K tokens of history. Summarizing turns 1-40 into 2K tokens and keeping only the last 10 turns verbatim can cut input costs by 80%.

4. Output length constraints. Setting max_tokens prevents runaway generation. A model asked to classify sentiment doesn’t need 4,096 output tokens — setting max_tokens: 10 ensures fast responses and minimal output cost.

5. Batch API usage. OpenAI and Anthropic offer batch APIs at 50% discount. If latency tolerance permits (typically 24-hour turnaround), batch processing halves the per-token cost across all models.

6. Cache-aware prompt design. Structure prompts so the static prefix (system prompt, tool definitions, few-shot examples) comes first and the variable portion (user query, retrieved context) comes last. This maximizes cache hit rates since providers match from the prefix.

# Cache-friendly ordering
messages = [
    {"role": "system", "content": STATIC_SYSTEM_PROMPT},      # Cached
    {"role": "user", "content": STATIC_FEW_SHOT_EXAMPLES},     # Cached  
    {"role": "user", "content": STATIC_TOOL_INSTRUCTIONS},      # Cached
    {"role": "user", "content": variable_user_query},           # Not cached
]

Tokenizer Edge Cases That Break Things {#tokenizer-edge-cases-that-break-things}

Tokenizers have surprising behaviors that cause real bugs in production systems.

Trailing whitespace. Some tokenizers treat trailing spaces as separate tokens. "Hello " and "Hello" might differ by one token. In structured output generation, stray whitespace in a JSON template can shift token boundaries and change model behavior.

Unicode normalization. The same visual character can have multiple Unicode representations. “é” can be U+00E9 (precomposed) or U+0065 U+0301 (decomposed). These tokenize differently. Text from different sources (web scraping vs database vs user input) may use different normalization forms, leading to inconsistent token counts for identical-looking text.

Code tokenization. Indentation style affects token count. Tabs are typically 1 token. Four spaces might be 1 or 2 tokens depending on the tokenizer. A Python file with 4-space indentation can produce 10-15% more tokens than the same code with tab indentation.

JSON structure overhead. The structural tokens in JSON ({, }, [, ], ", :, ,) add up. A JSON object with 20 key-value pairs incurs roughly 60-80 structural tokens before counting any content. For tool call responses, this overhead is unavoidable but should be factored into cost models.

Multilingual text. Chinese, Japanese, and Korean text typically produces fewer tokens on tokenizers trained with CJK-heavy data (Qwen, DeepSeek) and more tokens on English-centric tokenizers (OpenAI, Anthropic). A 1,000-character Chinese document might produce 400 tokens on Qwen but 800+ on GPT-6. For multilingual applications, tokenizer efficiency by language should drive provider selection.

Numbers and dates. Long numbers get split unpredictably. “1234567890” might become 2-4 tokens. Dates in different formats (2026-10-27 vs October 27, 2026 vs 10/27/26) produce different token counts. For data-heavy applications, choosing a compact date format saves tokens at scale.

Unicode normalization differences produce different token sequences from visually identical text.

Special tokens. Every provider injects special tokens for message boundaries, role markers, and control sequences. These are billed but invisible in the text. A simple single-turn request has approximately 5-15 overhead tokens from special markers. This is negligible for large requests but can represent 10-20% overhead on very short requests (e.g., single-word classification).

Token boundary effects on generation. Because models generate one token at a time, the tokenizer’s vocabulary affects what the model can produce atomically. If “JavaScript” is a single token in the vocabulary, the model produces it in one step. If it’s [“Java”, “Script”], the model must generate two sequential tokens correctly. This rarely causes visible issues but can affect spelling of unusual words or technical terms that span token boundaries.

Summary {#summary}

Tokenization is a billing mechanism, a capacity constraint, and a source of cross-provider incompatibility, all at once. The key points:

Token counts vary 30%+ across providers for identical text. Anthropic’s post-4.7 tokenizer inflates counts by ~30% versus OpenAI. Per-token price comparisons without normalizing for this are misleading.
Prompt length tiers create cost cliffs. GPT-6 Astra at 272K, Claude Haiku 5.5 at 100K, Grok at 200K, and Gemini 4 Argon at an unconfirmed threshold all double or quintuple rates when crossed.
Hidden tokens dominate cost. System prompts, tool definitions, conversation history, and thinking tokens typically account for 90%+ of billed tokens in agentic workloads. The user’s actual message is a rounding error.
Caching is a 5–20x cost reduction on static prompt components. Cache-aware prompt structure (static prefix, variable suffix) should be the default architecture for any application making repeated calls.
Output tokens cost 2.5–5x more than input tokens. Workload type (short-output classification vs long-output generation) is a larger cost driver than model choice in many cases.
Pre-flight token counting is possible for OpenAI (tiktoken, local, exact), Google (API or local SentencePiece), and open-source models (HuggingFace tokenizers). Anthropic requires an API round-trip or a 1.3x estimation multiplier.
Edge cases in tokenization — whitespace, Unicode normalization, number formatting, multilingual text — create real bugs and cost variance in production. Normalizing input text before tokenization improves consistency.

The practical advice: count tokens before sending them, structure prompts for cache hits, route requests to the cheapest model that meets quality requirements, and never compare per-token prices across providers without adjusting for tokenizer differences.

The Complete Guide to LLM Tokenization and Token Economics

The Complete Guide to LLM Tokenization and Token Economics

Table of Contents

How Tokenization Actually Works {#how-tokenization-actually-works}

BPE, SentencePiece, and the Tokenizer Zoo {#bpe-sentencepiece-and-the-tokenizer-zoo}

Token Counts Vary Across Providers — By a Lot {#token-counts-vary-across-providers}

Counting Tokens Before You Send Them {#counting-tokens-before-you-send-them}

Context Windows and Max Output: What the Numbers Mean {#context-windows-and-max-output}

How Billing Actually Works {#how-billing-actually-works}

Prompt Length Pricing Tiers {#prompt-length-pricing-tiers}

Cached and Prefilled Tokens {#cached-and-prefilled-tokens}

Hidden Token Costs: System Prompts, Tool Definitions, and Thinking {#hidden-token-costs}

Cross-Provider Token Cost Comparison {#cross-provider-token-cost-comparison}

Practical Token Optimization Patterns {#practical-token-optimization-patterns}

Tokenizer Edge Cases That Break Things {#tokenizer-edge-cases-that-break-things}

Summary {#summary}

Further Reading {#further-reading}