- Per-token list price is a small part of the real production cost picture.
- Context window usage and retry logic usually matter more than the headline rate.
- Prompt caching can cut real costs significantly for repeat-heavy workloads.
- The cheapest model on paper isn't always the cheapest once accuracy-driven retries are counted.
The list price is the least interesting number
Every provider publishes a per-million-token price, and teams compare those numbers like they're comparing gas prices. In practice, the list price is a small fraction of what determines the actual monthly bill — the bigger levers are how much context you send per request and how often a request needs to be retried.
Context length is where costs hide
Sending a full document as context on every request, when only a paragraph is relevant, multiplies cost for no benefit. Trimming context to what's actually needed — through better retrieval or summarization — often cuts real spend more than switching providers ever would.
Retries are the silent budget killer
If a model's output fails validation and the system retries automatically, a single user request can quietly become three or four billed requests. A model that's slightly more expensive per token but more reliable on the first try can end up cheaper overall than a 'cheap' model that needs constant retries.
Caching changes the math for repeat-heavy workloads
Workloads with a lot of repeated context — the same system prompt, the same reference document, across thousands of requests — benefit enormously from prompt caching features that some providers offer. For those workloads, the effective cost can drop well below the sticker price.
Benchmark on your own workload, not a leaderboard
Public benchmarks compare models on generic tasks that may have nothing to do with your specific use-case. The only cost comparison that matters is running your actual prompts, your actual data, and measuring real token usage and retry rates for each candidate before committing.