Managing Cost and Token Budgets for LLM APIs in Production: Caching, Batching, and Fallback Models

Cost Is a Product Decision, Not a Pricing-Page Lookup

Treat LLM spend the way you'd treat database load or CDN egress: a resource your architecture consumes, governed by decisions you make in code, not a bill that happens to you. The team that treats it as "just what the API costs" ships every feature at flagship-model, uncached, synchronous, unbounded-output settings — and then is surprised when spend scales linearly (or worse) with usage. The team that treats it as an engineering constraint builds routing, caching, and budget ceilings into the request path from day one, and their cost-per-active-user curve looks completely different.

This matters for a very practical reason: nobody else on your team is going to own this. Product wants the feature. Design wants it fast. Nobody's job description says "keep the token bill sane" until it becomes a fire. Own it before it's a fire.

What Actually Drives the Bill

Before touching any lever, understand what you're paying for, because the intuition most engineers start with is wrong in a specific, costly way.

Input and output tokens are priced differently, and output is typically the more expensive side of that ratio. This means the size of the context you send matters, but the size of what the model generates matters more per token. A chatty system prompt is a rounding error compared to a model that habitually writes three paragraphs when one sentence would do.

Long prompts compound cost in two ways. They cost more directly (you're paying for every token you send), and they slow down time-to-first-token, which hurts UX independent of dollar cost. If you're stuffing full conversation history, full document context, and a verbose system prompt into every call, you're paying full price for context the model may not even need for a given turn.

Re-read context is the silent multiplier. Every additional turn in a conversation, every retrieved document you paste into the prompt, every retry that resends the same payload — the model re-reads all of it, every time, from scratch. A ten-turn conversation with full history sent on every call doesn't cost 10x a single call; depending on how your context grows, it can cost meaningfully more, because turn 10 re-reads turns 1 through 9 in full.

Once you can point at your own traffic and say "this is expensive because of long re-sent context" versus "this is expensive because the model over-generates" versus "this is expensive because we default to the flagship model for everything," you know which lever to pull first.

Lever 1: Model Selection & Downshifting

This is the highest-leverage, lowest-effort change available to you, and most teams under-use it because the flagship model was the obvious first choice during prototyping and nobody revisited it.

The pattern: not every request needs your best model. Classification, extraction, formatting, short rewrites, and simple Q&A over a narrow domain are usually well within the capability of a smaller "mini" or "nano"-class model. Reserve the flagship-tier model for tasks that genuinely need stronger reasoning — multi-step planning, nuanced writing, ambiguous instructions.

A pragmatic rollout:

Audit your call sites. For each one, write down what the task actually requires.
For each call site, try the next tier down and run it against a fixed evaluation set of real (or representative) inputs. Compare output quality side by side, not just "does it look fine."
Where quality holds, downshift permanently. Where it doesn't, keep the stronger model but scope it tightly.
Don't forget previous-generation models can be considerably cheaper than the current flagship while still being more than adequate for constrained tasks — you don't always need the newest release.
Periodically re-benchmark. Model lineups and pricing change; a call site you locked to a mini model a year ago might now warrant re-evaluation in either direction.

The judgment call here is resisting the urge to use one model for everything because it's simpler to reason about. Simplicity in your codebase is not free — you're paying for it per token, per request, forever.

Lever 2: Prompt/Model Routing

Once you've accepted that different requests warrant different models, the next question is how you decide, at request time, which model handles which call.

Tiered routing is the simplest version: define small/standard/premium tiers and route by request type. A support-chat "what's my order status" query goes to the small tier; "help me resolve this billing dispute" goes to standard; anything flagged as complex or high-stakes goes to premium.

Query classification routing goes a step further — a fast, cheap classifier (or even a small LLM call) inspects the incoming request and decides which downstream model should handle it. The classification call itself needs to be cheap and fast, or you've just added latency and cost to save cost, which defeats the point.

Semantic routing uses embeddings to match incoming queries against known categories or historical query clusters, routing based on similarity rather than explicit rules. This is worth the complexity once you have enough traffic diversity that hand-written routing rules become unwieldy — not before.

Implementation-wise, this logic belongs in a routing layer, not scattered across your call sites. Whether that's a lightweight in-house module or a dedicated AI gateway, the key property is: your application code asks for "a completion for this task," and the routing layer decides which model and provider actually serves it. That indirection is what lets you change routing policy without touching every endpoint.

One caution: routing decisions have their own latency cost. A classification hop before every request adds round-trip time. Measure it. If your router adds more latency than the model downgrade saves, you've made UX worse to save money that a simpler static rule would have saved just as well.

Lever 3: Prompt Caching (Exact-Match)

Most modern provider APIs support some form of prompt caching, where a repeated prefix — a system prompt, a set of few-shot examples, a large static document — is cached on the provider side and billed at a steep discount on subsequent calls that reuse it, with a corresponding improvement in time-to-first-token. OpenAI's prompt caching offers significant cost reductions on repeated prefixes, and Anthropic's offering works similarly across its model lineup.

The architectural implication is simple but easy to get backwards: structure your prompts so static content comes first and dynamic content comes last. If your system prompt, tool definitions, and reference documents are identical across requests but you're interleaving them with per-request dynamic content, you break the ability to cache the static prefix. Reorganize so the stable content is a clean, unchanging prefix, and the variable part — the actual user message, the specific record being processed — comes after it.

This is close to free to implement (it's a prompt-structuring discipline, not new infrastructure) and it's worth doing before anything fancier, because it stacks with every other lever on this list.

Lever 4: Semantic Caching

Exact-match caching only helps when the prompt is byte-identical to a previous one. Semantic caching helps when prompts are similar in meaning but not identical — "what's your refund policy" and "how do refunds work" should probably hit the same cached answer.

The pattern, implementable with Redis and a vector-capable store (or Redis itself if it's configured for vector search) plus an embeddings model:

Embed the incoming query.
Search your cache for a previous query embedding above a similarity threshold.
On a hit, return the cached response (optionally with a freshness check).
On a miss, call the model, then store the query embedding and response for future hits.

from redis import Redis
import numpy as np

redis = Redis(host="localhost", port=6379)

def get_cached_response(query_embedding: np.ndarray, threshold: float = 0.92):
    # Use your vector store's similarity search here
    match, score = vector_search(redis, query_embedding)
    if match and score >= threshold:
        return match["response"]
    return None

Open-source tooling like GPTCache popularized this pattern if you want a starting point rather than building the similarity-search plumbing yourself. The threshold is the whole game: set it too loose and you serve wrong answers with false confidence; set it too tight and you rarely get a hit. Tune it against real query pairs from your own traffic, not a guess.

Set a TTL on cache entries. Semantic caching without invalidation eventually serves stale answers for content that's changed — pricing, policies, inventory. Match the TTL to how often the underlying answer actually changes, not to an arbitrary default. For more on cache strategy tuning with your full stack, see caching patterns for Next.js and FastAPI.

Lever 5: Batch API / Async Processing

If a workload doesn't need a synchronous response — nightly report generation, bulk document summarization, embedding backfills, content moderation sweeps — route it through your provider's batch processing offering instead of the standard synchronous endpoint. OpenAI's Batch API offers 50% cost discount compared to synchronous calls and processes within 24 hours; Anthropic similarly provides batch processing at reduced rates.

The FastAPI-shaped pattern: separate your "needs an answer now" call sites from your "needs an answer eventually" call sites explicitly. The latter go into a job queue, get submitted to the batch endpoint, and a worker polls for completion and writes results back to Postgres (or wherever the caller expects them). This is a straightforward extension of infrastructure you likely already have for background jobs — Celery, RQ, or a simple cron-triggered worker.

The judgment call: audit how much of your current synchronous traffic doesn't actually need to be synchronous. Teams are often surprised how much of their "real-time" LLM traffic is really "the user submitted this and would be fine checking back in ten minutes" traffic wearing a synchronous costume because that was the easiest way to build it initially.

Lever 6: Flex/Slow Synchronous Tiers

Between "instant and full price" and "batch and hours later" there's a middle tier some providers offer: a flex or slow tier that accepts synchronous-style requests but processes them with relaxed latency guarantees, in exchange for a meaningful discount off the standard rate.

This tier is the right fit for background UX work that isn't truly async (batch) but also isn't latency-critical — an in-app notification generated a few seconds later than usual, a search-result enrichment that can lag slightly behind the initial page render. Map your call sites against a simple question: does the user notice or care if this takes a few seconds longer? If not, this tier is close to free money.

Lever 7: Output Token Reduction & Structured Outputs

Given that output tokens are usually priced higher than input tokens, shrinking generation length is disproportionately effective per token saved.

Concrete tactics:

Set max_tokens deliberately, not as an afterthought. A generous, unbounded-feeling max length invites the model to ramble, and you pay for every token of ramble.
Ask for structured output — JSON or CSV with a fixed schema — instead of prose you then parse. Structured output is typically more compact than the equivalent explanation in natural language, and it's more reliable to consume downstream.
Prompt explicitly for conciseness. "Answer in one sentence" or "return only the JSON object, no explanation" measurably changes output length. Don't assume the model will infer brevity is wanted; say it.
Watch for redundant restating. Models asked to "explain your reasoning and then give the answer" will do exactly that, at double the output cost, when you often only need the answer.

Truncation via max_tokens is a blunt instrument — the model gets cut off mid-thought if you set it too low, which is worse than the cost problem you were solving. Pair a sane ceiling with explicit conciseness instructions and structured-output constraints, rather than relying on the ceiling alone.

Lever 8: Prompt Compression

Where output reduction shrinks what comes out, prompt compression shrinks what goes in — specifically, the parts of your input that are informationally redundant. Long few-shot example blocks, retrieved RAG context with overlapping or repetitive passages, and verbose instructions that repeat the same constraint three different ways are common offenders.

Dedicated prompt-compression techniques and tools (LLMLingua is a well-known example of this category) work by identifying and removing lower-information tokens while preserving the tokens that carry the meaning the model needs — in some cases reducing prompt length dramatically without materially harming task performance.

Even without adopting a dedicated compression library, you get a large share of the benefit by auditing manually: trim redundant few-shot examples down to the minimum that demonstrates the pattern, deduplicate overlapping RAG chunks before they hit the prompt, and cut instructional text that repeats itself. This is unglamorous work, and it's exactly the kind of thing that gets skipped when a feature ships under deadline pressure — which is precisely why it's usually still sitting there, uncut, in production.

Lever 9: Fallback Models & Reliability

Once you're routing across models and providers to save money, you've also built the infrastructure for reliability — the two problems share a shape.

Design fallback chains for at least two failure modes:

Rate-limit fallback: your primary model/provider is throttling you. Fall back to an alternate provider or a lower-tier model rather than failing the request outright.
Quality fallback: the cheap or fast model you routed to produced a response that fails a validation check (empty, malformed JSON, obviously off-topic). Escalate to a stronger model rather than serving garbage.

async def get_completion(prompt: str):
    try:
        return await call_primary(prompt, model="primary-model")
    except RateLimitError:
        return await call_fallback(prompt, model="fallback-model", provider="alt-provider")
    except Exception:
        return await call_fallback(prompt, model="stronger-model")

Lever 10: Budget Enforcement & Rate Limiting

Implement budget enforcement at multiple levels:

Per-user/per-team quotas, tracked in Redis with a simple counter-and-TTL pattern (reset daily or monthly, whatever matches your billing cycle).
Hard limits that reject requests outright once a ceiling is hit, and soft limits that trigger a warning, a downgrade to a cheaper model, or a notification, before the hard limit bites.
Cost-based rate limiting, where the limit is expressed in dollars or tokens rather than request count — a request to a flagship model with a huge context should consume more of the budget than a cached hit against a mini model.

import redis
from datetime import timedelta

r = redis.Redis()

def check_and_increment_budget(user_id: str, cost: float, daily_limit: float) -> bool:
    key = f"budget:{user_id}:daily"
    current = float(r.get(key) or 0)
    if current + cost > daily_limit:
        return False
    r.incr(key, cost)
    r.expire(key, timedelta(days=1))
    return True

Observability & Proving It Worked

Tie all of this together with cost-aware observability. Log the model, provider, token counts (input and output separately), cache hit/miss status, and total cost for every completion request. Aggregate by model, call site, user, time bucket. This observability setup lets you measure the impact of routing changes—for instance, shifts in flagship-model traffic share before and after an intervention—and makes the case to your team that the work was worth the sprint.

Set up a cost-per-task dashboard: classification requests should cost $X on average, search enrichment should cost $Y. Alert when actual cost drifts above expected. Most cost explosions are slow enough that a 2x threshold on an aggregate metric catches them before they become line-item problems.

For a production-ready full-stack implementation combining these patterns, see damianhodgkiss.com for additional tutorials on instrumenting FastAPI/Next.js stacks, Redis caching discipline, and cost-governance architecture.