Reducing LLM Costs in Production: Token Budgets, Caching, Context Management, and Model Routing

Teams discover tokens the hard way: a feature that felt magical in a demo becomes a monthly bill, a timeout, or a quality cliff the week after launch. Token optimization is usually framed as “prompt engineering.” In products, it is closer to capacity planning.

As an AI product engineer, I treat tokens like any scarce resource — CPU, bandwidth, or seats. The question is not “how do I stuff more into the context?” It is “what is the cheapest reliable path from customer value to accepted output?”

Tokens are three budgets at once

Every token you send or receive burns:

1. Money — especially with tool-heavy agents and long transcripts
2. Latency — time-to-first-token and total completion time shape UX
3. Attention quality — long contexts dilute the model’s focus; more is not always better

If your feature cannot state its budget in those three dimensions, you do not have a product design — you have a hope.

A practical worksheet, with example answers shown only to illustrate the questions (these are not measured results):

The product anti-pattern: “just add the transcript”

Meeting intelligence is the classic trap. A one-hour transcript can be tens of thousands of tokens. If you also add CRM history, prior tasks, and a system prompt novel, you have already lost before the model reasons.

Better product shapes:

• Summarize once, reason many times — store structured summaries with evidence pointers
• Retrieve narrowly — pull the 3–8 chunks that matter for this question
• Stage the pipeline — cheap models / heuristics first; expensive models only on hard slices
• Async by default for heavy jobs — interactive UI waits on status, not on the full generation

This is not only cheaper. It is often more accurate because the model sees a curated problem, not a haystack.

A hierarchy of token interventions

I apply interventions in this order — cheapest product leverage first:

1. Do less work

• Don’t call the model if a deterministic rule solves it
• Don’t re-extract what is already in canonical state
• Cache stable system instructions and tool schemas (provider prompt caching where available)

2. Shrink inputs intelligently

• Truncate with structure: keep headers, decisions, action items; drop filler
• Use extractive compression before abstractive summarization when provenance matters
• Prefer tables/JSON over prose for repeated machine-readable fields

3. Route models

Not every step needs your strongest model:

• Classification / routing → small/fast
• Structured extraction → mid-tier with strict schema
• Ambiguous synthesis → frontier model
• Embeddings → dedicated embedding model, not chat completions

Routing is a product decision: pay for judgment where users feel pain; economize elsewhere.

4. Control outputs

• Cap max_tokens to the real UI need
• Ask for structured JSON, not essays
• Stream for perceived latency; don’t generate paragraphs nobody will read

5. Only then: micro-optimize prompts

Prompt word-shaving helps. It rarely saves a drowning architecture.

Measuring what matters

Dashboards that only show “tokens used” train the wrong instinct. I track:

• Cost per successful outcome (not per request)
• Human acceptance / edit rate (quality signal)
• Retries and empty tool results (waste amplifiers)
• Context size distribution (find the whale requests)
• Cache hit rate when using prompt caching

A feature can look efficient on tokens while failing users — or look expensive while being the only path that earns trust. Optimize for accepted outcomes per dollar.

At the request level, a useful first approximation is:

estimated_model_cost =
  input_tokens × input_price_per_token
  + output_tokens × output_price_per_token

cost_per_accepted_outcome =
  total_model_cost_for_cohort / accepted_outcomes

Use the active provider's rate card and record the model/version with each measurement; prices and cached-token treatment vary by provider and change over time. Include retries and failed runs in cohort cost. Token count alone does not capture tool charges, storage, or the engineering cost of added routing complexity.

For a useful cohort comparison, persist a small usage record alongside the workflow outcome rather than retaining full prompts by default:

{
  "workflow_id": "opaque-id",
  "model_snapshot": "provider/model-version",
  "prompt_version": "sha256:…",
  "input_tokens": 8120,
  "cached_input_tokens": 6400,
  "output_tokens": 420,
  "attempts": 2,
  "outcome": "accepted"
}

Use provider-reported usage fields where available and version the price calculation separately. Keep identifiers opaque, apply retention limits, and avoid logging sensitive prompt content just to explain a cost change. Compare cohorts on the same task mix; a cheaper cohort with easier inputs is not an optimization result.

Design patterns that keep products honest

Evidence pointers instead of paste. Store meeting_id + timestamp span and retrieve on demand. The model reasons over the slice; the UI cites the source.

Two-pass extraction. Pass 1: cheap model proposes candidates. Pass 2: stronger model verifies only contested fields. Tokens go where uncertainty is.

Budget-aware agents. Agents that cannot see remaining token/tool budgets will happily burn the account. Put budgets in state and force needs_review when exhausted.

Product-facing limits. “We analyze the last N meetings by default” is better UX than silent degradation when the context overflows.

Where this showed up in my work

In professional intelligence pipelines, the expensive mistake is re-sending entire transcripts through every enrichment step. The durable approach:

1. Ingest audio/text once
2. Produce structured evidence artifacts
3. Run later features against structured state + selective retrieval
4. Keep frontier models for judgment calls humans reject most often

That architecture is token optimization and product clarity. Users understand what the system knows. Engineers can reason about cost. Ops can debug which stage blew the budget.

Trade-offs I still make deliberately

• I will spend more tokens on provenance-rich extraction when wrong answers destroy trust.
• I will spend fewer tokens on chatty assistants that do not write durable state.
• I accept slightly higher latency async if interactive quality stays sharp.

Token optimization is not austerity. It is allocating scarcity toward the moments that define the product.

Checklist before you ship an AI feature

1. Write the three budgets: money, latency, quality.
2. Identify what can be precomputed or cached.
3. Replace “full document in prompt” with retrieve-or-summarize.
4. Route model tiers by step difficulty.
5. Cap outputs to UI reality.
6. Instrument cost per accepted outcome.
7. Add a kill switch / budget breaker for agents.

If you treat tokens as a product constraint from day one, you ship features that survive the first invoice — and the first angry user who waited 40 seconds for a wrong answer.

That discipline is part of owning AI products end to end: design within limits, ship the reliable slice, and keep optimizing after deploy with real telemetry — not vibes.

A source-backed example: retrieve durable context instead of replaying it

In ProfMe, the product model stores structured professional memory with source references and retrieves context for later work. The design implication is useful even without a published cost benchmark: when the task is “prepare me for this person,” the candidate context can be the relevant decisions, commitments, and evidence references rather than the full history of every meeting.

question + user/workspace scope
             │
             ▼
       retrieve candidates ──► apply relevance, lifecycle, and scope rules
             │                                  │
             └───────────────► selected evidence and memory
                                                │
                                                ▼
                                   task-specific prompt budget
                                                │
                                                ▼
                                     bounded model response

This flow does not prove lower spend by itself. It creates a place to measure: tokens retrieved per task, useful evidence retained, answer quality, latency, and cost. If the prompt gets smaller but drops the one source that supports the answer, the optimization failed. ProfMe’s local retrieval report measures a narrow synthetic retrieval set and explicitly leaves answer faithfulness unmeasured; I do not use those results as a cost or quality claim.

The practical experiment is a paired comparison. Hold the query set and model fixed, compare full-history context with scoped retrieval, then inspect both token counts and task-level outcomes. Keep an error category for “required evidence omitted”; it makes the central tradeoff visible. Only promote a smaller context policy when savings are repeatable and omissions stay within the product’s accepted risk.

Limits and further reading

Prompt caching only helps when request prefixes and provider rules qualify; it is not a substitute for caching application data or controlling repeated work. A smaller or cheaper model can increase retries or human correction, so compare accepted outcomes and total latency rather than model-call price in isolation. Treat cost targets as product-specific inputs, not recommendations.

• OpenAI prompt caching guide
• OpenAI API pricing