Token Optimization Is a Product Constraint, Not a Prompt Trick

Teams discover tokens the hard way: a feature that felt magical in a demo becomes a monthly bill, a timeout, or a quality cliff the week after launch. Token optimization is usually framed as “prompt engineering.” In products, it is closer to capacity planning.

As an AI product engineer, I treat tokens like any scarce resource — CPU, bandwidth, or seats. The question is not “how do I stuff more into the context?” It is “what is the cheapest reliable path from customer value to accepted output?”

Tokens are three budgets at once

Every token you send or receive burns:

1. Money — especially with tool-heavy agents and long transcripts
2. Latency — time-to-first-token and total completion time shape UX
3. Attention quality — long contexts dilute the model’s focus; more is not always better

If your feature cannot state its budget in those three dimensions, you do not have a product design — you have a hope.

A practical worksheet I use before shipping an AI surface:

| Question | Example answer | |---|---| | What is a successful outcome worth? | One accepted task extraction | | Max cost per success? | $0.08 | | Max p95 latency? | 8s interactive / 90s async | | What can be precomputed? | Speaker diarization, chunk embeddings | | What must stay fresh? | Last meeting only |

The product anti-pattern: “just add the transcript”

Meeting intelligence is the classic trap. A one-hour transcript can be tens of thousands of tokens. If you also add CRM history, prior tasks, and a system prompt novel, you have already lost before the model reasons.

Better product shapes:

• Summarize once, reason many times — store structured summaries with evidence pointers
• Retrieve narrowly — pull the 3–8 chunks that matter for this question
• Stage the pipeline — cheap models / heuristics first; expensive models only on hard slices
• Async by default for heavy jobs — interactive UI waits on status, not on the full generation

This is not only cheaper. It is often more accurate because the model sees a curated problem, not a haystack.

A hierarchy of token interventions

I apply interventions in this order — cheapest product leverage first:

1. Do less work

• Don’t call the model if a deterministic rule solves it
• Don’t re-extract what is already in canonical state
• Cache stable system instructions and tool schemas (provider prompt caching where available)

2. Shrink inputs intelligently

• Truncate with structure: keep headers, decisions, action items; drop filler
• Use extractive compression before abstractive summarization when provenance matters
• Prefer tables/JSON over prose for repeated machine-readable fields

3. Route models

Not every step needs your strongest model:

• Classification / routing → small/fast
• Structured extraction → mid-tier with strict schema
• Ambiguous synthesis → frontier model
• Embeddings → dedicated embedding model, not chat completions

Routing is a product decision: pay for judgment where users feel pain; economize elsewhere.

4. Control outputs

• Cap max_tokens to the real UI need
• Ask for structured JSON, not essays
• Stream for perceived latency; don’t generate paragraphs nobody will read

5. Only then: micro-optimize prompts

Prompt word-shaving helps. It rarely saves a drowning architecture.

Measuring what matters

Dashboards that only show “tokens used” train the wrong instinct. I track:

• Cost per successful outcome (not per request)
• Human acceptance / edit rate (quality signal)
• Retries and empty tool results (waste amplifiers)
• Context size distribution (find the whale requests)
• Cache hit rate when using prompt caching

A feature can look efficient on tokens while failing users — or look expensive while being the only path that earns trust. Optimize for accepted outcomes per dollar.

Design patterns that keep products honest

Evidence pointers instead of paste. Store meeting_id + timestamp span and retrieve on demand. The model reasons over the slice; the UI cites the source.

Two-pass extraction. Pass 1: cheap model proposes candidates. Pass 2: stronger model verifies only contested fields. Tokens go where uncertainty is.

Budget-aware agents. Agents that cannot see remaining token/tool budgets will happily burn the account. Put budgets in state and force needs_review when exhausted.

Product-facing limits. “We analyze the last N meetings by default” is better UX than silent degradation when the context overflows.

Where this showed up in my work

In professional intelligence pipelines, the expensive mistake is re-sending entire transcripts through every enrichment step. The durable approach:

1. Ingest audio/text once
2. Produce structured evidence artifacts
3. Run later features against structured state + selective retrieval
4. Keep frontier models for judgment calls humans reject most often

That architecture is token optimization and product clarity. Users understand what the system knows. Engineers can reason about cost. Ops can debug which stage blew the budget.

Trade-offs I still make deliberately

• I will spend more tokens on provenance-rich extraction when wrong answers destroy trust.
• I will spend fewer tokens on chatty assistants that do not write durable state.
• I accept slightly higher latency async if interactive quality stays sharp.

Token optimization is not austerity. It is allocating scarcity toward the moments that define the product.

Checklist before you ship an AI feature

1. Write the three budgets: money, latency, quality.
2. Identify what can be precomputed or cached.
3. Replace “full document in prompt” with retrieve-or-summarize.
4. Route model tiers by step difficulty.
5. Cap outputs to UI reality.
6. Instrument cost per accepted outcome.
7. Add a kill switch / budget breaker for agents.

If you treat tokens as a product constraint from day one, you ship features that survive the first invoice — and the first angry user who waited 40 seconds for a wrong answer.

That discipline is part of owning AI products end to end: design within limits, ship the reliable slice, and keep optimizing after deploy with real telemetry — not vibes.