Reducing LLM Cost and Latency in Production

Why LLM cost and latency need a budget

Most LLM prototypes are built with one goal: make the answer good. Cost and latency only become visible when real users arrive. A request that costs a fraction of a cent in a demo can become a five-figure monthly bill at scale, and a response that takes eight seconds feels fine to a developer but loses users in a product. The fix is to treat both as first-class requirements with explicit budgets, exactly as you would treat uptime.

Start by writing the budget down. For a support assistant it might be: median response under three seconds, 95th percentile under eight seconds, and an average cost below one cent per resolved conversation. Numbers like these turn vague worry into engineering work, because every optimization can be judged against them. Without a target, teams either over-optimize a cheap path or ignore an expensive one.

• Define a latency target for time to first token and for full completion separately.
• Define a cost ceiling per request, per user, and per day.
• Decide which quality metric you refuse to trade away before you begin cutting.

Measure before you optimize

Cost and latency come from a small number of places, and guessing which one is wrong wastes weeks. Log every model call with the model name, input tokens, output tokens, latency, cache status, and the feature that triggered it. Aggregate by feature and by prompt version. You will usually find that a handful of calls produce most of the spend, often a large system prompt sent on every turn or a retrieval step that stuffs far more context than the model uses.

Latency has more components than people expect: network time, queueing at the provider, time to first token, generation speed, and any retrieval, tool call, or validation you add around the model. Break a slow request into a timeline and the culprit is often not the model at all. A slow vector query, a sequential chain of calls that could run in parallel, or a retry after a timeout can dominate.

• Track tokens in and tokens out separately, because output tokens are slower and usually cost more.
• Report percentiles, not averages. Averages hide the slow tail that users actually complain about.
• Tag each call with a prompt version so you can tell whether a change helped.

Route work to the right model

Not every request needs your most capable model. Classification, extraction, routing, and short rewrites are often handled well by smaller, cheaper, faster models, while multi-step reasoning and difficult writing justify a larger one. Model routing means sending each task to the smallest model that meets the quality bar for that task.

A practical pattern is a cascade. Send the request to a small model first, check the result with a cheap validator or a confidence signal, and escalate to the larger model only when the check fails. Another is static routing by feature: the summarizer uses one model, the planning step uses another. Whichever you choose, prove it with an evaluation set. A router that saves forty percent but quietly drops accuracy on hard cases is a regression, not an optimization.

Keep a fallback path as well. Provider outages and rate limits happen, and a secondary model turns an incident into a slight quality dip instead of downtime.

• Route by task type first, then refine with a confidence check.
• Evaluate the cascade end to end, not each model in isolation.
• Log which model served each request so you can audit the split.

Control the size of the context

Input tokens are the easiest lever. Every token you send costs money and adds processing time, and long prompts also dilute the model's attention. Audit what you actually send. System prompts grow by accretion, each incident adding another rule, until a thousand-token instruction is repeated on every call. Rewrite it, remove duplicated rules, and move rarely needed guidance into retrieved snippets that appear only when relevant.

For retrieval-based systems, send fewer and better chunks. Retrieve a wide candidate set, rerank, and pass only the top few to the model. Our guide on chunking strategy explains how better boundaries reduce the amount of text needed per answer. For conversations, summarize old turns and keep the raw history in storage rather than resending it, and cap tool outputs so a verbose API response cannot flood the window.

• Trim and version system prompts, and delete rules that no longer earn their tokens.
• Pass reranked evidence, not every retrieved chunk.
• Truncate or summarize tool results before they re-enter the prompt.

Use caching where it is safe

Caching is the largest saving when traffic repeats. Provider-side prompt caching reduces the cost and latency of a long, stable prefix such as a system prompt or a reference document, provided you place the unchanging content first and the variable content last. Application-level caching stores whole responses for identical or near-identical requests.

Be careful with semantic caches that return an answer for a merely similar question. They are powerful for frequently asked, low-risk questions and dangerous for anything personalized, time-sensitive, or permission-dependent. Key the cache on everything that changes the answer: user or tenant, language, document version, and prompt version. Set expiry that matches how quickly the underlying facts change, and never cache an answer that contained private data across users.

• Order prompts so the stable prefix comes first and can be cached.
• Include tenant, version, and language in every cache key.
• Track the hit rate; a cache that rarely hits is complexity without benefit.

Make it feel and be faster

Streaming is the cheapest way to improve perceived latency. Users start reading as soon as the first tokens arrive, so a ten-second answer that streams feels far better than a six-second answer that appears all at once. Pair streaming with progress indicators for tool calls so the interface never looks frozen.

To reduce real latency, run independent steps in parallel, such as retrieving from two sources or calling two tools at once. Limit output length with a clear instruction and a sensible maximum, since generation time grows with every token. Use structured output to avoid long free-form explanations when a short field would do. Set timeouts and retry budgets carefully: an unbounded retry loop can multiply both cost and delay during an incident. For work that does not need an immediate reply, such as bulk classification or nightly summaries, use batch endpoints, which trade speed for a lower price.

• Stream tokens to the interface and show tool progress.
• Parallelize independent retrieval and tool calls.
• Move non-urgent work to batch processing.

A practical optimization order

Work from the safest and most valuable change to the riskiest. First instrument the system so you can see where the money and time go. Second, shrink prompts and context. Third, add prompt caching and streaming. Fourth, parallelize and cap output. Fifth, introduce routing and cascades with an evaluation set guarding quality. Finally, consider fine-tuning a smaller model for a narrow, high-volume task once you have data proving the need.

After each step, rerun your evaluation and compare cost, latency, and quality together. Our LLM evaluation guide describes how to build that regression set. An optimization is only real if quality holds, and the only way to know is to measure it every time.

• Instrument, then trim, then cache, then parallelize, then route.
• Change one thing at a time and keep the before-and-after numbers.
• Re-run quality evaluations after every cost change.