AI features often start cheap: a prototype, a few hundred requests a day. Then usage grows, prompts get longer, agents make more calls, and the monthly bill becomes a topic in management meetings.
The good news: most AI applications can cut their LLM costs significantly without lowering quality. Here are 12 techniques I use when optimising AI systems.
First: measure where the cost comes from
You cannot optimise what you do not measure. Log for every request:
- Model used
- Input tokens, output tokens, cached tokens
- Feature or workflow name
- User or customer (for per-customer costs)
- Latency and success/failure
Then sort by total cost. Usually one or two features or prompt templates account for most of the spend.
1. Use the smallest model that does the job
Not every task needs the most capable model. Classification, extraction, routing and short answers often work well with smaller, cheaper models. Test smaller models on your own evaluation set before switching.
2. Route requests by difficulty
Use a cheap model or simple rules to decide which model handles each request:
- Simple questions → small model
- Complex reasoning or long documents → larger model
- Known questions → cached answer or no model at all
3. Use prompt caching
Many providers offer prompt caching: if the beginning of your prompt (system instructions, tool definitions, reference documents) is identical across requests, cached input tokens are billed at a much lower rate and processed faster.
To benefit:
- Put static content (instructions, tools, examples, documents) at the start of the prompt
- Put dynamic content (the user’s question) at the end
- Keep the static part byte-for-byte identical; small changes such as a timestamp at the top break the cache
Check your provider’s documentation for minimum lengths, cache lifetime and pricing.
4. Shorten system prompts
System prompts tend to grow over time with rules nobody remembers adding. Review them: remove duplicates, outdated rules and long examples that do not improve results. Every token is paid on every request (unless cached).
5. Send less context
In RAG systems, more retrieved text is not always better. Tune:
- Number of chunks retrieved
- Chunk size
- Reranking to keep only the most relevant passages
Better retrieval often improves quality and lowers costs (see how to build a RAG chatbot).
6. Manage conversation history
Chat applications often send the entire conversation on every turn. Instead:
- Keep only recent turns in full
- Summarise older parts
- Remove tool outputs that are no longer needed
7. Limit output length
Output tokens are usually more expensive than input tokens. Set sensible maximum output lengths and ask for concise formats (for example JSON with specific fields instead of free text).
8. Use batch processing for non-urgent work
Many providers offer batch APIs at a discount for jobs that do not need an immediate answer: classifying thousands of products, generating descriptions overnight, analysing reviews. If the result can wait hours, use batch.
9. Cache full responses
For repeated questions (FAQ-style), store answers and reuse them. A semantic cache can match questions with the same meaning, not just identical text. Be careful with personalised or time-sensitive answers.
10. Reduce agent loops
Agents can make many model calls per task. Reduce cost by:
- Giving clearer tool descriptions so the agent picks the right tool first time
- Setting a maximum number of steps
- Moving fixed parts of the process into normal code or workflows (see what AI agents are)
11. Avoid unnecessary retries
Failed JSON parsing, timeouts and validation errors often cause automatic retries that double costs. Use structured output features, validate early and fix the causes of failures.
12. Set budgets and alerts
- Per-feature and per-customer budgets
- Alerts on unusual spikes (a bug in a loop can burn a monthly budget in hours)
- Rate limits per user to prevent abuse
Quick wins vs bigger changes
| Technique | Effort | Typical impact |
|---|---|---|
| Prompt caching | Low | High for long, repeated prompts |
| Shorter system prompts | Low | Medium |
| Output limits | Low | Medium |
| Batch API | Low to medium | High for bulk jobs |
| Smaller model for simple tasks | Medium | High |
| Model routing | Medium | High |
| Better retrieval | Medium | Medium to high |
| Semantic caching | Medium | Depends on repetition |
Do not sacrifice quality blindly
Every optimisation should be checked against an evaluation set of real tasks with expected results. A cheaper system that gives wrong answers costs more in the end through support tickets, lost customers or manual corrections.
Self-hosting open models?
Running open-weight models on your own GPUs can make sense at very high, steady volumes or for strict data requirements. For most businesses, the hidden costs (GPU hardware or rental, operations, scaling, updates) make API usage cheaper and simpler. Do the math with realistic utilisation.
Frequently asked questions
What is the biggest cost driver in most AI applications?
Usually long prompts sent on every request: big system prompts, full chat histories or too much retrieved context.
Does prompt caching reduce quality?
No. The model sees the same prompt; only billing and speed change.
How much can I save?
It depends on the application. Systems that have never been optimised often have significant savings available from caching, routing and prompt cleanup alone.
Want lower AI costs without lower quality?
I audit and optimise AI applications: prompts, model choice, caching, retrieval and monitoring. See my AI integration service or request an AI cost review.