Getting an AI application to work is one problem. Making it fast, cheap and reliable enough to ship is another, and it's the question I had been quietly asking since the first week of the Problem-First AI course by Aishwarya Reganti and Kiriti Badam.
Week 4 answered it, then pushed further into voice agents, multimodal systems and something called a "System One" model. This post is my attempt to turn those lessons into something you can use, whether you're building your first AI app or already running one in production.
Performance first, then cost and latency
Every AI application lives inside three constraints: performance, cost and latency. It's tempting to optimize all three at once. You can't, and the order matters.
Always optimize for performance first. Then cost and latency.
That sounds obvious until someone shows you a 40% cost cut in the middle of a sprint. Before shipping it, ask one question: did we rerun the eval set? Benchmarks and user surveys don't count. A change that makes things cheaper and faster while degrading quality is a regression that happens to cost less.
Latency has its own trap. Picture a support chat that averages 4.5 seconds against a 5-second target, while users still complain that replies feel slow. The average hides the slow requests. The users waiting 8 seconds at the tail don't care that most requests are faster. Always look at p50 and p95 together.
Three levers, in order
When something is too slow or too expensive, there are three places to look. Check them in this order.
1. Application level: prompts, retrieval and tools
This is almost always where the problem lives, and it's the cheapest place to fix it.
Context bloat is the classic example. Your agent gets slower and more expensive as a task goes on, and the traces show later calls still carrying search results and full file contents from steps that finished three turns ago. A bigger context window just makes that bloat more expensive. Trim before each call instead: drop old search results, shrink tool outputs and summarize old turns.
Prompt caching is another application-level lever, and it fails in a subtle way. If your system prompt starts with the current timestamp and the user's name, the prefix changes on every request and your cache hit rate will be near zero. Put what never changes first. Tool definitions, long instructions and reference documents go at the top, and the dynamic parts come after.
2. Model level: swap carefully
Moving every step of an agent to a smaller model can cut cost by 90%, right up until the eval set shows quality dropped. The better experiment is surgical. Use the small model only on steps like routing and classification, test each swap on its own, and keep the larger model on the steps that need reasoning. If the small model keeps failing, that's when to consider fine-tuning.
Fine-tuning was the clearest lesson of the week for me:
RAG is a knowledge augmentor. Fine-tuning is a behavior augmentor.
If the model needs to know your company's refund policy, and that policy changes every month, fine-tuning every month is the wrong answer. Use RAG. Fine-tuning shapes how a model responds, not what it knows. It should also never be your V1. Build and evaluate a prompted baseline first, and fine-tune only for specific, diagnosed failures that survive careful prompt work.
3. Inference level: last, not first
Self-hosting is a cost and control decision, not a capability decision. Running the same open-weight model on your own infrastructure doesn't give you better answers or faster replies by default. It gives you control over the deployment environment and, at scale, possibly a lower marginal cost. In exchange, you now own that infrastructure.
Voice agents: same model, different problems
The framing that made voice click for me: a voice agent is a text agent with speech models on both ends. The LLM in the middle doesn't change. Everything around it does.
There are two architectures:
- Cascade: speech-to-text, then the LLM, then text-to-speech
- Speech-to-speech: one model handles the whole pipeline
Most production voice agents today use a cascade. You can inspect the text at every handoff, audit every step, and swap the text-to-speech model per region when pronunciation matters. Speech-to-speech offers lower latency and more natural turn-taking, but you lose those inspection points. As one classmate put it, the inputs and outputs your user needs should decide the architecture.
Three problems surprised me:
- Silence. If a tool call takes eight seconds and the agent goes quiet, callers hang up. Fill the wait with a short line before the call ("let me pull that up for you") and progress updates while it runs. The latency stays the same, but the experience feels different. A note my cohort highlighted put it well: the hardest part of a voice agent isn't the AI, it's the awkward silence. Two seconds of quiet and the caller assumes the line is dead.
- Turn detection. A fixed 300ms silence threshold will cut off a caller who pauses mid-thought ("where is order... um... 1042?"). Semantic turn detection judges whether the thought is complete instead of just listening for silence. The agent also has to learn when "Great!" doesn't actually mean great.
- Interruptions. If a caller cuts in mid-answer, only the part that was actually spoken should go into the conversation history. Log the full generated response and the agent will say "as I mentioned" about things the caller never heard. The orchestration layer has to track what was delivered, not what was generated.
Multimodal: pay only for the pixels you need
Image resolution is a cost decision. Higher resolution means more tokens, so use only the resolution the task requires.
There's a catch, though. Vision transformers split images into patches, and fewer pixels mean fewer patches. Shrink receipt photos to save tokens and the totals can become unreadable, because small, dense text like numbers is the first thing to go. I hadn't thought about that failure mode before.
Where you can, start with text. For a product search where users type descriptions and get photos back, put captions in the text index first, and add image embeddings only if evals show captions falling short.
Video follows the same rule. Keyframe embeddings for each 30-second segment capture what's on screen but miss everything the presenter said. The most important missing piece in most video retrieval systems is a transcript with segment timestamps, so spoken explanations become searchable and results link back to the right moment.
System One models: not everything needs a frontier LLM
This is the concept I expect to use most. A System One model is a fine-tuned classifier that returns typed answers with probabilities, instead of a frontier LLM reasoning step by step.
Routing a million support tickets a month by asking a frontier LLM to return JSON is using a sledgehammer where you need a scalpel. A System One model doesn't reason longer or write richer explanations. It returns a structured prediction quickly and cheaply, which is exactly what high-volume routing needs.
Find the steps in your pipeline that genuinely require reasoning, and run everything else on something smaller and faster.
What my cohort added
One classmate pointed out that a model doing well on a benchmark won't necessarily do well with real callers who have accents, background noise and business-specific vocabulary. Benchmarks aren't evals. Your eval set comes from the actual failure modes in your system.
My own contribution, which the instructors highlighted, was that this week's complexity made it the most worth rewatching. The idea I kept coming back to: start with the problem, not the model.
The cheat sheet
| Topic | Rule of thumb |
|---|---|
| Order of attack | Application, then model, then inference |
| Performance vs. cost and latency | Performance first. Rerun evals before shipping any optimization |
| Latency | Check p95, not just the average |
| RAG vs. fine-tuning | RAG for knowledge, fine-tuning for behavior. Never fine-tune for V1 |
| Voice | Fill the silence. Log what was heard, not what was generated |
| Images and video | Start with captions and transcripts, add embeddings if evals call for them |
| System One models | Use them for high-volume classification |
Going deeper
If this sparked something, I highly recommend the Problem-First AI course by Aishwarya Reganti and Kiriti Badam at LevelUp Labs. Everything here came out of a week of assignments, quizzes, and cohort discussions built around one discipline: start with the problem.
This is part of an ongoing series on the Problem-First AI course. Previous post: MCP, A2A, Skills, Plugins and Harnesses: How the Pieces of an AI Agent Fit Together.
