AI models are getting smarter, tokens are getting cheaper, and agents are improving. But the context gap sits between the modern enterprise and agentic nirvana, keeping many companies from realizing the ROI they hoped for from their AI investments.
As with most problems we’ve seen in the AI era, tech companies are rushing to fill the gap. The New Stack recently spoke with Sri Desikan, an Elastic executive focused on search, to learn more about the AI context gap and how his company is tackling the challenge.
The context gap is easy to understand. As Desikan explains, “frontier models are getting better and better and better,” but if you don’t feed them proper context, “all that money you’re spending on tokens is going to waste.” Getting your agents the information they need, when they need it, quickly, is the ticket to moving from impressive demos to durable automation.
The enterprise is feeling the pinch. AI models can reach out to MCP servers to request information, but merely connecting a company’s corpus to a bridge isn’t enough. You can’t just “shove all this data into [a model’s] context window,” Desikan argues. Why? Because doing so is both computationally expensive (lots of tokens!) and slow.
So what do we need to do? Pre-conform corporate data into an AI index that knows where data is stored and how it relates to other data. Desikan described the process as creating a “container which stores metadata about everything in your enterprise.” Under this formulation, AI agents can access correct, useful information faster and with lower token burn (in some examples, sharply lower).
There’s another wrinkle worth considering. If you follow Elastic’s guidance and pre-compute your AI context (the index), you can “make sure that stale information” isn’t included, letting companies “know how the LLM is going to think.”
Of course, using AI to pre-prepare data for use by AI agents raises all sorts of privacy and agentic autonomy questions, which I got into with Desikan. Hit play if you want to go deeper on those issues. But looking at the conversation as a whole, I was encouraged by how well-thought-out industry approaches to novel problems seem; nothing that bedevils us today necessarily has to bother us next quarter.
The post The context problem: It’s what’s really breaking AI agents at scale appeared first on The New Stack.
