Treat the context layer as the services, data, and tools around the model, not as a framework feature. Jerry Liu defines it as anything surrounding the model that it uses to do things: documents, the web, structured data, connected tools, and a system of record. Scope the task by naming which of those sources it needs, and the minimum permission on each, before you change the model.
Jerry Liu is co-founder and CEO of LlamaIndex. The conversation aired in June 2026 on Chain of Thought episode 61.
What are the five sources of context?
The five labels are a way to inspect one task's inputs. They overlap, and they are not a required architecture.
- Documents. PDFs and PowerPoints hold information the agent cannot use until something reads it off the page. He says those files need new-age OCR to read the page, and that using a frontier model as the OCR step is a different, worse fit for cheap high-accuracy parsing.
- The web. Search and crawling supply live information past the training cutoff. A fetched page still needs a check for whether it is current.
- Structured data. SQL databases and similar stores hold records that can ground an answer. Decide which records the task needs, and which permissions reach them.
- Connected tools. Liu gives MCP connections to Confluence and Salesforce as examples. Those connectors are providers of context. Reading a record and taking an action are separate decisions.
- Systems of record. The agent needs a place to store state and retrieve it later. Liu returns to this when he talks about memory.
A single product can fill more than one label. Salesforce can be the connected tool and the system of record in the same task. A PDF starts as a document and, once extracted into a store the agent queries, becomes part of the record.
"And so when I think of the context layer, it's literally anything surrounding the model, like the set of services that it has access to in order to do things."
Jerry Liu, co-founder and CEO of LlamaIndex, on Chain of Thought ep 61
The AI Framework Era Is Over: … - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts
Podcast Episode · Chain of Thought | AI Agents, Infrastructure & Engineering · June 3 · 53m
podcasts.apple.com How do you scope an agent's credentials to one task?
Write the task, list the sources it must touch, and grant the smallest permission that lets that step finish. Read access and write access stay on different lines.
Liu's framing is a task specification plus the external data that makes the task specific enough to solve. A single sentence is often too thin. In his example of preparing for a customer call, the agent also needs the previous 5 to 10 transcripts and the recent email thread. Those lookups are part of the task boundary.
The developer work has moved up from wiring Python classes around model providers. Liu describes general agents, including Claude Code, where you define a natural-language set of skills, an MCP tool, and the task, then program an engine that is already running. On a given ticket, that means the source list and the permissions, not a new abstraction for the loop.
Why does sensitive document work sit at 95%?
For sensitive use cases, Liu's operating bar is 95% plus, with human review on some of the extracted fields. He says frontier models are not bad at medium-complexity documents, but that gives a deceptive 80% accuracy over a corpus. 20% of documents will hallucinate a number or represent a table wrong. That gap disrupts the downstream agent workflow and makes straight-through automation a little harder to trust.
He also argues those models are tuned for high-intelligence tasks such as coding and reasoning. Parsing a million PDFs needs accuracy and low cost together, which is a different target. The document-heavy verticals he names are legal, financial services, insurance, manufacturing, healthcare, education, government, and the tech versions of those industries. Workflows he cites include onboarding, claims, invoices, and KYC.
A minimal sketch to adapt, not a drop-in library.
def gate_sensitive_fields(fields, min_confidence): # Hypothetical gate before an agent writes extracted values. # min_confidence is a cutoff you pick on labeled documents so that # accepted sensitive fields clear your accuracy bar; a 0.95 score # is not the same thing as 95% accuracy. for field in fields: if field.sensitive and field.confidence < min_confidence: return "human_review" return "continue" "And so a lot of these require really high accuracy, 95% plus, especially in more sensitive use cases, with human-in-the-loop review on some of the extracted data."
Jerry Liu, co-founder and CEO of LlamaIndex, on Chain of Thought ep 61
How should you store agent memory?
Treat memory as persistent context in a system of record the agent can update and read back. Liu says he is still forming his view. His simple versions are a row in a database, an entry in a CRM or Notion, or a small set of files on disk. He points to an internal wiki of files, which a coding agent can build and reopen, as something a general user can start with now.
He separates that from heavier designs: graphs, entity relationships, and synthesized summaries. He says he has not spent much time building those, and he leaves the more advanced primitives to people who have. For a first implementation, pick one store, write the task's state into it, and load that state at the start of the next session.
When a recommendation is wrong, check the evidence and tool results the agent received before you change the model, the instructions, or the retrieval step. A stale document can still produce a bad recommendation when the model follows its instructions.
"Yeah, I think memory is basically just a form of I guess like persistent context."
Jerry Liu, co-founder and CEO of LlamaIndex, on Chain of Thought ep 61
FAQ
Which verticals have the most context locked in documents?
Liu names legal, financial services, insurance, manufacturing, healthcare, education, government, and the tech versions of those industries. He describes pools of millions of unstructured documents and projects that automate work humans used to do, including onboarding, claims, invoices, and KYC. His company does not typically index code bases; he leaves those to coding agents.
Can I rely on a frontier model to parse all my documents?
Liu says models in this class are not bad at medium-complexity documents, and that the result can look like 80% accuracy over a corpus. He puts the miss rate at about 20% of documents, including a hallucinated number or a table represented wrong. He adds that frontier models are tuned for coding and reasoning, so they are a weak fit when the job is extremely accurate, cheap parsing of a million PDFs.
What did Liu mean when he said the framework era is over?
He still counts web frameworks and higher-level abstractions as relevant. What matters less is importing Python classes to wrap model providers and hand-building agent internals, because those patterns have solidified into general agents such as Claude Code and Manus. The layer he describes above that is skills, MCP tools, and programming an engine that is already running.
Takeaway
- List the five sources for the task: documents, web, structured data, connected tools, systems of record.
- Separate read permission from write permission on every connected tool.
- Liu's bar for sensitive cases is 95% plus accuracy, with human review on some of the extracted data.
- On a bad result, inspect the evidence and tool results the agent received before changing the model.
- Store memory as persistent state in a system of record, including a simple set of files.
The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
