AI Application Engineering: From LLM APIs to Agents, Harnesses, and Production Systems

An end-to-end guide to prompts, context engineering, multimodal AI, tool calling, hooks, approvals, compaction, memory, orchestration, and production architecture.

Introduction: AI Applications Are More Than Model Calls

Integrating a large language model (LLM) into an application is relatively straightforward. You send a request, receive a response, and display it in your interface.

Building a reliable AI application is a different challenge.

Real applications must manage context, execute tools, preserve state, handle interruptions, request approvals, recover from failures, and communicate progress to users. Some applications also need to process images, audio, video, documents, structured data, or real-time events.

As these requirements grow, the model becomes only one component of a larger system.

This article explores the complete landscape of AI application engineering, from model capabilities and SDK integration to agent loops, context management, harness engineering, and production operations.

We'll use the Vercel AI SDK for practical TypeScript examples where appropriate, while discussing concepts that also apply to OpenAI, Anthropic, and other model providers.

The goal isn't to build an exhaustive list of APIs. It's to understand how the pieces fit together, which problems they solve, and where each responsibility belongs.

1. Understanding the AI Application Stack

Before exploring individual features, it's useful to establish a mental model.

A production AI application can be understood as several cooperating layers.

1.1 Model layer

The model provides capabilities such as:

• Text generation and summarization
• Reasoning and problem-solving
• Image and document understanding
• Audio understanding and generation
• Structured output generation
• Tool selection
• Embeddings and semantic representations
• Video understanding, where supported

Not every model supports every capability. Availability, quality, limits, and pricing depend on the specific model and provider.

1.2 API and SDK layer

The provider API exposes model capabilities over a network interface. An SDK makes that interface easier to use through typed functions, request builders, streaming helpers, and provider-specific abstractions.

Examples include:

• Vercel AI SDK
• OpenAI API documentation
• Anthropic documentation

1.3 Application runtime

The runtime controls what happens around the model:

• Constructing prompts and context
• Selecting available tools
• Executing tool calls
• Managing agent loops
• Applying authorization rules
• Handling hooks and middleware
• Maintaining state
• Managing failures and retries
• Coordinating long-running tasks

1.4 Experience layer

The user interface communicates the runtime's behavior:

• Streaming text
• Tool-call progress
• Structured results
• Interactive components
• Clarification questions
• Approval requests
• Task status and errors

1.5 Operations and governance

Production systems additionally need:

• Evaluation and regression testing
• Observability and tracing
• Cost and latency controls
• Data privacy and retention policies
• Security boundaries
• Human oversight
• Recovery and auditability

A useful rule is that the model proposes, the runtime coordinates, tools perform work, and the application enforces policy.

That separation becomes increasingly important as AI applications become more autonomous.

2. How LLMs Actually Work in an Application

You don't need to understand every detail of transformer architecture to build AI applications, but you should understand the concepts that influence system design.

2.1 Models and inference

A model generates outputs based on an input and its learned parameters. In an application, the process of running the model to produce an output is called inference.

The input might include instructions, conversation messages, retrieved documents, images, audio, tool definitions, or structured data.

The output might be plain text, structured content, an audio response, or a request to invoke a tool.

The model doesn't automatically have access to your database, filesystem, internal APIs, or business systems. Those capabilities must be exposed through an appropriate integration.

2.2 Tokens and context windows

Models process text and other content through their underlying representations. For text, these are commonly measured in tokens rather than words or characters.

Tokens influence:

• Context-window capacity
• Input and output costs
• Time to generate a response
• How much conversation history can be included
• How many tool definitions and retrieved documents fit into a request

A context window is the amount of context a model can process for a request. Exact limits and accounting rules vary by model.

A larger context window doesn't automatically produce a better result. Irrelevant or contradictory information can make a request less effective even when it fits within the available capacity.

2.3 Messages and roles

Many conversational APIs represent interactions as messages with roles or equivalent content types.

Common roles include:

• System or developer instructions: establish behavior, constraints, and application-level rules.
• User: contains the user's request and supplied information.
• Assistant: contains the model's response or a request to use a tool.
• Tool result: contains information returned by an external operation.

The exact message structure differs between providers. Some APIs also support richer content blocks for images, audio, reasoning-related data, and tool interactions.

A conversation is not necessarily just an array of plain-text messages. It can be a sequence of different types of content and events.

2.4 Reasoning and extended thinking

Some models support reasoning-oriented capabilities or configurable thinking modes that can improve performance on complex tasks.

These can be useful for coding, planning, analysis, and multi-step tool use. However, reasoning capabilities differ across models and providers.

Application developers should distinguish between:

• The model's internal computation
• Any reasoning-related content exposed by the API
• The user-facing explanation
• The actual operations performed by tools

Do not assume that exposing internal reasoning is necessary for transparency. A useful application can instead show a concise explanation, sources, action history, and verifiable results.

Reasoning also has practical costs: more computation can mean increased latency and token usage. For example, Anthropic's extended-thinking documentation describes the interactions between thinking, tool use, streaming, and context management.

3. AI Capabilities Beyond Text Chat

One of the most important things to understand is that AI is not synonymous with generating text in a chat window.

Modern AI APIs expose multiple capabilities, and an application may combine several of them in one workflow.

3.1 Text generation

Text generation is the familiar capability behind chat assistants, summarization, rewriting, translation, classification, and content generation.

Typical use cases include:

• Answering questions
• Summarizing reports
• Drafting emails
• Explaining code
• Extracting information from documents
• Generating contextual recommendations

The output can be streamed incrementally or returned after generation completes.

3.2 Structured output

Sometimes an application needs a predictable data structure instead of free-form prose.

For example, a task classifier might need to return:

{ "category": "billing", "priority": "high", "requiresHumanReview": true } 

Structured output features let developers describe the expected shape using a schema or supported structured-generation mechanism.

This is useful for:

• Classification
• Data extraction
• Form population
• Report generation
• Workflow routing
• Producing typed UI data

Schema-constrained output can reduce formatting errors, but it does not guarantee that the values are factually correct or that an action is safe. Validate the result and enforce business rules independently.

3.3 Vision and image understanding

Vision-capable models can analyze images supplied through supported APIs.

Potential use cases include:

• Understanding screenshots
• Extracting information from charts
• Analyzing product images
• Interpreting diagrams
• Reading supported document images
• Helping users navigate a visual interface

Image support varies in accepted formats, resolution, detail handling, and token accounting.

Document understanding may also require OCR, layout analysis, PDF parsing, or a dedicated document-processing pipeline. A model's ability to understand an image does not mean it will perfectly extract every character or table.

3.4 Audio understanding and generation

Audio capabilities can support:

• Speech-to-text transcription
• Audio classification
• Spoken question answering
• Text-to-speech generation
• Voice assistants
• Spoken summaries
• Audio translation, where supported

A simple voice assistant may use a pipeline such as:

1. Capture audio.
2. Transcribe the speech or send audio to a model that can process it directly.
3. Interpret the request.
4. Invoke tools if necessary.
5. Generate a response.
6. Convert the response to speech.
7. Play the audio to the user.

Other architectures use speech-capable models that handle multiple stages together.

These designs have different latency, cost, and interaction characteristics. Real-time voice applications may also need interruption handling, turn detection, streaming audio, and session management.

3.5 Video and real-time perception

Video understanding can be built from supported video inputs, sampled frames, audio tracks, or model-specific video interfaces.

Potential use cases include:

• Summarizing recorded meetings
• Understanding demonstrations
• Analyzing a sequence of events
• Extracting information from training videos
• Interpreting visual changes over time

Developers must consider frame sampling, timestamps, audio synchronization, data volume, and processing cost. A model receiving selected frames does not necessarily understand every moment of a video.

3.6 Embeddings and semantic similarity

Embeddings represent content as numerical vectors that can be compared mathematically.

They are useful for:

• Semantic search
• Document retrieval
• Similarity detection
• Recommendation systems
• Clustering
• Retrieval-augmented generation (RAG)

For example, two documents may discuss the same concept using different vocabulary. A semantic search system can retrieve related documents even when the query doesn't share exact keywords.

Embeddings are not answers by themselves. They are representations used by retrieval and similarity systems.

3.7 Tool use and external actions

Models can request tools that expose capabilities such as:

• Searching internal documentation
• Querying a database
• Reading a file
• Calling a business API
• Creating a ticket
• Running a supported computation
• Searching the web

The model typically proposes a tool call and supplies arguments. The application or provider-managed execution environment determines what actually runs.

This distinction is fundamental: a tool call is not merely a piece of generated text, and describing an action is not the same as executing it.

3.8 Computer use and browser interaction

Some systems support computer-use or browser-use capabilities that let an agent interact with supported interfaces through screenshots, clicks, typing, or other actions.

These capabilities can enable:

• Browser-based research
• Interaction with legacy interfaces
• Repetitive UI workflows
• Testing user journeys
• Tasks involving applications without suitable APIs

Computer use introduces additional risks. An agent may misinterpret a screen, click the wrong control, or encounter untrusted content. Sensitive operations require strong execution boundaries, validation, and appropriate human oversight.

Where a stable API exists, a dedicated API integration is often more reliable than controlling the same system through its UI.

3.9 Model capabilities versus application features

It helps to classify features by where they belong.

Not every provider supports every feature, and similarly named features may have different semantics.

4. Prompt Engineering vs. Context Engineering

These terms are closely related, but they solve different problems.

4.1 Prompt engineering

Prompt engineering focuses on the instructions and framing given to a model.

A prompt may define:

• The task
• The expected tone
• Output format
• Constraints
• Examples
• Decision criteria
• Rules for handling uncertainty

For example:

You are a support assistant. Classify the incoming request into one of: - billing - technical_support - account_access Return only the requested structured result. If the information is insufficient, mark the result as uncertain. 

This tells the model how to approach the task.

4.2 Context engineering

Context engineering focuses on selecting and organizing the information available to the model at a particular moment.

This can include:

• System and developer instructions
• Recent conversation messages
• User preferences
• Retrieved documents
• Tool definitions
• Tool results
• Current workflow state
• Relevant historical decisions
• Output constraints and token budgets

The central question is not simply, "What should we tell the model?"

It is also, "What information should be available to the model for this specific decision?"

A model might have excellent instructions but still produce a poor result if it receives stale data, irrelevant documents, missing permissions, or contradictory conversation history.

4.3 Context assembly

A practical application often has a context-building stage before inference.

For example:

Incoming request | v Identify task and permissions | v Load relevant session state | v Retrieve relevant information | v Select available tools | v Assemble model context | v Run inference 

This process should be deliberate. Sending every available document, tool definition, and previous message to every model call is usually not a good default.

4.4 Context pollution

Context pollution happens when irrelevant, redundant, outdated, or contradictory information consumes attention and capacity.

Common causes include:

• Repeated instructions
• Large tool outputs that are no longer useful
• Unrelated conversation history
• Too many tool definitions
• Retrieved documents with weak relevance
• Old summaries that contradict current state

More context is not always better. The goal is to provide the most useful context for the current task.

4.5 Prompt caching

Prompt caching allows supported providers to reuse processing for eligible repeated prompt prefixes.

It can reduce latency and input-processing costs in appropriate workloads, especially when stable instructions or tool definitions are reused.

However, prompt caching is not the same as conversational memory or context compaction:

• Prompt caching reuses eligible repeated input processing.
• Memory preserves information across interactions.
• Compaction reduces the context carried forward.
• Retrieval selects information relevant to the current request.

These mechanisms can complement one another.

Reference: Anthropic prompt caching.

5. Working With AI SDKs and Provider APIs

SDKs simplify integration, but they do not eliminate the need to understand the underlying model and API behavior.

5.1 Provider SDKs

Provider-specific SDKs generally expose the capabilities and conventions of a particular provider.

They may offer direct access to provider-specific request parameters, response formats, streaming events, and specialized capabilities.

This is useful when an application needs precise control over a provider's API.

5.2 Unified AI SDKs

A unified SDK can provide common interfaces across multiple providers.

The Vercel AI SDK, for example, offers functions for text generation, streaming, tool calling, structured output, embeddings, and other supported AI tasks.

A basic TypeScript example:

import { generateText } from "ai"; import { openai } from "@ai-sdk/openai"; const result = await generateText({ model: openai("gpt-4.1"), prompt: "Explain context engineering in simple terms.", }); console.log(result.text); 

This illustrates a non-streaming text-generation request. The model identifier is an example; select a model that is available to your account and supports the desired operation.

The SDK provides a convenient interface, but your application still needs to handle authentication, errors, timeouts, cost controls, and appropriate data handling.

5.3 Streaming

For interactive applications, waiting for the entire response can create a poor experience.

Streaming allows the application to consume output incrementally.

import { streamText } from "ai"; import { openai } from "@ai-sdk/openai"; const result = streamText({ model: openai("gpt-4.1"), prompt: "Explain how an AI agent uses tools.", }); for await (const chunk of result.textStream) { process.stdout.write(chunk); } 

This example streams text to a server-side output. A web application would normally send the stream to the frontend using an appropriate response format.

Streaming can improve perceived responsiveness, but it doesn't necessarily reduce the total computation required to produce the final response.

Also, not every streamed event is user-facing text. Tool calls, structured data, reasoning-related content, and status events may require different handling.

5.4 SDK abstractions are not universal contracts

Two SDKs may expose similarly named concepts with different semantics.

For example:

• A lifecycle callback in one framework may not behave like a hook in another.
• A tool call may execute automatically in one configuration and require application handling in another.
• Structured-output constraints may differ by provider.
• Streaming events and interruption behavior may not be identical.

Treat SDK abstractions as implementation interfaces, not proof that every provider behaves identically.

References:

• Vercel AI SDK documentation
• OpenAI API documentation
• Anthropic documentation

6. Tool Calling: Connecting Models to Real Systems

Tool calling is one of the most important concepts in AI application engineering.

A tool exposes a defined operation that the model can request. The application decides how that request is validated, authorized, and executed.

6.1 Defining a tool

A tool generally includes:

• A name
• A description
• An input schema
• An execution handler, if execution happens in the current runtime
• An output or result representation

For example, a support assistant might have a get_order_status tool.

The model might request:

{ "orderId": "ORD-12345" } 

The application validates the arguments, checks whether the user can access that order, queries the appropriate service, and returns the result.

The model can then use that result to formulate an answer.

6.2 Tool calls versus tool execution

These are separate stages:

1. The model requests a tool.
2. The runtime receives the request.
3. The runtime validates the arguments.
4. The runtime checks permissions and policies.
5. The tool executes, if authorized.
6. The runtime captures the result.
7. The result is supplied to the model or presented to the user.
8. The model may continue or finish.

Some APIs and SDKs automate parts of this process. Others leave more of it to the application.

The important point is that the application must know which component is responsible for execution.

6.3 A practical tool example

import { generateText, tool } from "ai"; import { openai } from "@ai-sdk/openai"; import { z } from "zod"; const result = await generateText({ model: openai("gpt-4.1"), prompt: "Find the status of order ORD-12345.", tools: { getOrderStatus: tool({ description: "Retrieve the status of an order.", inputSchema: z.object({ orderId: z.string(), }), execute: async ({ orderId }) => { // Authenticate and authorize the request in a real application. // Query the order service here. return { orderId, status: "shipped", }; }, }), }, }); console.log(result.text); 

This is an illustrative integration pattern, not a complete production order service. The example handler returns a static result. A real implementation would validate access, handle service errors, and avoid trusting the model-supplied order identifier without authorization checks.

Check the current AI SDK tool-calling documentation for version-specific API details.

6.4 Tool routing and discovery

As the number of tools grows, deciding which tools to expose becomes an architectural problem.

Possible strategies include:

• Exposing a small, fixed tool set
• Selecting tools based on the user's task
• Loading specialized tools on demand
• Grouping tools by domain
• Using a router to identify the relevant subsystem
• Restricting tools according to the user's permissions

For example, a developer assistant might have access to GitHub, issue tracking, internal documentation, and deployment systems. A question about a pull request doesn't necessarily require loading every deployment and documentation tool.

Dynamic discovery can reduce context usage, but the discovery mechanism itself needs authorization and observability.

6.5 Parallel tool calls

Some tasks can be completed faster by running independent operations concurrently.

For example, an assistant might retrieve documentation and issue metadata at the same time.

Parallel execution requires attention to:

• Rate limits
• Resource limits
• Dependencies between operations
• Error handling
• Partial results
• Side effects

Independent reads are often good candidates for parallelism. Conflicting writes or operations with dependencies need more careful coordination.

7. Agent Loops, Workflows, and Orchestration

Tool calling alone doesn't make an application a complete agent system.

An agentic workflow usually involves repeated decisions and operations until a goal is reached or the system determines that it cannot continue.

7.1 The agent loop

A simplified loop looks like this:

Receive goal | v Assemble context | v Ask model for next step | v Tool call requested? / \ No Yes | | v v Finish Validate and authorize | v Execute tool | v Record result | +------> Continue loop 

The loop must have termination conditions. Otherwise, an agent can repeatedly call tools, consume resources, or fail to make progress.

Useful controls include:

• Maximum step counts
• Deadlines
• Token and cost budgets
• Tool execution timeouts
• Retry limits
• Cancellation
• Progress checks

7.2 Workflows versus agents

A workflow follows a predefined sequence of steps.

For example:

Receive support request | v Classify request | v Retrieve account information | v Draft response | v Human review | v Send response 

An agent has more flexibility in choosing the next step based on its current context and the results of previous actions.

A workflow is useful when the process is known in advance. An agent is useful when the path depends on information discovered during execution.

Many production systems combine the two: deterministic workflows establish boundaries, while an agent handles selected reasoning and tool-use steps.

7.3 Orchestration

Orchestration coordinates the execution of multiple steps or components.

It may handle:

• Step ordering
• Parallel execution
• Retries
• Branching
• Timeouts
• State transitions
• Human interruptions
• Failure recovery

Orchestration is not synonymous with a multi-agent architecture. A single agent can be orchestrated through a sophisticated workflow, and multiple agents can still operate within a relatively simple orchestration system.

7.4 Multi-agent systems

A multi-agent system divides responsibilities among multiple model-driven components.

For example:

• A research agent gathers evidence.
• An analysis agent evaluates the evidence.
• A coding agent prepares a change.
• A review agent checks the proposed result.

This can help when tasks have distinct contexts, capabilities, or responsibilities.

However, multiple agents also introduce coordination overhead, additional inference costs, more failure modes, and potential disagreement. They should be used when specialization offers a measurable advantage, not simply because the architecture permits it.

8. Hooks, Middleware, and Runtime Lifecycle

Hooks and middleware are frequently mentioned in AI engineering, but their meanings vary across frameworks.

A useful way to understand them is to consider the lifecycle of an AI request.

8.1 What are hooks?

A hook lets a runtime or application attach custom behavior to a defined lifecycle event.

Depending on the framework, relevant events might include:

• Before a model request
• After a model response
• Before tool execution
• After tool execution
• When an agent starts or finishes
• When a task fails
• When a user approval is required

Hooks can be used for logging, validation, metrics, policy checks, and other cross-cutting behavior.

The exact API and execution semantics are framework-specific. A hook in an AI SDK, a lifecycle callback in an orchestration framework, and a hook in a coding agent's runtime are not automatically interchangeable.

8.2 What is middleware?

Middleware wraps or intercepts operations to apply reusable behavior.

Examples include:

• Request logging
• Authentication
• Input validation
• Error transformation
• Rate limiting
• Model configuration
• Output processing

Middleware is often useful when the same behavior must apply consistently across many requests.

8.3 Hooks versus tools

A tool is an operation the model can request.

A hook is application or runtime logic associated with an event.

For example:

• searchDocumentation is a tool.
• Logging every documentation search is a lifecycle concern.
• Checking whether the current user may access a document is an authorization concern.
• Rejecting a forbidden operation is a policy-enforcement concern.

A hook may invoke application logic, but it should not be confused with a model-selected tool.

8.4 Hooks are not automatically security boundaries

A lifecycle callback is only useful for security if it reliably runs at the relevant execution boundary and cannot be bypassed by another path.

For sensitive operations, authorization should be enforced by the actual service or execution layer. UI restrictions or prompt instructions alone are insufficient.

9. Human-in-the-Loop: Approvals, Interrupts, and Clarifications

Autonomy is not always the right behavior.

Some actions should require explicit user authorization, and some tasks cannot continue until the application obtains additional information.

9.1 Approvals

An approval introduces a decision point before a proposed action proceeds.

Examples include:

• Sending an external email
• Deleting records
• Creating a production deployment
• Updating financial or account information
• Publishing content
• Running a destructive command

A safe execution flow looks like this:

Model proposes action | v Runtime validates arguments | v Policy checks required authorization | v Approval required? / \ No Yes | | | v | Pause execution | | | v | Present details | | | v | User approves? | / \ | No Yes | | | v v v Execute or reject action 

Approval is not simply a prompt instruction asking the model to be careful.

The runtime must actually block execution until the appropriate authorization is received.

An approval record should ideally identify the proposed action, its parameters, the relevant user, and the result of the decision. If the action changes between approval and execution, the system should validate it again.

9.2 Interrupts

An interrupt pauses a workflow because it needs something external before continuing.

The reason might be:

• User approval
• Missing information
• An unavailable dependency
• A required external event
• A manual review step

An interrupt should preserve enough state to resume safely, rather than forcing the entire task to restart.

9.3 Asking the user questions

Sometimes the model needs clarification before it can make a useful decision.

For example, a user might ask:

Schedule a meeting with the team next week.

The application may need to know which team, which day, and how long the meeting should last.

A good system should ask only the questions needed to proceed. It should also preserve the answers so the user doesn't have to repeat them.

Clarification and approval are different:

• Clarification gathers missing information.
• Approval authorizes a proposed action.
• Authentication establishes identity.
• Authorization determines which operations that identity may perform.

These distinctions are especially important in business applications.

9.4 Resumability

Long-running tasks may outlive an HTTP request, browser tab, or user session.

Resumable workflows need durable state, a way to identify the paused execution, and a controlled mechanism to continue from the appropriate point.

They should also account for duplicate events, expired approvals, cancellation, and partial side effects.

Resumability is therefore an application-runtime capability, not merely a feature of a chat interface.

10. Rendering AI Activity in the User Interface

The UI is part of the system's interaction model, not just a place to display generated text.

10.1 Chat messages versus execution events

A naive chat interface may store each response as a string.

An agent interface needs to represent richer information:

• Text deltas
• Tool calls
• Tool results
• Progress updates
• Approval requests
• Clarification questions
• Errors
• Completion state

These events should be modeled explicitly instead of being squeezed into a single text field.

10.2 Rendering tool calls

Suppose an assistant searches internal documentation.

The interface might show:

Assistant Searching internal documentation... Search results - Deployment guide - Service ownership - Incident playbook Summary The deployment guide describes... 

For an operation that changes data, the interface might show the proposed action, its target, the expected impact, and an approval control.

The frontend should render tool activity based on structured state and known event types, rather than trying to infer everything from natural-language messages.

10.3 Generative UI

Generative UI means that an AI workflow can produce structured information that the application maps to interactive interface components.

For example:

• A chart from structured data
• A comparison table
• A form asking for missing information
• A confirmation panel
• A list of search results
• A workflow status component

A safer design uses a defined set of UI component types and validated input schemas.

The model selects or supplies data for supported components; the application retains control over what can actually be rendered and executed.

10.4 Streaming and partial states

During a streamed response, the UI may need to distinguish between:

• Generating text
• Waiting for a tool
• Executing a tool
• Waiting for user input
• Awaiting approval
• Recovering from a failure
• Completed execution

Clear status information makes the system more understandable and helps users recognize when an operation is still running.

Reference: AI SDK UI documentation.

11. Context Compaction, Memory, and State

Long conversations and multi-step workflows accumulate information. Eventually, some information may no longer fit within the available context or may no longer be useful.

This is where context compaction, memory, and persistent state become important.

11.1 What is context compaction?

Compaction reduces the amount of information carried forward into subsequent model requests.

A compaction strategy may summarize older conversation history, remove obsolete tool results, retain important decisions, or transform detailed execution history into a smaller representation.

For example, a long coding task might accumulate:

• The original request
• Repository structure
• Hundreds of tool calls
• File contents
• Test output
• Error messages
• Intermediate plans
• Decisions and completed steps

The next model call may not need every intermediate result. It may need the current objective, important constraints, files changed, tests completed, unresolved failures, and next steps.

Compaction aims to preserve what matters while reducing irrelevant history.

11.2 Compaction is not ordinary summarization

A general summary might produce a readable overview.

A compaction process must preserve information needed for future execution.

A useful compacted state might include:

Objective: Fix the failing authentication integration test. Constraints: Do not change the public API. Completed: - Identified the failing test. - Updated the mock token validation. Verification: - Unit tests pass. - Integration test still fails. Important findings: The integration test uses an expired fixture token. Next action: Update the fixture and rerun the integration test. 

This is more actionable than a generic summary of the conversation.

Compaction should preserve identifiers, decisions, constraints, outstanding work, and relevant evidence when those details are necessary for correctness.

11.3 What should be retained?

Depending on the application, useful information may include:

• Current objective
• User requirements
• Decisions already made
• Relevant source references
• Tool execution results
• Current workflow status
• Completed and remaining steps
• Authorization state
• Outstanding questions
• Important errors and recovery information

A poorly designed compaction process can discard the very detail needed to complete the task correctly. Important state should be preserved separately where appropriate instead of relying entirely on a generated summary.

11.4 Session state versus memory

A session represents an interaction or execution context. State describes what is currently true about that interaction.

Memory refers to information retained for later use.

These concepts overlap, but they aren't interchangeable.

For example:

• Conversation history: what the user and assistant said.
• Session state: the current status of the conversation or workflow.
• Persistent memory: relevant information intended to survive across sessions.
• Execution state: which steps ran, which succeeded, and what remains.
• Retrieval store: documents or records searched when relevant.

Persistent memory should be designed deliberately. Not every message needs to become a permanent fact.

11.5 Memory architectures

Memory can be implemented through several mechanisms:

Database-backed state: stores structured facts, session records, preferences, and workflow status.

Conversation storage: preserves messages and execution events for later reconstruction.

Summaries: retain compressed information about earlier interactions.

Vector retrieval: retrieves semantically relevant content from an embedding index.

External knowledge stores: provide access to documents, tickets, source code, or business records.

A robust system may combine these approaches. The correct design depends on the type of information, consistency requirements, privacy constraints, and retrieval needs.

11.6 RAG versus memory versus long context

These three approaches solve related but different problems.

RAG is particularly useful when an application must answer questions using a large, changing collection of external information.

Memory is useful when relevant inf...