Building an AI Agent Harness in Java with Spring AI

In Part 1 we learned two ideas:

• Orchestration decides the next step. It's either a loop (the model decides) or a graph (your code decides).
• Agent = Model + Harness. The model is the brain; the harness is everything around it: orchestration, tools, memory, rules and logs.

Now let's see what that looks like in real Java code.

We'll build a tiny order-support agent with Spring AI, using the same example as before:

A customer asks: "Where's my order #123? If it's late, I want a refund."

The map: which code is the harness?

In Spring AI:

• The model is the ChatModel: Azure OpenAI, OpenAI, Anthropic or another provider. It only decides what to say, or which tool to ask for. It runs nothing itself.
• The harness is your Spring code: ChatClient, your @Tool methods, chat memory, advisors and the checks you write.

Let's build each piece.

Step 1: Tools (harness: the toolbox)

@Component public class OrderTools { @Tool(description = "Get the status of an order by its ID") public String getOrderStatus(String orderId) { return orderService.findStatus(orderId); // real DB/API call } @Tool(description = "Issue a refund for an order") public String issueRefund(String orderId, double amount) { if (amount > 100) { // guardrail return "Refund needs manager approval."; } return refundService.refund(orderId, amount); } } 

The model never runs this code. It can only ask for it, by name. The harness runs it and passes the result back.

The description matters: it's how the model knows when to use each tool. Treat it like documentation written for the model.

Step 2: Wire the harness together

@Configuration public class AgentConfig { @Bean ChatClient supportAgent(ChatClient.Builder builder, OrderTools tools) { ChatMemory memory = MessageWindowChatMemory.builder() .maxMessages(20) // context management .build(); return builder .defaultSystem("You are a support agent. Be brief and polite.") .defaultTools(tools) // register the toolbox .defaultAdvisors( MessageChatMemoryAdvisor.builder(memory).build(), // memory new SimpleLoggerAdvisor() // logs ) .build(); } } 

Every line here is harness:

• defaultSystem(...): the rules and instructions
• defaultTools(tools): the toolbox
• MessageWindowChatMemory: memory, capped at the last 20 messages so the context doesn't grow forever
• SimpleLoggerAdvisor: logs of each request and response

Step 3: The endpoint

@RestController class SupportController { private final ChatClient agent; SupportController(ChatClient supportAgent) { this.agent = supportAgent; } @PostMapping("/chat") String chat(@RequestParam String sessionId, @RequestBody String message) { return agent.prompt() .user(message) .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, sessionId)) .call() // the loop runs inside here .content(); } } 

The sessionId keeps each customer's conversation separate in memory.

Step 4: What actually happens inside .call()

1. Harness → sends prompt + tool list to the model 2. Model → "call getOrderStatus(123)" 3. Harness → runs getOrderStatus("123") → "Shipped, arriving today" 4. Harness → sends the result back to the model 5. Model → "Your order arrives today!" (no more tool calls) 6. Harness → no tool call, so the loop stops → returns the answer 

Only steps 2 and 5 are the model. Everything else is the harness.

Step 5: Take control of the loop

Spring AI hides the loop inside .call(). You can switch that off and write it yourself, which makes the harness visible:

ChatOptions options = ToolCallingChatOptions.builder() .toolCallbacks(ToolCallbacks.from(orderTools)) .internalToolExecutionEnabled(false) // "I'll run the loop myself" .build(); Prompt prompt = new Prompt(userMessage, options); ChatResponse response = chatModel.call(prompt); while (response.hasToolCalls()) { // the loop ToolExecutionResult result = toolCallingManager.executeToolCalls(prompt, response); // run tools prompt = new Prompt(result.conversationHistory(), options); // feed back response = chatModel.call(prompt); // ask again } return response.getResult().getOutput().getText(); 

This while loop is exactly what .call() normally does for you. Owning it lets you add things like:

• a max-turns limit, so a confused model can't loop forever
• a cost cap
• an approval pause before a risky tool such as issueRefund

Bonus: the same agent as a graph

Everything above is a loop: the model decides which tool to call. What if the refund decision must never be the model's?

Spring AI doesn't include a graph engine, but a simple graph is just plain Java where your code chooses the path and the model only writes text:

public String handle(String orderId) { Order order = orderService.find(orderId); // node: plain code if (!order.isLate()) { // edge: YOUR code decides return chatClient.prompt() .user("Write a short, friendly tracking update: " + order.tracking()) .call() .content(); // node: LLM writes text } String offer = chatClient.prompt() .user("Draft a polite refund offer for order " + orderId) .call() .content(); // node: LLM writes text approvalService.requestApproval(orderId, offer); // human-in-the-loop return "Your refund request is waiting for manager approval."; } 

No tools are registered here, so the model can't refund anything. It can only write the messages.

For bigger graphs with saved state, checkpoints and pause/resume, use a graph library such as LangGraph4j, or a workflow engine you already trust.

Cheat sheet

• ChatModel is the model. Everything else you write is the harness.
• .call() runs the tool loop for you. Turn it off with internalToolExecutionEnabled(false) to own the loop.
• @Tool methods are the toolbox. The model asks; your code runs them.
• Memory, system prompt, advisors and checks in your tools are the rest of the harness.
• Need a graph? Plain Java if/else for small flows; a graph library for big ones.

Next in this series: who builds the harness when you build agents on Azure: Copilot Studio vs Foundry Agent Service vs your own code.

What does your Spring AI harness look like? Let me know in the comments 👇

Code & Learn