An Invoice Agent That Doesn't Forget Yesterday

An AI agent can analyse an invoice, identify a discrepancy, and suggest a resolution. But what happens when the same vendor sends another invoice with a similar problem tomorrow?

Does the agent remember what happened previously, or does it start from zero again?

That was the problem I wanted to explore while building AP Memory Agent, an Accounts Payable assistant powered by Python, Hindsight, Groq, and Llama 3.

The idea was simple: give an AI agent persistent memory so that information from previous vendor interactions can provide useful context for future invoice analysis.

The Problem With Stateless AI Agents

Accounts Payable teams regularly process invoices from multiple vendors.

Consider a simple example.

A vendor submits an invoice containing an unexpected freight charge. The issue is analysed, and a resolution is suggested.

A few days later, the same vendor submits another invoice with a similar discrepancy.

An experienced Accounts Payable clerk might recognise the pattern because they remember the previous interaction.

A stateless AI agent doesn't automatically have that advantage. Unless historical information is provided, it has to analyse the new invoice without that context.

I wanted to explore how persistent memory could help address this limitation.

Introducing AP Memory Agent

AP Memory Agent is an AI-based invoice analysis workflow that uses persistent memory to retrieve previous vendor interactions.

The implementation combines three main technologies:

• Python: Coordinates the application and agent workflow.
• Hindsight: Handles persistent memory and information retrieval.
• Groq with Llama 3: Processes invoice information and generates analysis.

The interesting part isn't simply connecting an LLM to an invoice-processing workflow. It's introducing a memory layer that allows information from previous interactions to become available during future requests.

The Core Architecture: Recall, Reason, Retain

I designed the workflow around three operations.

1. Recall

Before analysing a new invoice, the agent queries Hindsight for relevant information associated with the vendor.

For example, it can search for previous invoice discrepancies involving Acme Corp.

This gives the agent access to potentially useful historical context.

2. Reason

The retrieved information is combined with the current invoice details.

The resulting context is passed to Groq, which runs Llama 3 to analyse the invoice and generate a response.

The model can consider both the current invoice and the available historical information.

3. Retain

After generating its response, the agent stores information from the current interaction using Hindsight.

This information can potentially be retrieved during subsequent interactions.

The complete workflow looks like this:

Recall → Reason → Retain

The separation is intentional. Hindsight handles memory operations, while the language model handles reasoning.

How I Integrated Hindsight

Hindsight is the central memory component of this project.

I used its Python client to connect the application to a memory bank.

The recall operation retrieves relevant historical information:

memory_results = hindsight.recall( bank_id=BANK_ID, query=f"{vendor_name} invoice discrepancies" ) 

The retrieved information is then incorporated into the prompt sent to the language model.

After the analysis, the agent retains the current interaction:

hindsight.retain( bank_id=BANK_ID, content=new_memory ) 

These two operations form the foundation of the memory workflow.

Instead of depending entirely on temporary conversation history, the application can retrieve information retained from previous interactions.

Testing the Memory Workflow

I wanted to demonstrate the difference between an agent with no previous context and one that can retrieve historical information.

For this, I used two consecutive invoice interactions involving Acme Corp.

First interaction:

• Vendor: Acme Corp
• Invoice: #1001
• Previous memory: None available.

The agent analyses the invoice and retains the interaction.

Second interaction:

• Vendor: Acme Corp
• Invoice: #1002
• Previous memory: The agent searches Hindsight for previously retained information.

The second interaction now has an opportunity to use historical context from the first interaction.

This demonstrates an important distinction: the agent isn't necessarily starting every interaction with an empty memory.

Of course, retrieving information doesn't guarantee that the generated recommendation will always be correct. Retrieval relevance and model reasoning still matter.

What I Learned

Building this project helped me understand a few important aspects of memory-enabled AI agents.

1. Memory and reasoning are different responsibilities.

An LLM can reason over available information, but persistent memory requires a separate mechanism to retain and retrieve useful context.

2. Context matters.

A response generated without historical information can differ from one generated with relevant previous interactions.

3. First-time interactions need special consideration.

The agent must handle situations where no previous memory exists instead of assuming historical information is always available.

4. Retaining information creates continuity.

An interaction can become useful context for a future request.

5. Memory doesn't guarantee correctness.

An agent can retrieve relevant information and still generate an incorrect recommendation. Financial workflows need validation, human oversight, and auditability.

What's Next?

AP Memory Agent currently focuses on demonstrating persistent memory within an Accounts Payable workflow.

There are several directions in which I would like to take it further:

• Automated invoice data extraction.
• More advanced vendor profiling.
• Duplicate invoice detection.
• Improved discrepancy classification.
• Vendor-specific memory retrieval.
• Human-in-the-loop approval workflows.
• Integration with accounting and ERP systems.

These improvements could help develop the current implementation into a more comprehensive Accounts Payable solution.

Final Thoughts

One thing I found interesting while building this project is that making an AI agent remember something is different from simply making it answer a question.

The ability to retrieve useful information from previous interactions introduces continuity into the workflow.

With Hindsight, I explored a practical way to implement this through Recall, Reason, and Retain.

It's a small implementation, but it demonstrates an idea I believe is important when building agents that interact with users and data over time.

Project Repository: AP Memory Agent on GitHub

Explore Hindsight: GitHub Repository

Documentation: Hindsight Documentation

Learn more: What Is Agent Memory?