An Incident Response Agent: From Past

Incidents to Future Fixes
**
The most dangerous thing an incident tool can do at 3 a.m. is confidently repeat last month's
fix. I built RecallOps to avoid that, and the reason it works at all comes down to one decision: I
put a real memory system underneath the agent instead of a prompt full of pasted runbooks.
**

What RecallOps does

**
RecallOps is a small Python service with a Streamlit front end. An on-call engineer describes a
production incident in plain text, picks the affected service and a severity, and hits one button.
Behind that button, three things happen, and every one of them runs through

Hindsight:

The incident description is sent to Hindsight as a recall query against a bank of past incidents.
The memories Hindsight returns, plus the new incident, go into a reasoning prompt.
The model returns a structured analysis, and once the engineer resolves the incident for real, the actual fix and outcome are written back to Hindsight with retain. That's the whole loop: recall, reason, resolve, retain. The files that matter are small: agent.py # recall -> reason -> retain, the core loop app.py # Streamlit UI hindsight_memory.py # minimal store/recall round trip seed_memory.py # loads the initial incident bank into Hindsight The reasoning model is openai/gpt-oss-120b, served through Groq at temperature 0.2. But the model is the least interesting part of this system. Swap it for another vendor and the agent barely changes. Hindsight is the part that actually determines whether the agent is useful, because it's what decides which past incident the model even gets to see.

Why I didn't build my own memory layer

**
My first instinct, and probably yours, is to reach for a vector database, embed some incident
text, and write a cosine-similarity query. I've built that before. It's a weekend of work and it looks
like memory. It isn't, not for something you'd trust during an outage.
What a vector table doesn't give you is a place to put the parts that actually matter for incident
response: what changed between the last incident and this one, how to weigh a five-year-old
resolution against one from last week, and how to keep the store coherent as hundreds of
incidents pile up instead of degrading into noise. Those are retrieval-quality problems, not
storage problems, and they're exactly what a dedicated memory system is supposed to solve.
That's the gap
agent memory as a category exists to fill, and it's why I integrated against
Hindsight's API instead of writing my own retrieval code.
The integration is almost embarrassingly small:
hindsight = Hindsight(
base_url=os.getenv("HINDSIGHT_BASE_URL"),
api_key=os.getenv("HINDSIGHT_API_KEY")
)
BANK_ID = "RecallOps"
One client, one bank ID, two methods used anywhere in the codebase: recall and retain. I
wanted the memory system to be a dependency I trust and don't think about, not a subsystem I
maintain. That decision is the through-line of this whole project: the agent's intelligence is
really Hindsight's retrieval quality wearing a prompt.

How Hindsight is queried

**
The recall call is deliberately plain:
def investigate_incident(incident):

Search Hindsight for relevant past experience

result = hindsight.recall(
bank_id=BANK_ID,
query=incident
)

Convert memories into readable text

memories = "\n\n".join(
memory.text for memory in result.results
)
The query passed to Hindsight is the incident text exactly as the engineer typed it. No keyword
extraction, no classification step, no rewriting. Engineers describe incidents in symptom
language, things like "timeouts," "error rate increased to 30%," "connection errors." The
incidents stored in Hindsight are written in that same register, so semantic recall works without
me pre-processing anything. Every transformation I'd insert between the engineer's words and
the query is a place where the match could get worse, so I left the boundary alone and let
Hindsight do the matching.
What comes back is a list of memory objects, each with .text. I join them into one block, and
that block does two jobs: it's the context the reasoning model sees, and it's also, unmodified,
what the engineer sees in the UI. That second use matters more than it looks like it should.
What goes into the Hindsight bank
Retrieval quality depends entirely on what's in the bank, and I was deliberate about the shape of
what gets stored. Every incident record, whether it's part of the initial seed or written by an
engineer after a real resolution, follows the same structure:
"""
**

Production Incident:

**
Notification service experienced a 35% failure rate with email
request timeouts and intermittent 504 errors from the external
email provider.
**

Actual Resolution:

**
The external email provider was experiencing an outage.
Enabled retry with exponential backoff and queued failed
notifications for later delivery.
**

Outcome:

**
Notification failures dropped below 2% after the provider
recovered and queued notifications were delivered successfully.
**

Lesson Learned:

**
When an external notification provider returns intermittent
504 errors, verify provider health before increasing local resources.
"""
I made sure the seed bank isn't one-note. The first incident in Hindsight is a payment API with an
exhausted database connection pool, fixed by raising the pool size. If that were the only kind of
memory in the bank, recall would nudge every future incident toward "add more connections,"
regardless of whether that's actually the right move. So the other seeded incidents point the
opposite direction on purpose: one where the fix is restarting stalled queue workers, and the
notification case above, where the correct move is to wait on an external provider rather than
touch local infrastructure at all. A memory system is only as good as the diversity of what's in it,
and that's on me to seed well, not something Hindsight can fix after the fact.
The Lesson Learned line is the part I spent the most time on, because it's phrased as a
condition, not an instruction: "when X and Y occur together, check Z first," rather than "do Z."
Conditional lessons retrieve into a new, slightly different incident more gracefully than
imperative ones do, and they give the reasoning model something to test against the current
evidence instead of something to obey.
**

Turning recall into judgment, not

autopilot
**
Hindsight will happily return a memory that looks like a match. What the agent does with that
memory is the part I had to get right, because a close match is not the same thing as a
confirmed cause. Payment timeouts with connection errors could mean pool exhaustion, like
last time, or it could mean a slow query, a downstream dependency stalling, or a deploy that
leaked connections. If the model treats last month's fix as this month's answer, it can make an
outage worse.
So the prompt puts the Hindsight memories in front of the model as evidence to interrogate, not
as a decision to execute:
**

IMPORTANT RULES

**:

Do NOT blindly copy a previous solution.
First identify what is similar between the new incident and the previous experience.
Clearly identify what is different.
A previous solution should only be recommended directly if the current evidence supports it.
If the previous incident only provides a similar pattern, describe it as a reference rather than a confirmed solution.
Do not claim certainty when there is insufficient evidence.
Prioritize checks that an engineer should perform before making a potentially risky production change. The output is locked into a fixed structure, and two of the eight sections exist solely to force a comparison against what Hindsight retrieved: "Why It Is Relevant" and "What Is Different." Both come before "Recommended Action." "Previous Solution," where the Hindsight memory's fix is most directly discussed, sits near the bottom, with an explicit instruction not to present it as guaranteed. The retrieved memory is available the whole way through, but it's never allowed to be the first or the last word. I'm not going to claim a benchmark I haven't run. What I can say from reading the code is that this structure makes it obvious, section by section, whether the model is actually reasoning from the Hindsight memory or just restating it, and that's a much easier thing to review than a paragraph of free-form advice. **

Closing the loop back into

Hindsight
**
The part of the system I care about most is the write side, because it's what makes the memory
bank better over time instead of static.
def save_experience(incident, resolution, outcome):
experience = f"""
Production Incident:
{incident}
Actual Resolution:
{resolution}
Outcome:
{outcome}
Lesson Learned:
This production experience should be considered
when investigating similar future incidents.
"""
hindsight.retain(
bank_id=BANK_ID,
content=experience
)
I never write the model's analysis back into Hindsight. Only the incident as described, the
resolution the engineer actually applied, and the outcome they observed go into retain. The
model's suggestion might have been right, wrong, or ignored, and none of that should become
part of the historical record as if it were fact. Hindsight's bank stays a record of what happened,
confirmed by a human, not a record of what an LLM guessed.
Because the write path uses the same four-part shape as the seed data, new memories retrieve
exactly as well as the old ones. Six months in, an incident resolved last week and an incident
from the original seed set are indistinguishable to Hindsight's recall, which is exactly what I
want. The bank doesn't need re-indexing or manual curation to stay useful as it grows; that's
Hindsight's job, and it's the whole reason I didn't build this part myself.
**

What it looks like in use

An engineer types:
Payment API is experiencing severe timeouts. Error rate has increased to 30%. Database
connection errors are also appearing.
Hindsight's recall returns the earlier payment incident, where the rate was 35% and the cause
was pool exhaustion. The analysis that comes back, in shape rather than verbatim, reads like
this:
Likely Root Cause: pool exhaustion, stated as the leading hypothesis rather than a
certainty.
Hindsight Experience: the earlier payment incident, in its own words, shown in the UI's
evidence panel exactly as Hindsight returned it.
Why It Is Relevant: same service, same symptom pairing.
What Is Different: the error rate is lower this time, and nothing in the new report confirms
the pool is actually saturated, only that connection errors are occurring.
Recommended Immediate Checks: current pool utilization and whether connections are
being held by long-running queries, before anyone touches a config value.
Previous Solution: raising the pool limit worked last time and could be adapted, offered as a
reference, not a directive.
Change the input to a payment-provider outage with no database errors, and Hindsight's recall
shifts with it: the notification-provider memory, with its lesson about verifying provider health
before adding local resources, becomes the relevant precedent instead. Nothing in the agent's
code special-cases "external dependency." Hindsight surfaces the right memory because the
stored lesson was written to match that situation, and the agent reasons from whatever it's
handed.
The UI shows the raw Hindsight memories next to the analysis, in an expandable panel. I did that
on purpose. If the agent's reasoning depends on a memory, the engineer should be able to read
that memory and judge the analogy themselves. A recommendation with no traceable source is
one a good engineer will, correctly, ignore.
**

Lessons learned

Let the memory system own retrieval quality. Building a vector search integration would have gotten me matching text. It wouldn't have gotten me a bank that stays coherent and useful as it grows, which is the actual hard problem, and it's why routing everything through Hindsight's recall and retain was the right call.
Recalled memory is evidence, not an instruction. The single biggest behavior change came from prompt rules that force the model to state similarities and differences before it's allowed to recommend anything drawn from Hindsight.
Seed the bank with contradicting lessons on purpose. If every memory in Hindsight pointed toward the same fix, recall would just be an expensive way to bias every incident toward one answer.
Only write confirmed outcomes back to Hindsight. Keeping the model's own guesses out of retain is what keeps the memory bank trustworthy instead of self-reinforcing.
Show the retrieved memory, not just the conclusion. Surfacing Hindsight's raw output next to the analysis cost nothing to build and does more for an engineer's trust than any amount of prompt tuning. There's more I'd want before I'd call this finished: recall filtered by service, and some notion of recency weighting so a fix from last week counts for more than one from a year ago. But the core bet has held up. The agent isn't smart on its own. It's honest about uncertainty, and it has a memory that gets a little more useful every time someone tells it how an incident actually ended