Teaching an SRE Agent Not to Repeat Mistakes Using Hindsight

Imagine troubleshooting the same Kubernetes issue repeatedly, even after your team has already solved it. That's a major limitation of AI-powered troubleshooting agents: they can analyze incidents, but often fail to learn from previous ones.

While building CLUSTERSAGE, we wanted to solve this problem by giving our Kubernetes troubleshooting agent persistent operational memory using Hindsight.

How CLUSTERSAGE Works

CLUSTERSAGE combines Kubernetes diagnostics, AI reasoning, and persistent memory to investigate recurring infrastructure issues.

Its architecture includes:

Next.js: Web console for investigations and post-mortems.

FastAPI: Coordinates data ingestion and troubleshooting.

Kubernetes & GitOps: Provide cluster telemetry, logs, and manifests.

Hindsight: Stores and retrieves previous incident knowledge.

Google Gemini: Analyzes symptoms and generates potential fixes.

InsForge: Handles authentication and user history.

Before information reaches the AI or memory system, sensitive credentials are sanitized.

Why Hindsight Instead of Basic Vector Search?

Traditional vector search retrieves information based on semantic similarity. However, Kubernetes logs contain exact error codes, resource names, and configuration details that may be critical to identifying a problem.

Hindsight combines:

BM25: Finds exact keywords and error patterns.

Vector Search: Retrieves semantically related incidents.

Reranking: Improves the relevance of retrieved memories.

Reflect: Helps identify patterns across previous incidents.

This allows CLUSTERSAGE to retrieve useful operational knowledge instead of relying entirely on generic AI responses.

How Hindsight Learns

We integrated three key memory capabilities:

Recall: Retrieves relevant historical incidents during troubleshooting.

Reflect: Identifies patterns and lessons across incidents.

Retain: Stores verified resolutions and post-mortems for future use.

This creates a continuous learning loop where every resolved incident can improve future investigations.

A Real-World Example

Consider a Kubernetes application repeatedly entering CrashLoopBackOff.

During the first investigation, CLUSTERSAGE identifies a missing REDIS_CACHE_URL environment variable. After the engineer verifies the fix, the incident is stored in Hindsight.

Three weeks later, another deployment experiences the same issue.

Instead of starting from scratch, CLUSTERSAGE recalls the previous incident, identifies the missing environment variable, and recommends the previously successful resolution.

The result: Less repeated investigation and faster access to proven solutions.

Key Lessons

Building CLUSTERSAGE taught us three important things:

AI troubleshooting needs more than a large context window; it needs persistent memory.

Historical fixes are useful only when retrieval is relevant and tenant data remains isolated.

Post-mortems should become reusable operational knowledge, not documents that are forgotten after an incident.

Conclusion

An effective SRE agent should do more than identify problems. It should remember what happened, learn from verified solutions, and use that knowledge when similar incidents occur.

By combining ai
reasoning with Hindsight's episodic memory, CLUSTERSAGE aims to turn Kubernetes troubleshooting from repeated guesswork into knowledge-driven operations.

Explore Hindsight: https://github.com/vectorize-io/hindsight
Here's Github Url: https://github.com/manishgowda17/cloudsage-agent