Hindsight memory informs. Current evidence verifies. The engineer decides.

Picture this. Your tests are green and the build passes. You deploy, and twenty minutes later production is crawling, because of the same database problem your team fixed in the spring.

Nothing was wrong with the pipeline. It just has no memory. Every run starts fresh, checks the current commit and exits. Back in April, one engineer lost an afternoon to a migration that locked a table and starved the connection pool, then wrote up the fix. That knowledge sits in a wiki or a Slack thread, and the next deploy never sees it.

That gap is why I built ReleaseTrace.

🔗 Live demo · Source code

The idea: let past incidents speak before a release goes out

ReleaseTrace is a release-risk tool that looks at a deployment before it ships and asks a question most pipelines never ask: has something like this hurt us before?

The dashboard's own explanation uses an analogy I like. Think of it as a flight safety check before takeoff. Planes don't skip the checklist because the last flight landed fine, and your deploy shouldn't either.

Whenever a pull request opens or code is pushed, ReleaseTrace:

  1. Reads the Git diff
  2. Inspects Prisma migrations for risky table locks
  3. Compares staging and production configuration
  4. Searches the project's memory for similar past incidents

Catching a lock or pool-exhaustion problem in CI costs nothing. Catching it after the site goes down costs downtime, customers and a very bad night.

But memory alone is dangerous

My first instinct was to give an AI model access to old incident reports and let it advise. I quickly saw the flaw: a memory that looks relevant isn't necessarily relevant.

Say last year's outage was a PostgreSQL migration lock combined with a tiny connection pool. Today's release has a similar-looking migration, but it runs on a newer database version behind a connection proxy. A model will happily say "this looks like the earlier incident, apply the same fix." The words match, but the situation may not.

So I made one rule the backbone of the whole system:

Hindsight memory informs. Current evidence verifies. The engineer decides.

The stack, briefly

Layer Tool Responsibility
App & API Next.js Application and API layer
Data Neon PostgreSQL + Prisma Projects, deployments and assessments
Workflows Inngest Background jobs, so webhooks don't wait on slow work
Memory Hindsight (Vectorize) Long-term memory of past engineering incidents
Explanation Groq Turns findings into clear explanations and recommendations

I kept the two kinds of data apart on purpose. Neon holds what is happening now. Hindsight holds what happened before. Mixing them would blur the difference between current telemetry and historical context, and the dashboard says so directly: historical context is not current telemetry.

Retrieved memories have to earn their place

When Hindsight returns a match, ReleaseTrace treats it as a candidate, not an answer. Each one is scored on three things:

  • How strongly Hindsight ranked it
  • How closely the old failure resembles the current change
  • How well the old environment matches the new one (runtime, database type and version, pool size, infrastructure, feature flags) 
const finalRelevanceScore = hindsightScore * 0.40 + signatureSim * 0.35 + environmentComparison.matchScore * 0.25; const isRelevant = finalRelevanceScore >= relevanceThreshold; // default 0.65

To be clear, these weights are a tuned heuristic, not a statistical probability of failure. They exist to filter out results that are only semantically close. When a memory fails the check, the system can explain why the conditions differ instead of quietly reusing an old fix.

Saying "I don't know" on purpose

An empty memory bank shouldn't produce an invented incident, and a weak match shouldn't be dressed up as precedent. ReleaseTrace has an explicit abstention path for both cases:

const { state, shouldAbstain, abstentionReason, qualifiedMemories } = evaluateRecallRelevance(memoryComparisonPairs, this.relevanceThreshold);
  • Nothing qualifies: the assessment falls back on current evidence and deterministic rules.
  • Something qualifies: the engineer sees the root cause, the mitigation that worked, and the conditions it worked under.

Either way the dangerous migration still gets caught. A relevant memory simply adds "this has happened here before, and here is what fixed it." I'd much rather the tool admit it has no precedent than confidently quote an unrelated incident.

The LLM explains, but it doesn't get the final say on safety

ReleaseTrace runs deterministic rules first. They check for:

  • Unsafe production migrations
  • Missing lock-timeout safeguards
  • Low replica counts
  • Excessive total connection capacity
  • Unhealthy pre-deploy telemetry

The model then interprets and explains those findings, but it can't downgrade a serious one:

const finalRisk: RiskLevel = baselineRisk === 'CRITICAL' || baselineRisk === 'HIGH' ? baselineRisk : llmOutput.overallRisk;

The rules enforce safety, Hindsight supplies history, and the model translates both into plain language. Its output is validated before anything uses it.

What deserves to be remembered?

This was the hardest part. A wrong memory does damage long after the original mistake. If the system saves an AI's first guess ("probably a timeout") as a confirmed cause, and the real culprit was a leaked connection, every future assessment inherits that error.

So ReleaseTrace separates an assessment (findings and recommendations) from a verified engineering experience (confirmed root cause, applied fix, environment details, and confirmation). Before anything is written to Hindsight, it is validated and must carry engineer confirmation:

const memory = validateVerifiedMemory(input); assertEngineerConfirmation({ confirmation: memory.confirmation, isHypothesis: false, });

Writes get a deterministic document ID and automatic retries, production writes require bearer authorization, and the demo path is isolated from the real project memory.

Note: A shared credential doesn't prove which individual engineer confirmed something, so identity still needs proper operational care. Still, a model's hypothesis can't quietly become part of the team's verified history.

What I learned

  1. Quality beats quantity in retrieval. One genuinely relevant incident is worth more than ten loosely related ones.
  2. Abstaining is a feature. "My memory has nothing relevant" is an honest, useful answer.
  3. Memory needs provenance. Engineers should see where advice came from, what conditions applied, and who confirmed it.
  4. Honest interfaces matter. A polished dashboard can easily imply that something really happened when it was only simulated. That's why ReleaseTrace separates real deployment records from sandbox scenarios, and why I'd rather state its limits than call it production-ready too early. Some workflows still need deeper integration and verification.

Why I built it

I don't think an LLM should approve production deployments. I built ReleaseTrace because teams solve the same problems again and again, and the solutions never reach the next release decision. Hindsight brings that experience back, the relevance checks decide whether it applies, deterministic rules guard the critical cases, and an engineer decides what becomes lasting knowledge.

Memory-enabled tools get interesting when they carry real experience and are honest about where it came from and where it stops being useful.

Hindsight memory informs. Current evidence verifies. The engineer decides.

Links