What we learned building ALIBI at HackPrinceton 2026.
Imagine a future where AI agents handle everything from business operations to sensitive data.
Now imagine one of those agents going rogue, copying a restricted file, and deleting its own activity logs.
How would you know what happened?
That's the question behind ALIBI, the project my team built at HackPrinceton.
Instead of creating another chatbot that analyzes logs, we built a controlled environment where AI workers carry out tasks, a simulated rogue worker attempts to cover its tracks, and an independent AI investigator tries to reconstruct the incident.
The twist? We know exactly what happened, but the investigator doesn't.
And we can measure whether its conclusions are actually supported by evidence.
First, we needed a world worth investigating
ALIBI runs three AI workers inside a simulated environment.
They ingest datasets, prepare summaries, and archive files. In our live experiments, real language models select their ordinary work actions.
At a predetermined point, the simulator changes one worker's objective. A scripted policy makes it perform bounded unauthorized actions, such as copying restricted synthetic data and removing local log entries.
We deliberately kept the malicious actions scripted rather than claiming an AI model independently planned an attack.
Every action leaves traces across different evidence sources: application logs, an independent log collector, network flow records, and scheduler assignments.
The simulator freezes this evidence, generates a reproducible snapshot, and seals the true action history.
Then we hand the surviving evidence to an investigator that has no access to the answer.
The investigator is a real DeepAgent
This was the most interesting engineering challenge.
We used LangChain DeepAgents to build an investigator that doesn't just receive a pile of evidence and generate an explanation.
It actually operates through a multi-step investigation.
In Specialists mode, there's a lead investigator, an integrity auditor, and a skeptic.
The lead creates a plan using write_todos, delegates evidence analysis through the task tool, and uses the results to decide which leads deserve further investigation.
The integrity auditor examines contradictions between evidence sources.
The skeptic challenges findings by looking for explanations such as authorized transfers, routine log rotation, or collection gaps.
The lead also maintains a case-scoped virtual filesystem. It writes hypotheses, evidence references, and open questions, then reads those notes as the investigation develops.
We even tested checkpointing: an investigation stopped by its resource allowance and later resumed in another process while retaining its case notes.
This isn't three agents exchanging fabricated messages. They're making actual tool calls against indexed evidence.
In recorded real-model investigations, we observed the lead revise its plan after delegated evidence checks revealed new leads. Those revisions weren't guaranteed in every run, which became an important part of our evaluation.
The moment it found evidence that wasn't there
One of our favorite moments is deceptively simple.
The workstation's local log is missing an entry.
But the independent collector still has its copy.
Our deterministic checks reconcile the two sources, account for ordinary log rotation and coverage gaps, and correlate the discrepancy with network activity.
The investigator can then examine the suspicious transfer, consider alternatives, and build a report using exact evidence IDs.
Every finding opens the original record.
There's also an important distinction between an account being used, a worker assigned to that account, and the person or process actually controlling it.
ALIBI doesn't automatically treat those as equivalent.
The checks establish observations. The agent's job is to decide what to investigate, connect the observations, challenge interpretations, and produce an auditable account.
But how do we know the investigator was right?
Most AI demos end when the model produces something convincing.
We wanted to go further.
The investigator submits a structured, cited report. A verifier checks the claims against available evidence. A human then locks the report.
Only after that can the simulator reveal the sealed answer key.
We compare the report against the actual executed effects, including what the investigator missed and whether it made unsupported allegations.
We also built a way to remove an evidence source and see which conclusions can no longer be established.
And because the action history is reproducible, we can compare investigator configurations on the same frozen evidence.
That turns ALIBI from an investigation demo into an experimental environment for evaluating AI investigators.
Then our multi-agent system surprised us
We expected specialist delegation to improve results.
So we measured it against single-agent investigators, deterministic checks, and a basic keyword baseline.
Across six simulated evaluation cases, including four incidents and two benign controls:
System
Incident effects established
Keyword baseline
4/16
10/16
Single agent + deterministic checks
15/16
Three agents + deterministic checks
12/16
Deterministic checks alone
16/16
The most surprising part wasn't that the multi-agent system made mistakes.
It was why.
In three evaluated cases, the skeptic downgraded correct findings that the deterministic checks supported.
Specialization added reasoning and review, but sometimes became counterproductive.
We repeated eight model evaluations once on identical evidence. Only two produced the same number of established effects, reminding us not to overinterpret small differences.
These were small experiments on simulated data, not real-world forensic accuracy measurements. Our checks were also designed for the simulator.
But the overall lesson was valuable:
An agentic architecture isn't better just because it has more agents.
The useful part is giving agents meaningful tools, context, memory, and evidence they can actually verify.
What we're taking away
Building ALIBI changed how I think about agentic AI.
Giving an LLM a role and calling it an agent is easy. Building an agent that can plan, delegate, use tools, update its approach, maintain state, and defend its conclusions is much more interesting.
And testing whether that system actually works is harder still.
As AI agents take on increasingly important work, I think we'll need better ways to investigate their failures—and better ways to evaluate the investigators themselves.
That's what we're exploring with ALIBI.
Try the live demo: Link
GitHub: ALIBI
Built at HackPrinceton Fall 2026.
