Agent Replay Lab: a local-first VCR for TypeScript agent tool I/O

I open-sourced Agent Replay Lab, a local-first TypeScript toolkit that records, mock-replays, diffs, and CI-tests the tool I/O of tool-using agents.

It answers a narrower question than observability products, which usually re-run live: does this agent still take the same tool path (name / order / args / responses) against the same recorded tool world, offline, in CI? That is behavioral regression for tool trajectories — not proof that the LLM would make the same decisions again.

Local-first. SQLite plus portable JSON. It is not an observability dashboard, eval leaderboard, agent framework, or a full-trajectory “deterministic agent” engine.

Honest modes in v0.1

Mode What it does In v0.1?
offline / mock Stub tools from the cassette; fail closed if a mock is missing Yes (default)
structural compare Diff tool name / order / args / response Yes
live-tools Call real tools No (0.2)
live-model Call a real model No (0.2)

Offline mock stubs tools only. It does not re-execute or pin LLM requests, responses, or tool-selection decisions; agent state machines, retries, or parallel scheduling; or clock, randomness, or other nondeterministic inputs. You can optionally record model.requested / model.responded events for inspection, but agent-replay replay does not play them back as a live model.

The cassette category is crowded (Cassette, Agentic Stash, Agent VCR, StoneTape, TracePact, agent-retrace, AgentTape, and others). Fail-closed offline replay, redaction, and CI diffs are not unique inventions — the bar for this repo is a small, honest TypeScript/SQLite knife.

Install (not on npm yet)

v0.1.0, Apache-2.0. Requires Node.js 22+ and pnpm (the repo pins pnpm@10.33.4). Clone the monorepo:

git clone https://github.com/JarvisOsorio/agent-replay-lab.git cd agent-replay-lab pnpm install pnpm build pnpm test # 5-minute proof pnpm --filter @agent-replay-lab/example-basic-agent demo

Packages: @agent-replay-lab/core and @agent-replay-lab/cli (agent-replay commands: init, list, inspect, replay, diff, fixture, test, export, import, doctor). fixture create pins tool sequence, args, and responses. test golden --run fails with exit 1 on structural drift. Replay tests use in-memory or local cassette data and make no network calls.

Capture seam is wrapTool from @agent-replay-lab/core. Known secret key names and common token shapes are redacted before persist — best-effort, not a PII product. Pair with agentscrub for after-the-fact store remediation.

Not in v0.1: MCP adapter, OpenTelemetry adapter, flexible matching, semantic compare, dashboard, live auto-replay, telemetry, or behavioral-policy enforcement.

I'm Jarvis Osorio; I built this. Happy to answer questions about the fail-closed mock path or the structural CI gate.