You ask Claude Code to fix a failing checkout test. A few minutes later it replies: "Fixed. The tests pass now." You merge. Then CI goes red, and when you scroll back through the session you can't find the moment a test actually ran.

It's a common enough experience that a recent DEV post title sums it up: I Got Tired of Claude Code Saying 'It's Fixed' When Nothing Had Run. The summary at the end of a session is the model's account of what happened. It isn't a record.

Claude Code can give you that record. It has built-in OpenTelemetry support, and with it turned on, every tool call becomes an event: which tool, whether it succeeded, how long it took and, if you opt in, the exact shell command. In an earlier article I covered what to log for an AI agent you run in production. This one points the same idea at the agent that writes your code: send its events to a file on your machine, then check what actually ran after the last edit.

Everything about Claude Code's telemetry below comes from its monitoring documentation.

What Claude Code can record

Telemetry is off until you set CLAUDE_CODE_ENABLE_TELEMETRY=1. Claude Code then exports metrics (token and cost counters, sessions, commits) if you configure a metrics exporter, and events if you configure a logs exporter. The events are what matter here:

  • claude_code.user_prompt is logged when you submit a prompt. The prompt text is redacted unless you set OTEL_LOG_USER_PROMPTS=1.
  • claude_code.tool_result is logged when a tool finishes. It carries tool_name, success ("true" or "false"), duration_ms, and an error_type such as ShellError when the tool failed.
  • claude_code.tool_decision is logged when a permission decision is made. A call you reject produces this event with decision: "reject" and no tool_result, because the tool never ran.
  • claude_code.api_error and claude_code.api_retries_exhausted tell you when requests to the model failed after retries, which explains some sessions that simply stall.

Two attributes tie it together. Every event produced while Claude Code works on one prompt shares a prompt.id, so you can pull up everything that happened for "fix the checkout test" in one filter. event.sequence is a counter that orders events within a Claude Code process.

By default, the tool_result event tells you that Bash ran, not what it ran. Commands are part of the detail you have to opt into with OTEL_LOG_TOOL_DETAILS=1. With it set, a Bash event gets a tool_parameters JSON string containing full_command, and every tool's arguments are added as tool_input.

Step 1: Run a collector that writes to a file

You don't need a dashboard to start. The OpenTelemetry Collector can receive Claude Code's events over OTLP and write them to a local file with its file exporter. This config is all it takes (I used otelcol-contrib 0.165.0):

receivers: otlp: protocols: grpc: endpoint: localhost:4317 http: endpoint: localhost:4318 exporters: file: path: ./claude-events.jsonl service: pipelines: logs: receivers: [otlp] exporters: [file]

Start it with otelcol-contrib --config collector.yaml. With the default format: json, the exporter writes newline-delimited OTLP JSON, one line per batch. Binding to localhost keeps the receiver off your network. Note that the file exporter is marked alpha for logs, and its README says exact field names aren't guaranteed to stay stable.

Step 2: Point Claude Code at it

In the shell where you start Claude Code:

export CLAUDE_CODE_ENABLE_TELEMETRY=1 export OTEL_LOGS_EXPORTER=otlp export OTEL_EXPORTER_OTLP_PROTOCOL=grpc export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 export OTEL_LOG_TOOL_DETAILS=1 claude

This is the docs' logs-only configuration plus the tool-details flag. To check it works, submit a prompt and look for a user_prompt event in the file. If nothing arrives, the docs suggest starting Claude Code with claude --debug-file <path> and looking for [3P telemetry] errors in that log.

A word on what you've just turned on: with OTEL_LOG_TOOL_DETAILS=1, every command and every tool argument goes into that file, including anything sensitive the agent passes on a command line. That's one reason to keep the destination local while you try this. Leave OTEL_LOG_RAW_API_BODIES off: the docs say it exports full API requests and responses, including the whole conversation, and implies consent to everything the other content flags would reveal.

Step 3: Ask the file what ran

Now the useful part. This script reads the collector's file, keeps the tool_result events, groups them by prompt.id, prints every call in order, and flags two cases: files were edited but no test command ran afterwards, or a test command after the last edit failed. It uses only the Python standard library.

#!/usr/bin/env python3 """Summarise what Claude Code actually ran, per prompt, from an OTLP JSON file.""" import json import re import sys from collections import defaultdict TEST_CMD = re.compile(r"\b(pytest|npm (run )?test|pnpm test|yarn test|go test|cargo test|make test)\b") EDIT_TOOLS = {"Edit", "Write", "NotebookEdit"} def value(v): return next(iter(v.values()), None) if v else None def events(path): with open(path) as f: for line in f: for rl in json.loads(line).get("resourceLogs", []): for sl in rl.get("scopeLogs", []): for rec in sl.get("logRecords", []): yield {a["key"]: value(a["value"]) for a in rec.get("attributes", [])} prompts = defaultdict(list) for e in events(sys.argv[1]): if e.get("event.name") == "tool_result": prompts[e.get("prompt.id", "?")].append(e) for pid, calls in prompts.items(): calls.sort(key=lambda e: int(e.get("event.sequence", 0))) print(f"prompt {pid}") edited, tests = False, [] for c in calls: params = json.loads(c.get("tool_parameters") or "{}") cmd = params.get("full_command", "") ok = "ok " if c.get("success") == "true" else "FAIL" print(f" {ok} {c['tool_name']:<6} {c.get('duration_ms', '?'):>5} ms {cmd}") if c["tool_name"] in EDIT_TOOLS: edited, tests = True, [] # tests only count if they ran after the last edit elif cmd and TEST_CMD.search(cmd): tests.append(c.get("success") == "true") if edited and not tests: print(" !! files changed, no test command ran after the last edit") elif edited and not all(tests): print(" !! a test command after the last edit failed")

Change TEST_CMD to match how your project runs tests.

Running it on example events

To test the pipeline without pretending to have a real session log, I wrote a small script that sends example events to the collector over OTLP/HTTP. They use the attribute names from the Claude Code docs, but the values are made up: three prompts, one where a test ran after the edit, one where only git status ran, and one where npm test failed. The collector wrote them to claude-events.jsonl, and this is the real output of python3 what_ran.py claude-events.jsonl (Python 3.13.5) on that file:

prompt p1 ok Read 40 ms ok Edit 40 ms ok Bash 2140 ms pytest tests/test_cart.py prompt p2 ok Edit 40 ms ok Edit 40 ms ok Bash 35 ms git status !! files changed, no test command ran after the last edit prompt p3 ok Write 40 ms FAIL Bash 880 ms npm test !! a test command after the last edit failed

Prompt p2 is the "it's fixed" case: two edits, and the only command afterwards was git status. Whatever the final message said, nothing tested that change. Prompt p3 ran the tests, and the event says the call failed. If the summary for p3 claimed passing tests, you now have something concrete to point at.

What this does and doesn't tell you

The telemetry records what Claude Code executed, which is exactly what the session summary can't prove. It still has limits worth knowing before you rely on it:

  • success describes the tool call. The docs don't spell out how every exit code maps to it, so run one failing test on your own setup and check what the event shows before you trust the FAIL flag.
  • The regex is a heuristic. A test run hidden inside a script called check.sh won't match. Add your own commands to the pattern.
  • Rejected calls aren't tool results. If you denied a command, look for a tool_decision event with decision: "reject".
  • Ordering is per process. The docs note that event.sequence restarts with each Claude Code process, so a resumed session can repeat values. For anything beyond one sitting, sort by event.timestamp first.

If you want to stop Claude from finishing without running tests, that's enforcement, and it belongs in a hook. I covered that with a Stop hook in Not Every Rule Belongs in CLAUDE.md. Telemetry is the other half: it doesn't prevent anything, but it gives you a record you didn't have to trust the model for. The docs also note that tool_use_id on these events matches the ID passed to hooks, so the two can be joined.

Checklist

  • Set CLAUDE_CODE_ENABLE_TELEMETRY=1 and a logs exporter; metrics alone won't show tool calls.
  • Add OTEL_LOG_TOOL_DETAILS=1 only where the destination is somewhere you control.
  • Keep the collector bound to localhost while you experiment.
  • Group events by prompt.id to see everything one request triggered.
  • After any "fixed" message, check that a test command ran after the last Edit or Write.
  • Confirm on your setup how a failing test shows up in success before automating on it.
  • Watch for api_retries_exhausted when a session goes quiet.
  • Leave raw API bodies and prompt logging off unless you have a reason and a safe place to store them.

Where to go next

This is one small observability loop: capture what the agent did, then check it against what it said. LLM Observability & Production Monitoring: The Complete Guide covers the full version for LLM systems in production: tracing agents and multi-step workflows, the metrics that matter, measuring output quality without a reference answer, alerting without noise, detecting drift, and turning real production failures into permanent test cases. It's 40 pages, with a fully worked example and a 30-day path from a black box to an observable system.

This article was written with the assistance of AI and reviewed by the author. Claude Code telemetry behavior is described from Anthropic's official monitoring documentation, and the collector setup from the OpenTelemetry Collector's file exporter and OTLP receiver READMEs. The events shown are hand-written examples sent through a real local collector, and the output is the script's real output on them, not output from a Claude Code session. PromptMaster guides are independent and not affiliated with or endorsed by Anthropic or the OpenTelemetry project.