In Part 1 we saw that multi-agent systems fail in the gaps between agents, and we built a handoff record to capture what each agent sent, passed and understood.
But a handoff record floating on its own isn't much use. You need it sitting inside one trace that holds the entire task: every agent, every handoff, every tool call, every MCP server, in order. Otherwise you're back to four green traces and one wrong answer.
This post walks through five steps that turn a team of agents into one readable story:
- One trace per task
- A span per agent turn
- A span per handoff
- A clean hierarchy
- Outcomes tagged per agent
Then we'll write a small root-cause walker that reads the trace and points at the agent that broke the task.
Step 1: One trace per task
Create a root span the moment the task starts, and pass its trace context to every agent, sub-agent, tool and MCP server the task touches. OpenTelemetry's context propagation handles this across process and service boundaries using the W3C traceparent header.
# orchestrator.py from opentelemetry import trace from opentelemetry.propagate import inject tracer = trace.get_tracer("multi-agent") def run_task(user_request: str): with tracer.start_as_current_span("task") as root: root.set_attributes({ "app.task.type": "churn_report", "app.task.goal": user_request[:500], }) plan = call_agent("planner", {"goal": user_request}) for step in plan["steps"]: call_agent(step["agent"], step) Agents over HTTP:
import httpx def call_agent(name: str, payload: dict) -> dict: headers = {} inject(headers) # adds traceparent / tracestate r = httpx.post(AGENT_URLS[name], json=payload, headers=headers, timeout=120) return r.json() # researcher_service.py (receiving side) from opentelemetry.propagate import extract @app.post("/run") def run(request: Request, payload: dict): ctx = extract(dict(request.headers)) with tracer.start_as_current_span("invoke_agent researcher", context=ctx): return researcher.run(payload) Agents over a queue (very common for long-running agent work): put the carrier in the message itself.
# producer carrier = {} inject(carrier) queue.publish({"task": step, "otel": carrier}) # consumer msg = queue.consume() ctx = extract(msg["otel"]) with tracer.start_as_current_span("invoke_agent writer", context=ctx): writer.run(msg["task"]) Do the same for MCP servers and any downstream services. A single missed hop splits the trace again, and the split always seems to happen at exactly the hop you need to see.
Step 2: A span per agent turn
Each agent's work becomes a child span carrying its name, role and version, so you can filter and compare by role later.
AGENT_META = { "researcher": {"role": "research", "version": "researcher-v7", "model": "your-model-id"}, "writer": {"role": "drafting", "version": "writer-v12", "model": "your-model-id"}, "reviewer": {"role": "verify", "version": "reviewer-v4", "model": "your-model-id"}, } def agent_span(name: str): meta = AGENT_META[name] span = tracer.start_as_current_span(f"invoke_agent {name}") return span, { "gen_ai.operation.name": "invoke_agent", "gen_ai.agent.name": name, "gen_ai.request.model": meta["model"], "app.agent.role": meta["role"], "app.agent.version": meta["version"], } The version tag matters more here than in single-agent systems. You might ship a new writer prompt on Monday and a new reviewer on Wednesday. When quality dips Thursday, per-agent versions tell you which change to suspect.
Step 3: A span per handoff
Handoffs get their own spans, not just log lines. That puts them on the timeline, between the agents they connect.
def handoff(sender: str, receiver: str, task: str, context: dict, permissions: list[str], original_goal: str): with tracer.start_as_current_span(f"handoff {sender}->{receiver}") as span: span.set_attributes({ "app.handoff.sender": sender, "app.handoff.receiver": receiver, "app.handoff.task_sent": task[:1000], "app.handoff.context_keys": list(context.keys()), "app.handoff.context_tokens": count_tokens(context), "app.handoff.permissions": permissions, "app.task.original_goal": original_goal[:500], }) result = call_agent(receiver, { "task": task, "context": context, "permissions": permissions, "original_goal": original_goal, }) # receiver returns its one-sentence restatement (see Part 1) span.set_attribute("app.handoff.task_understood", result.get("task_understood", "")[:1000]) return result Now the comparison from Part 1 (sent vs. understood vs. original goal) can run directly on spans in your trace backend.
Step 4: Model the hierarchy
Multi-agent traces get unreadable fast. A clean, consistent hierarchy keeps them navigable:
task ← root: one per user request └── invoke_agent planner ← agent turn ├── chat plan ← model call ├── handoff planner->researcher ← handoff │ └── invoke_agent researcher │ ├── chat search_strategy │ ├── execute_tool sql_query ← tool call │ └── execute_tool web_fetch ├── handoff planner->writer │ └── invoke_agent writer │ └── chat draft └── handoff writer->reviewer └── invoke_agent reviewer └── chat review The pattern: task → agent → (handoff → agent)* → model call → tool call. Pick a shape and stick to it across every agent and team. Consistent span names (invoke_agent X, handoff A->B, execute_tool Y) make queries like "all handoffs into the writer this week" trivial.
Step 5: Tag outcomes per agent
Mark every agent span with its own result, not just the task's. Otherwise every failure gets blamed on whoever acted last, which is usually the writer or the reviewer, and usually wrong.
from opentelemetry.trace import Status, StatusCode def finish_agent(span, outcome: str, reason: str | None = None): # outcome: ok | degraded | failed | rejected_input span.set_attribute("app.agent.outcome", outcome) if reason: span.set_attribute("app.agent.outcome_reason", reason) if outcome in ("failed", "rejected_input"): span.set_status(Status(StatusCode.ERROR, reason or outcome)) Where do per-agent outcomes come from? Cheap checks you already have: schema validation on each agent's output, the handoff drift score, a tool error, the reviewer's verdict, an eval run on a sample.
Finding the agent that broke the task
When a task fails, read the trace from the root and find the first handoff where the task changed shape. That's usually where the problem started, even if the damage showed up three agents later.
Here's that idea as code, working on spans exported from your backend:
from dataclasses import dataclass @dataclass class Suspect: span_name: str agent: str reason: str def first_bad_hop(spans: list[dict], drift, threshold=0.75) -> Suspect | None: """spans: one trace, sorted by start time.""" for s in spans: a = s["attributes"] # 1. A handoff where the received task drifted from the goal if s["name"].startswith("handoff "): score = drift(a.get("app.handoff.task_understood", ""), a.get("app.task.original_goal", "")) if score < threshold: return Suspect(s["name"], a["app.handoff.receiver"], f"task changed shape (similarity {score:.2f})") # 2. An agent that marked its own output as failed or degraded if a.get("app.agent.outcome") in ("failed", "degraded"): return Suspect(s["name"], a.get("gen_ai.agent.name", "?"), a.get("app.agent.outcome_reason", "agent reported failure")) # 3. A tool error that the agent carried on past if a.get("gen_ai.operation.name") == "execute_tool" and s["status"] == "ERROR": return Suspect(s["name"], a.get("gen_ai.tool.name", "?"), "tool error before task completed") return None Run it on the EU-churn example from Part 1:
>>> first_bad_hop(spans, drift) Suspect(span_name='handoff planner->researcher', agent='researcher', reason='task changed shape (similarity 0.71)') Not the writer. Not the reviewer. The first hop. Run the same function across every failed trace in a week and group by span_name, and you get a ranked list of your weakest handoffs, which is a much better backlog than "the agents are flaky."
Practical notes
- Sample whole tasks, not spans. If you sample, decide at the root and keep or drop the entire task. Half a multi-agent trace is worse than none. Tail-based sampling in the OpenTelemetry Collector can keep 100% of failed or high-risk tasks.
- Truncate, don't drop, large attributes. Store the first N characters of task text and context on the span; put the full payload in logs keyed by span ID.
- Redact at capture. Handoff context often carries customer data. Scrub it before it leaves the process.
- Expect convention churn. OpenTelemetry's GenAI conventions are still in Development. Keep attribute names in one module.
For a broader look at what to collect across single and multi-agent setups, see this complete guide to AI agent observability.
TL;DR
- Propagate context so one task = one trace, across HTTP, queues and MCP servers.
- Give every agent turn and every handoff its own span, tagged with name, role and version.
- Keep a consistent hierarchy: task → agent → handoff → agent → model call → tool call.
- Tag outcomes per agent, then walk the trace for the first hop where the task changed shape.
In Part 3, we'll move from single traces to dashboards: system, role, handoff and verification metrics, the alert that catches a rubber-stamp reviewer, and the security signals for when one compromised agent starts instructing the rest. It all builds on the checklist in this breakdown of multi-agent monitoring metrics and security signals.
