7 Ways MCP and Retry Plumbing Quietly Break Production AI Agents (And the Deterministic Fixes)

Published: October 31, 2026

Publisher: Lali Publications

Web Edition & Free Weekly Briefing Signup: https://tools.lalipublications.com/agent-backbone/

Issue #2 Permalink: https://tools.lalipublications.com/agent-backbone/issue-2/

Previous Issue: https://tools.lalipublications.com/agent-backbone/issue-1/

Editorial Contact: larrys@lalipublications.com

The hook: Your agent retried a dropped tool call and refunded a customer twice. Your overnight run is still "retrying" a model that no longer exists. Your code-review bot approved a commit that isn't the one that got merged. None of these are model problems. They are plumbing problems, and all seven show up in public release notes and changelogs from AI coding agents and an app builder dated October 21–28, 2026.

Executive Summary

Last week's issue covered how agents quietly break inside a single conversation. This week the failures live one layer down: in the transport, retry, and orchestration plumbing that connects an agent to its tools, its workers, and its model provider. None of these bugs make a demo look worse. All of them make week-three production worse: a refund issued twice, an unattended run that never finishes, a subagent that "can't find" a tool that exists, a code review that approved the wrong commit.

Six of the failure modes below were fixed in public releases of widely used AI coding agents between October 21 and October 28, 2026; the seventh is a documented provider model retirement on an AI app builder. We re-checked each primary source on October 29, 2026 and cite the exact version and date. The point is not that any one vendor got it wrong. The point is that teams with large engineering budgets shipped these bugs, which means your homegrown agent stack almost certainly has some of them too.

1. Duplicate Side Effects on Mid-Call Disconnects (a non-idempotent tool call re-sent after the connection dropped).
2. Retrying What the Server Said Not to Retry (retry storms and unattended runs that hang indefinitely).
3. Subagents Blind to Late-Connecting MCP Servers (tool registries snapshotted at spawn time).
4. Cancellation That Never Reaches the Remote Server (the client stops waiting, the server keeps working).
5. Failed Worker Requests That Leave the Caller Hanging (a swallowed error instead of an explicit failure).
6. Reviewing a Stale Commit After a Late Push (event-driven automation keyed to "latest" instead of a SHA).
7. Silent Model Rerouting by the Provider (a retired model ID quietly served by a different model with different cost and behavior).

Inside this issue:

• Section 1: The 7 Failure Modes Checklist (Symptom, Root Cause, Deterministic Fix)
• Section 2: Agent Backbone Briefing Issue #2, each failure mode in depth with Preferred vs. Rejected snippets
• Section 3: The Fixed 8-Question Reliability Regression Test Block
• Section 4: How to Score These Failures With the Locked 4-Criteria Rubric
• Section 5: Primary Sources (Verified October 29, 2026)

1. The 7 Failure Modes Checklist

2. Agent Backbone Briefing — Issue #2: 7 Ways MCP and Retry Plumbing Quietly Break Production Agents

1. Duplicate Side Effects When a Connection Drops Mid-Call

What goes wrong in production: An agent calls a tool over HTTP: create_refund, send_invoice, open_ticket. The server does the work, but the connection closes before the response arrives. The client sees a failure, retries, and the server does the work a second time. From the agent's point of view, one tool call happened. From your customer's bank statement, two did.

Verified example: Claude Code 2.1.307 (published to npm October 21, 2026) lists: "Fixed HTTP MCP tool calls being sent twice when the connection closed mid-call; only read-only or idempotent tools are now retried".

Root cause: A lost response is ambiguous. The request may have failed before execution, or it may have succeeded with only the reply lost. Blind transport-level retries treat both cases as "safe to resend." In MCP revision 2026-07-28, a broken Streamable HTTP response stream can't be resumed. The in-flight request is lost, the server treats the disconnect as cancellation, and the changelog says clients MUST re-issue it as a new request with a new request ID. For a write, that re-issue is only safe with an idempotency key; without one, re-issue a status lookup instead of the raw write, because the server may have committed the write before it noticed the disconnect.

The deterministic fix:

• Use tool annotations as the first gate. The MCP specification (current revision 2026-07-28) defines readOnlyHint, destructiveHint, and idempotentHint on ToolAnnotations. Retry automatically only when a tool is read-only or idempotent. The spec also says annotations are hints and clients MUST treat them as untrusted unless they come from trusted servers, so pair them with your own server allowlist. Tool names are only unique within one server, so key that allowlist by (server, tool), never by tool name alone.
• Make writes idempotent with a key. MCP has no standard idempotency field, so this works only when the tool's inputSchema declares one (for example idempotency_key) and the server dedupes on it. Generate one key per logical action (not per attempt) and reuse it on every retry. Each retry is still a new JSON-RPC request with a new request ID. If the tool has no key, never resend: return an explicit unknown outcome and reconcile with a status lookup.

Rejected (retries every tool the same way):

def call(name, args): for attempt in range(3): try: return mcp.call_tool(name, args) except ConnectionError: continue # resends create_refund too 

Preferred (retry only safe tools; keyed writes):

# Pseudocode: adapt names to your MCP client SDK. import random from time import sleep # Key by (server, tool): tool names are only unique per server. SAFE_TO_RETRY = {(srv.id, t.name) for srv in servers if srv.id in TRUSTED_SERVERS for t in srv.tools if t.annotations and (t.annotations.readOnlyHint or t.annotations.idempotentHint)} AMBIGUOUS = (ConnectionError, TimeoutError) # response lost: outcome unknown  def call(server, name, args, action_id, reconcile_step): tool = registry.get(server, name) keyed = "idempotency_key" in tool.inputSchema.get("properties", {}) if keyed: # only where the schema allows it  args = {**args, "idempotency_key": action_id} # one key per logical action  safe = (server, name) in SAFE_TO_RETRY or (keyed and server in TRUSTED_SERVERS) read_only = (server in TRUSTED_SERVERS and tool.annotations is not None and bool(tool.annotations.readOnlyHint)) attempts = 3 if safe else 1 for attempt in range(1, attempts + 1): try: return mcp.call_tool(server, name, args) # new request ID each attempt  except AMBIGUOUS: if attempt < attempts: sleep(random.uniform(0, min(8, 0.5 * 2 ** attempt))) # full jitter  if read_only: return {"status": "error", "code": "TOOL_UNREACHABLE", "tool": name} # Anything that writes may have committed before the response was lost.  return {"status": "unknown_outcome", "action_id": action_id, "next_step": reconcile_step} 

The key detail in the Preferred version: when a write's outcome is unknown, the agent gets an explicit unknown_outcome result and a reconcile step, not a silent second attempt.

2. Retrying Requests the Server Marked Non-Retryable

What goes wrong in production: The model API or a downstream service returns an error and explicitly says it should not be retried. The client retries anyway, with backoff, again and again. In an interactive session, the user waits minutes. In an unattended overnight run, the job never ends and never reports failure.

Verified examples:

• Claude Code 2.1.311 (published to npm October 26, 2026): "Fixed Claude Code retrying requests the API marked not to retry, which could hold a turn for minutes or, in unattended retry mode (CLAUDE_CODE_RETRY_WATCHDOG), indefinitely".
• OpenAI Codex CLI 0.165.0 (changelog dated October 27, 2026): "Stopped retrying when a provider reports that a model was not found, preserving its error message."

Root cause: Retry logic written as "catch error, back off, try again" with no error classification. A model-not-found or explicit do-not-retry response will never succeed on attempt 50.

The deterministic fix:

• Classify before you retry. An explicit server signal beats the status code (Anthropic's SDKs, for example, obey an x-should-retry header). Terminal: 400/422 validation, 403 permission, 404 not found (including unknown model IDs), 501, quota-exhausted 429s, and any explicit do-not-retry signal. On 401, or an MCP 403 insufficient_scope, refresh or re-authorize once, then treat a second failure as terminal. Retryable: connection resets, 408, 429, 500, 502, 503, 504, and provider overload codes such as 529. For writes, retry these only when the request carries an idempotency key. An MCP tool result with isError: true is not a transport failure; return it to the model instead of retrying.
• Honor Retry-After (seconds or an HTTP date) on 429/503 as the minimum wait, but cap it: if the server asks for longer than your remaining time budget, stop now and surface the error instead of sleeping.
• Cap both attempts and total wall time, and surface the original error message to the agent and the logs. Unattended mode should fail fast and loudly, never wait forever.
• Add jitter and retry at one layer only. Use exponential backoff with full jitter so clients don't retry in lockstep. Turn off retries in either the inner layers (SDK, HTTP client) or the outer ones (agent loop). Three layers that each make up to 4 attempts can send 4 × 4 × 4 = 64 requests for one call. Stop calling a dependency whose recent failure rate crosses a threshold (a circuit breaker).

Rejected:

retry: on: any_error max_attempts: unlimited # "unattended mode should be resilient" backoff: exponential 

Preferred:

# Illustrative config, not a specific library's schema retry: layer: agent_client # SDK/HTTP-client retries off (max_retries: 0) precedence: explicit_signal_over_status # e.g. x-should-retry: false wins terminal: [400, 403, 404, 422, 501, model_not_found, quota_exhausted, do_not_retry_signal] refresh_auth_once_then_terminal: [401, insufficient_scope_403] retryable: [408, 429, 500, 502, 503, 504, 529, connection_reset] retryable_writes: idempotency_key_required # otherwise return unknown_outcome tool_result_is_error: return_to_model # not a transport retry honor_retry_after: { enabled: true, fail_fast_if_over_budget: true } backoff: { type: exponential, base_seconds: 1, max_seconds: 30, jitter: full } max_attempts: 4 max_total_seconds: 120 on_exhausted: return_error_with_original_message # never hang 

3. Subagents Blind to Tools From Late-Connecting MCP Servers

What goes wrong in production: Your orchestrator spawns a subagent while one MCP server (say, a slow remote CRM connector) is still connecting. The subagent's tool list is built at spawn time, so the CRM tools never appear. The subagent tells the user it has no way to look up the account, or worse, guesses. The bug is timing-dependent, so it passes in staging and fails on cold starts.

Verified example: Claude Code 2.1.311 (published to npm October 26, 2026): "Fixed subagents being unable to use MCP tools from servers that finish connecting after the subagent starts".

Root cause: The tool registry is treated as immutable for the life of a subagent, while MCP servers connect asynchronously and their tool lists can change.

The deterministic fix:

• Refresh the registry on connect and on change. In MCP 2026-07-28, servers that declare "tools": {"listChanged": true} send notifications/tools/list_changed only to clients that opened a subscriptions/listen stream with toolsListChanged: true, so open one for every server that supports it. Re-run tools/list and update every live subagent's toolset when the notification arrives, when a server becomes ready, or when the cached list's ttlMs expires.
• Declare required servers per task. If a task needs the CRM server, either wait for it (with a deadline) before spawning or return an explicit "required tool server unavailable" error. Never let the model improvise around a missing tool.

Rejected:

def start(task): return spawn_subagent(task, tools=registry.snapshot()) # frozen at t=0 

Preferred:

# Pseudocode: adapt to your orchestrator. Register handlers before spawning. # once, at startup registry.on("server_ready", registry.refresh) # re-run tools/list registry.on("notifications/tools/list_changed", registry.refresh) # needs subscriptions/listen  async def start(task): await registry.wait_for(task.required_servers, timeout_s=15) # raises -> explicit error  return spawn_subagent(task, tools=registry.live_view()) # sees every refresh 

4. Remote MCP Cancellation That Never Reaches the Server

What goes wrong in production: A user hits stop, or your orchestrator times out a slow tool call. Locally, the request is abandoned. Remotely, the server has no idea and keeps running a long query, a batch write, or a paid API loop. You pay for work nobody will read, and side effects land after the user thinks the action was stopped.

Verified example: OpenAI Codex CLI 0.165.0 (changelog dated October 27, 2026): "Cancelled remote MCP tool requests when their local response futures are dropped."

Root cause: Cancellation is implemented as "stop awaiting" rather than "tell the other side to stop."

The deterministic fix:

• Propagate cancellation on every exit path (user stop, timeout, parent task cancelled, future dropped). In MCP 2026-07-28 the mechanism depends on transport. Over Streamable HTTP, closing the request's response stream is the cancellation; no notifications/cancelled is required or expected. Over stdio, the client sends notifications/cancelled with the request ID and an optional reason. Either way, ignore late responses, and treat a write's outcome as unknown until you check, because the server may have finished before the cancel arrived.
• Set timeouts per tool. If a tool sends progress notifications, you may extend the wait on each one, but keep a hard maximum; the cancellation spec says implementations SHOULD always enforce one.
• Make servers honor it: check for cancellation between steps of long-running tools, and make partial writes either transactional or reported.
• Log it: record cancel_sent with the request ID and time, then whether any response or side effect appeared after it. MCP has no cancel acknowledgment, so a late side effect is the only evidence that the cancel lost the race.

Rejected:

import asyncio async def call(client, name, args): try: return await asyncio.wait_for(client.call_tool(name, args), timeout=30) except asyncio.TimeoutError: return "Tool timed out" # server may still be running; no cancel on the wire 

Preferred:

# Pseudocode: assumes your SDK cancels the request on the wire (closes the HTTP # stream, or sends notifications/cancelled on stdio) when the awaiting task is # cancelled. Verify that with test Q5 below. import asyncio async def call(client, name, args): try: async with asyncio.timeout(tool_timeout_s(name)): # Python 3.11+; per tool  return await client.call_tool(name, args) except TimeoutError: log.info("cancel_sent", tool=name, reason="timeout") return {"status": "cancel_requested", "outcome": "unknown_until_checked", "tool": name} except asyncio.CancelledError: log.info("cancel_sent", tool=name, reason="user_stop_or_parent_cancel") raise # never swallow cancellation 

5. Failed Worker Requests That Leave the Caller Hanging

What goes wrong in production: A UI process or orchestrator sends a request to a worker. The worker throws, crashes, or hits an unhandled branch, and never answers. The caller awaits a response that will never come. There is no error in the logs, just a spinner or a stuck pipeline step.

Verified example: OpenCode v1.19.5 (GitHub release published October 28, 2026 UTC; late October 27 Pacific): "Failed TUI worker requests now return an error instead of leaving the caller hanging." (PR #54936)

Root cause: The request/response contract only defines the success path. Failure is "no message," which is indistinguishable from "still working."

The deterministic fix:

• Exactly one terminal response per request: build the success or typed-error result inside try/catch, then send it with a single reply after the block, so a failing reply can't trigger a second one.
• A deadline on every await on the caller side that also aborts the request (so the worker stops too), converted into an explicit error the agent and user can act on.

Rejected:

worker.on("request", async (req) => { const out = await handle(req); // throws -> no reply ever sent worker.reply(req.id, { ok: true, out }); }); 

Preferred:

worker.on("request", async (req) => { let res; try { res = { ok: true, out: await handle(req) }; } catch (err) { res = { ok: false, error: { code: "WORKER_FAILED", message: String(err) } }; } worker.reply(req.id, res); // exactly one reply on every path }); // caller side: the deadline also aborts the request, so the worker stops too const res = await call(req, { signal: AbortSignal.timeout(20_000) }).catch((err) => ({ ok: false, error: { code: err?.name === "TimeoutError" ? "WORKER_TIMEOUT" : "WORKER_FAILED", message: String(err) }, })); 

6. Code Review Running on a Stale Commit After a Late Push

What goes wrong in production: An automated reviewer (human-in-the-loop bot, AI review, or CI gate) is triggered by a push event. Events arrive late or out of order, and the job reviews whatever commit it resolves at run time, or the wrong one. The check goes green on code that is not what gets merged.

Verified example: Claude Code 2.1.312 (published to npm October 27, 2026): "[Code Review] Fixed Code Review reviewing an outdated commit, or none at all, when GitHub delivered a push event late; it now reviews the pull request's current commit".

Root cause: The review is keyed to an event or to "latest," not to an immutable commit identity.

The deterministic fix:

• Pin every review run to a commit SHA and print it in the review output.
• Re-check the PR head before posting. If head.sha changed while the review ran, don't post. The new push's own run reviews the new head, and a concurrency group with cancel-in-progress stops the stale run early.
• Make merge gates require a review result whose SHA equals the merge candidate's SHA.

Rejected (fragment):

# fragment: steps excerpt, not a complete workflow on: push steps: - run: git checkout origin/${BRANCH} # whatever is latest when the job starts - run: ai-review --post-result 

Preferred (complete workflow; ai-review stands in for your review CLI):

name: ai-review on: pull_request: types: [opened, synchronize, reopened] concurrency: group: ai-review-${{ github.event.pull_request.number }} cancel-in-progress: true # a new push cancels the stale run permissions: contents: read pull-requests: write jobs: review: runs-on: ubuntu-latest env: SHA: ${{ github.event.pull_request.head.sha }} PR: ${{ github.event.pull_request.number }} GH_TOKEN: ${{ github.token }} steps: - uses: actions/checkout@v7 with: ref: ${{ github.event.pull_request.head.sha }} - run: ai-review --sha "$SHA" --out review.json - name: Post only if the PR head has not moved run: | CURRENT=$(gh pr view "$PR" --repo "$GITHUB_REPOSITORY" --json headRefOid -q .headRefOid) if [ "$CURRENT" != "$SHA" ]; then echo "Head moved to $CURRENT; not posting review for $SHA" exit 0 fi ai-review --post-result review.json --sha "$SHA" 

7. Silent Model Rerouting by the Provider

What goes wrong in production: You never changed a line of code, but answers drift and the bill moves. The provider retired the model ID you call and now serves those requests with a different model.

Verified example: Lovable's changelog (entry dated October 22, 2026): Gemini 2.5 Pro, 2.5 Flash, and 2.5 Flash Lite chat models for app AI features "retired on October 20, 2026." Requests for 2.5 Pro and Flash now go to Gemini 3.8 Flash, and Flash Lite requests go to Gemini 3.1 Flash Lite; "the replacement can cost a different amount per request and respond differently." To its credit, Lovable documented the change publicly, and rerouting keeps apps running. The architectural lesson is that "keeps working" is not the same as "behaves the same," and your agent needs to notice.

Root cause: The application treats a model name as a stable contract. In practice it is a lease with an end date.

The deterministic fix:

• Pin full model IDs in config, ...