Disclosure: This article and its code are AI-generated. The results below were verified by executing the local synthetic test. No LLM, live customer system or paid API was called by the experiment.
A booking agent returns a clean response: the request succeeded, the JSON contains a booking identifier, and the assistant tells the user that the appointment is confirmed. There is still a question neither the status code nor the JSON shape can answer: did the system book the appointment the user actually authorized?
This is a practical evaluation problem for developers building agents around APIs. Infrastructure checks remain necessary, but a request can succeed at one boundary while the larger task fails at another. A booking for the wrong day is not made correct by a well-formed receipt.
We can examine that difference without asking a language model to judge its own work. This article builds a small, deterministic Python lab with explicit input, simulated state and acceptance rules. You can run it locally, inspect every case and replace its fixtures with evidence from your own staging workflow.
What the experiment actually measures
Quick answer: In this hand-selected set of 12 synthetic fixtures, an HTTP-2xx-only rule accepted eight of nine known-invalid outcomes. A booking-shape-only rule accepted seven. There are also two valid outcomes and one outcome whose state cannot be observed. These counts demonstrate specific weaknesses in the acceptance rules; they do not estimate a production failure rate or the accuracy of any model or product.
The distinction matters because the fixtures were deliberately chosen as counterexamples. There is no random traffic sample, customer cohort or benchmark leaderboard here. The chart describes these files and these rules. If you change the fixtures, the counts may change.
The task-level rule is also deliberately simple. It is an oracle for a controlled environment where we supply authoritative simulated state. It is not an independently calibrated universal evaluator. Its matching labels are checked for fixture consistency, not offered as evidence that it can classify arbitrary production conversations correctly.
Start with a contract a reader can challenge
The synthetic user wants an appointment on a particular day for a particular customer. The contract permits two successful outcomes:
- If a slot is available, exactly one matching booking exists, the required confirmation has been supplied, and the assistant accurately reports completion.
- If no slot is available, no booking is created and the assistant accurately reports that limitation.
The second branch prevents an evaluator from rewarding an agent simply for making a booking every time. The user asked for appropriate behavior, not an unconditional database write.
The lab uses invented customer and booking identifiers and a future test date. They identify fixtures, not real people or events. The available-slot condition, confirmation flag and final state are supplied directly by the fixture. A real integration would need to establish those observations from appropriately authorized records.
| Layer | Question | Evidence in this lab |
|---|---|---|
| Request result | Did the operation return a success status? | Synthetic HTTP status |
| Output shape | Does the body look like a booking response? | Required string fields and a status label |
| Task outcome | Does established state satisfy the request? | Simulated booking records, confirmation and availability |
| User explanation | Does the response describe the established result? | A controlled reply classification |
| Observability | Is enough state available to decide? | Explicit None for unavailable evidence |
These layers complement each other. A consumer still needs to parse the response and handle transport errors. The mistake is treating an earlier layer as sufficient evidence for every later one.
Why a correct HTTP status can coexist with a failed task
RFC 9110 defines HTTP success in relation to the request and method semantics. That does not make a successful API operation proof that an agent selected the correct day, customer or action. If the client submits the wrong date and the server creates exactly that booking, the server may have fulfilled its interface contract while the agent failed the user's task.
The distinction is even clearer for asynchronous work. A 202 response concerns acceptance for processing, not proof that processing has finished. An assistant that announces completion immediately can overstate the available evidence. The lab includes this case with a queued response and no established booking.
Output validation solves a different problem. The shape rule checks that booking_id, status, date and customer_id are strings and that the status belongs to a small allowed set. It does not compare the returned date with the requested date. It cannot establish that a receipt corresponds to a real state change. A schema is still useful; it simply answers a narrower question.
Keep the counterexamples inspectable
The complete set includes a correct booking, an appropriate unavailable-slot explanation, a wrong day, missing confirmation, a duplicate, queued work described as complete, a receipt with no operation, a wrong customer, a cancellation described as complete, a friendly message without a useful result, a known dependency failure and an unobservable outcome.
There is an intentional asymmetry here. The unavailable-slot explanation is a valid task outcome, but it does not look like a booking object. A shape checker designed only for bookings rejects that valid branch. In a real API, a tagged union or separate response type could describe both outcomes. It would still need task-specific checks to establish whether the right branch occurred.
The unknown-state fixture is not silently labelled a failure. A timeout does not, by itself, tell us whether a state-changing operation happened. If the state record is unavailable, the task evaluator returns UNKNOWN. An operational report can separately count the execution problem. Mixing those classifications loses the distinction between a demonstrated wrong result and missing evidence.
This is also why retry behavior needs a contract. After an uncertain write, another attempt may create a duplicate. Use the provider's actual status lookup and replay guarantees where they exist; this local example does not establish them.
Run the complete experiment
Save the following program as acceptance_lab.py. The acceptance checks and CSV/JSON export use only Python's standard library. Plotting is optional and imports Matplotlib only when requested.
"""Controlled acceptance-rule demonstration. No network, LLM or customer data. Run: python3 acceptance_lab.py --out results Optional figure: python3 acceptance_lab.py --out results --plot """ import argparse import copy import csv import json import platform from datetime import datetime, timezone from pathlib import Path REQUEST = {"date": "2030-04-18", "customer_id": "synthetic-customer"} BOOKING = {"booking_id": "synthetic-booking", "status": "booked", **REQUEST} def fixture(name, **changes): case = {"name": name, "http": 200, "response": copy.deepcopy(BOOKING), "bookings": [copy.deepcopy(BOOKING)], "reply": "confirmed", "confirmed": True, "available": True, "expected": "PASS"} case.update(changes) return case CASES = [ fixture("correct_booking"), fixture("no_slot_explained", response={"availability": "none"}, bookings=[], reply="unavailable", confirmed=False, available=False), fixture("wrong_day", response={**BOOKING, "date": "2030-04-19"}, bookings=[{**BOOKING, "date": "2030-04-19"}], expected="FAIL"), fixture("missing_confirmation", confirmed=False, expected="FAIL"), fixture("duplicate_booking", bookings=[BOOKING, {**BOOKING, "booking_id": "second"}], expected="FAIL"), fixture("queued_but_claimed_done", http=202, response={**BOOKING, "status": "queued"}, bookings=[], expected="FAIL"), fixture("receipt_without_operation", bookings=[], expected="FAIL"), fixture("wrong_customer", response={**BOOKING, "customer_id": "another-synthetic-customer"}, bookings=[{**BOOKING, "customer_id": "another-synthetic-customer"}], expected="FAIL"), fixture("cancelled_but_claimed_done", response={**BOOKING, "status": "cancelled"}, bookings=[], expected="FAIL"), fixture("friendly_message_only", response={"message": "Done!"}, bookings=[], expected="FAIL"), fixture("known_dependency_failure", http=503, response={"error": "unavailable"}, bookings=[], reply="error", expected="FAIL"), fixture("outcome_not_observable", http=504, response=None, bookings=None, available=None, reply="unknown", expected="UNKNOWN"), ] def http_success(case): return 200 <= case["http"] < 300 def booking_shape(case): body = case["response"] fields = ("booking_id", "status", "date", "customer_id") return (isinstance(body, dict) and all(isinstance(body.get(key), str) for key in fields) and body["status"] in {"booked", "queued", "cancelled"}) def task_contract(case): """Oracle for this controlled fixture, NOT a production truth service.""" if case["bookings"] is None or case["available"] is None: return "UNKNOWN" if not case["available"]: return "PASS" if not case["bookings"] and case["reply"] == "unavailable" else "FAIL" if not case["confirmed"] or len(case["bookings"]) != 1: return "FAIL" booking = case["bookings"][0] matches = all(booking.get(key) == value for key, value in REQUEST.items()) return "PASS" if matches and booking.get("status") == "booked" and case["reply"] == "confirmed" else "FAIL" def run(out, plot=False): out.mkdir(parents=True, exist_ok=True) rows = [] for case in CASES: decision = task_contract(case) assert decision == case["expected"], (case["name"], decision) rows.append({"case": case["name"], "expected": case["expected"], "http_2xx_accepts": http_success(case), "shape_accepts": booking_shape(case), "task_decision": decision}) invalid = [row for row in rows if row["expected"] == "FAIL"] summary = {"fixtures": len(rows), "valid": sum(r["expected"] == "PASS" for r in rows), "invalid": len(invalid), "unknown": sum(r["expected"] == "UNKNOWN" for r in rows), "false_acceptances": {"HTTP 2xx only": sum(r["http_2xx_accepts"] for r in invalid), "Booking shape only": sum(r["shape_accepts"] for r in invalid)}, "scope": "Hand-selected synthetic counterexamples; not production frequency or model accuracy.", "python": platform.python_version(), "run_utc": datetime.now(timezone.utc).isoformat()} with (out / "cases.csv").open("w", newline="") as handle: writer = csv.DictWriter(handle, fieldnames=list(rows[0])); writer.writeheader(); writer.writerows(rows) (out / "summary.json").write_text(json.dumps(summary, indent=2)) (out / "fixtures.json").write_text(json.dumps(CASES, indent=2)) if plot: import matplotlib matplotlib.use("Agg") import matplotlib.pyplot as plt fig, ax = plt.subplots(figsize=(10, 5.6), dpi=160) labels, values = zip(*summary["false_acceptances"].items()) bars = ax.barh(labels, values, color=["#2563a6", "#c16b22"], height=.5) for bar, value in zip(bars, values): ax.text(value + .12, bar.get_y() + bar.get_height()/2, f"{value} of {len(invalid)}", va="center", fontsize=13) ax.set_xlim(0, len(invalid) + 1); ax.set_xticks(range(len(invalid) + 1)) ax.set_xlabel("Known-invalid fixtures accepted by the rule", fontsize=12) ax.set_title("Success status and output shape missed task failures", loc="left", fontsize=15, pad=22) ax.spines[["top", "right", "left"]].set_visible(False) ax.tick_params(axis="y", length=0, labelsize=12) ax.grid(axis="x", alpha=.18); ax.set_axisbelow(True) fig.text(.02, .02, "Controlled synthetic lab: 12 fixtures = 2 valid, 9 invalid, 1 unknown.\nHand-selected counterexamples; no LLM calls, customer data or production-rate estimate.", fontsize=10, color="#374151") fig.tight_layout(rect=(0, .13, 1, 1)) fig.savefig(out / "false-acceptances.png", facecolor="white") fig.savefig(out / "false-acceptances.svg", facecolor="white") plt.close(fig) print(json.dumps(summary, indent=2)) if __name__ == "__main__": parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--out", type=Path, default=Path("results")) parser.add_argument("--plot", action="store_true") args = parser.parse_args() run(args.out, args.plot) Run it in a clean directory:
python3 acceptance_lab.py --out results The run writes three inspectable artifacts: fixtures.json contains every input and expected label, cases.csv contains each rule's decisions, and summary.json records the counts, Python version and execution timestamp. The local run used for this article completed on October 29, 2026, with Python 3.14.2. There are no random seeds, model versions or remote services involved.
To reproduce the figure in an isolated environment:
python3 -m venv .venv .venv/bin/python -m pip install matplotlib .venv/bin/python acceptance_lab.py --out results --plot This also writes PNG and SVG versions of the figure. The recorded plotting environment used Matplotlib 3.11.2. The acceptance counts do not depend on that plotting library. A different plotting version can change rendering without changing the fixture decisions.
