This is a submission for the Kaggle Benchmarking Challenge.
What I Benchmarked
AI coding demos show the happy path: the form renders, you click the button, a green "Success!" appears. I wanted to know what happens when a real user shows up.
RealityBench gives a model a spec for one of twelve small web apps, then opens the page in a headless browser and does what users do:
- the server answers with a 500, or a 409 "slot already taken"
- they double-click Pay
- they submit an empty form, a negative quantity, or a
.exeinstead of a PDF - they only use the keyboard, or a 375px phone screen
Every page gets two scores, from 0 to 100:
- Demo: the happy path works, and the main action actually reaches the server.
- Reality: it still works under those conditions.
The grader only counts what a user can see. That rule exists because the first version of this benchmark was easy to fool, which I'll get to.
This sits next to a point other people are already making: AI-generated UI looks finished on the first screen, and Playwright is being used as a safety net before that code gets merged. RealityBench is that check, pointed at the network call.
Models Tested
- Claude Opus 5.5, all 12 tasks, as the reference for a strong coding model.
- Gemini 3.7 Flash, all 12 tasks, as a fast model people use to generate UI in bulk.
GPT-5.6 Sol, Gemini 3.8 Flash, Gemma 4 31B, and Qwen 3 Coder were on the lineup too. On the current graders, GPT and Gemini 3.8 each finished one task before later runs errored (the account's per-call quota reservation ran out). Gemma and Qwen only have scores from an earlier grader. I left all of those out of the table. One task is not a model score.
Each model got the same 12 prompts, one completed run per task.
| Task | What it asks the model to build |
|---|---|
| Authentication & Login | A login form that survives bad credentials, a 500, and a double submit |
| Search & Autocomplete | Search that debounces, shows an empty state, and recovers from a 500 |
| Order Checkout | Checkout that keeps the cart when payment fails and ignores a double click |
| File Upload | An upload control that rejects the wrong type, the wrong size, and a second click |
| Task CRUD Dashboard | Create and delete tasks, and roll back when the server says no |
| Appointment Booking | A booking form that handles a taken slot (409) and a past date |
| Live Chat | A chat box that rejects an empty send and shows an in-thread error |
| Product Detail Page | A product page that disables an out-of-stock option and a negative quantity |
| Admin Data Table | A table that paginates, searches, and stays readable when a field is missing |
| User Settings Panel | Settings that keep unsaved edits when the save fails |
| Dependent API Form | Country and region dropdowns that reset cleanly when the parent changes |
| Multi-Step Workflow Wizard | A wizard that validates each step and keeps earlier answers on Back |
Findings
| Model | Tasks | Demo | Reality | Pages that replaced fetch |
|---|---|---|---|---|
| Claude Opus 5.5 | 12 | 96.9 | 99.6 | 1 |
| Gemini 3.7 Flash | 12 | 69.8 | 86.1 | 7 |
Models build a fake backend into the page
The prompt says "send a POST request to /api/book". Gemini's booking page replaces the browser's fetch and answers locally. This is the model's own code, shortened only by cutting a 409 branch:
/* MOCK SERVER INTERCEPTOR (For self-contained standalone execution) Ensures fetch('/api/book') behaves accurately in standalone browsers. */ const originalFetch = window.fetch; window.fetch = async function (resource, init) { if (typeof resource === "string" && resource.endsWith("/api/book")) { await new Promise((r) => setTimeout(r, 450)); return new Response(JSON.stringify({ status: "confirmed", booking_id: "BK-" + Math.floor(100 + Math.random() * 900), }), { status: 200, headers: { "Content-Type": "application/json" } }); } return originalFetch.apply(this, arguments); }; The page can show "Booking confirmed" and a booking id. Nothing was booked. The model is trying to make a standalone demo work, and the comment says so.
The grader checks this directly. After the page loads, it asks whether window.fetch is still the browser's own function. If not, the report sets fake_backend: true.
On Gemini 3.7 Flash that flag is set on 7 of 12 pages. Eight of its pages failed the happy path, and seven of those eight were flagged. The exception is its login page: demo 62.5, and fetch was still native. Its file-upload page passed the demo (100) and lost the reality points on an unlabeled file input (88.3). A fake server explains most of the misses, not all of them.
Opus has the same instinct and a different default. Seven of its twelve pages contain a mock fetch. Six of them install it only for ?demo, ?mock, or a file:// URL, so a normal load still uses the real function. Those six scored 100 on both demo and reality. The seventh, settings, replaces fetch on every load. The checkbox "Use mock server" starts checked. That is the only Opus page the grader flagged, and its only failed task: demo 62.5, reality 95.
Across the two full runs, 8 of the 9 happy-path failures were pages that had replaced fetch.
My own grader fell for the same kind of trick
The first grader searched the page source. A hidden Booking confirmed! passed the confirmation check, and catch (error) inside a script passed the "error shown" check. Some pages scored higher on the stress tests than on the demo, because I was rewarding text that a user never sees.
It also marked good pages as broken. Opus disables Upload until a valid file is chosen. The grader clicked that disabled button, waited 30 seconds, and recorded a failure. It typed into checkboxes, and it double-clicked the quantity "−" button instead of "Add to Cart".
The grader now reads visible text and the values in form fields, presses the button whose label matches the action, and treats a disabled control as "can't press". The numbers above are from that grader, run on Kaggle. After the fix, Opus has one failed task, and it is the settings page with the mock left on.
What I'd measure next
- Several runs per task, so the table can show a range instead of one sample.
- The rest of the lineup, once a run can finish without the quota reservation killing it.
- The same prompts with one added line: "a real backend is already running at this URL." I want to know whether the fake server disappears, or whether models keep it "just in case."
My Benchmark
- Code: github.com/fab-c14/realitybench
- Kaggle: the twelve RealityBench tasks, each public, with the model runs on the task page. The names and links are in the table above.
You can grade a page locally:
uv sync && uv run playwright install chromium uv run realitybench prompt checkout > spec.txt uv run realitybench grade checkout page.html grade accepts a plain HTML file or a model reply with the HTML in a code block. If the page replaced fetch, it prints a warning.
