We're a small team of software architects, and between us we've spent more than 19 years designing enterprise systems. We've lost count of the meetings that went sideways because of one wrong arrow on a slide. Once someone spots it, nobody trusts the rest of the picture. Fair enough. We wouldn't either.
So when we started building CloudSketch AI, a tool that turns a plain-English description into an architecture diagram, we had one fear. Not that it would be slow, or ugly. That it would be confidently wrong.
Every AI diagram tool looks great in a demo. You type the same five prompts you always type, and out come five lovely diagrams. The trouble starts around prompt 37: a payroll system in Germany, a farm in Maharashtra, a Git workflow with hotfixes and three release branches. That's where we wanted to look.
So before asking anyone to pay us a rupee, we built a test we couldn't quietly tune our way around. Across nearly 4,000 fresh prompts (about 7,700 graded diagrams) and all 18 diagram types, 95 to 100% of the diagrams came out with no real error, and every single one drew on screen. Getting there was humbling. This is what we measured, what broke, and what we changed.
How we set up the test
For each diagram type we wrote sets of 40 prompts we had never seen before, then sealed them. A set runs once. If we change any code after that, the set stops counting as proof and we write a new one. That rule cost us a lot of evenings, and it's the only reason we trust the numbers.
We wanted the prompts to look like real requests from real places, so every set has 10 prompts set in India, 20 in the rest of the world, and 10 with no country at all. The rest of the world gets double weight on purpose, because that's where we were weakest (more on that later).
An AI reviewer grades each diagram as pass, weak or fail. A fail is a wrong or broken diagram. "No real error" means pass or weak, and we always report the strict score (pass only) next to it, because a weak diagram is one we'd still tidy up before a client meeting.
Then every diagram has to actually draw. We render each one in a real browser with the same Mermaid version the site uses. If it doesn't draw, it's a failure, however clever the text looks.
A type only counts once two fresh sets score 95% or more on the same code. One lucky run proves nothing.
Three views, three levels of detail
One thing we learned the hard way as architects: the same system needs a different picture for every room you walk into. So every diagram in CloudSketch AI can be written for one of three readers:
- Technical: for engineers. Real component names, protocols, ports, the lot.
- Managers: plain language. What happens, who does it, where it can go wrong.
- Investors: value and flow. Where the money and the customers move.
On top of that, each diagram has three levels of detail. Extreme shows everything. Standard keeps the main path. Executive is the big picture for the CEO, who will still ask for a bigger font. The AI writes only the full diagram; the two smaller levels are cut down from it by plain code, so they always agree with each other and switching between them costs nothing.
We want to be straight about what the test covered. Every prompt ran twice, once for the Technical view and once for the Managers view, so each set is 80 diagrams. The reviewer graded the full diagram, the Extreme level. The Investors view and the Standard and Executive cuts weren't graded separately in these sets. They're next on our list.
The results
All 18 types cleared the bar on two fresh sets. We've put the strict score next to each one too, because hiding it is exactly what made us distrust demos in the first place.
| Diagram type | No real error, set 1 | No real error, set 2 | Strict (pass only) |
|---|---|---|---|
| ERD | 100% | 100% | 90–96% |
| C4 (three sets) | 98.6% | 98.6% · 97.2% | 90–94% |
| Deployment | 99.0% | 96.9% | 88–92% |
| Data flow | 97.5% | 98.8% | 87.5% |
| Mind map | 96.5% | 97.9% | 94–95% |
| Sequence | 96.3% | 97.4% | 86–91% |
| Chart (pie, bar, line) | 97.5% | 96.3% | 90–92.5% |
| Sankey | 96.3% | 98.8% | 86–97.5% |
| Class | 96.3% | 96.3% | 89–90% |
| Network | 96.3% | 96.3% | 96.3% |
| Use case | 96.3% | 96.3% | 77.5–84% |
| State machine (three sets) | 97.5% | 93.8% · 96.3% | 82.5–90% |
| Gantt (project plans) | 97.5% | 96.3% | 89–91% |
| User journey | 95.0% | 96.3% | 95% |
| Flowchart | 95.0% | 96.3% | 92–94% |
| Swimlane | 96.3% | 95.0% | 80–82.5% |
| Prioritisation matrix | 96.3% | 95.0% | 91–92.5% |
| Gitgraph | 96.3% | 95.0% | 81–89% |
State is the one that made us sweat: its second set came in at 93.8%, so we ran a third on the same code and counted all three together (95.8%). Gantt passes for software, IT and business project plans. For crop seasons, festivals and events it's still beta, and we'll explain why below.
What broke
Diagrams that never drew
Early on, our scores looked healthy. Then we opened a few diagrams by hand and some of them simply didn't render. The reviewer reads the code, so it had no idea. That was a bad afternoon. Since then, every run ends by drawing every diagram in a browser, and a blank one counts as a failure, full stop.
Mermaid has rules the AI doesn't know
We draw everything with Mermaid, and Mermaid has some sharp edges. In one day we hit three of them. Sankey charts refuse anything that isn't plain ASCII, so a single rupee sign in a name and nothing appears. Bar charts with more than one series stack the bars on top of each other and show no legend. And Git graphs refuse impossible steps, like merging a branch into itself. The model has no way of knowing any of this, so we now fix each of these in code before the diagram reaches you.
Examples in a prompt get copied
This one surprised us. Our Gantt instructions once said "for example, RERA approval for housing projects". Soon RERA was turning up in plans that had nothing to do with housing. The same happened with UPI in customer journeys and a two-way arrow in network diagrams. Now we write rules (when something applies) and leave the examples out.
Past 94%, code beats more words
For network, swimlane, Sankey, chart and Git diagrams, the last few points never came from better prompts. Every extra instruction just moved the mistakes somewhere else. What worked was small, boring code: no link that skips the firewall, every step inside a lane, flows that actually add up.
The grader gets things wrong too
Our AI reviewer misread arrow directions in use case diagrams and miscounted Gantt dates. So wherever code can measure a fact (dates, totals, durations), we measure it ourselves and hand the reviewer the numbers.
The rest of the world is harder than home
Prompts set outside India were the weakest part of almost every set. Units, regulations, local names: small things that trip the model up. That's why new sets carry twice as many of them.
Some mistakes are about knowledge, not drawing
Gantt plans for projects passed easily. Plans for crop seasons and festivals kept failing, and not because of the drawing: the model simply didn't know when a harvest really happens, or what has to wait until it's over. You can't fix that with a better layout. So we claim Gantt for project plans only and call the rest beta.
One small one from this week, because it made us laugh. A payroll diagram labelled the SQL Server connection "TDS/TCS". TDS is the database protocol. TCS is an Indian tax. The model saw "payroll" and "tax" and helpfully mixed the two. That's a code rule now too.
What we're not claiming
This is our own test, graded by AI. Nobody independent has checked it, so please read it as exactly that: an internal, AI-graded test.
"No real error" isn't "perfect". Strict scores run from 77.5% to 97.5%, and a weak diagram is usable but still worth a look before it goes in front of a client. Some types only just made it: flowchart, state, swimlane, prioritisation, Sankey and Git diagrams cleared 95% by a small margin. The Investors view and the smaller detail levels haven't been graded on their own yet.
And your prompt isn't our prompt. A test set is a sample. If you find a diagram that's wrong, we genuinely want to see it. That's how the next set gets harder.
Try it on a system you know
The fairest test is your own. Describe a system you know inside out, switch between the Technical, Managers and Investors views, flip through Executive, Standard and Extreme, and check it against what you know. You can try it without signing up at cloudsketchai.com, and every diagram in the template library shows the exact prompt that made it.
Then tell us where it went wrong. We read every message.
— The CloudSketch AI Team
