Ask an AI for your home loan EMI and it will be off by ₹10. Every month.

This is a submission for the Kaggle Benchmarking Challenge

Last week I built a little app about Ananya. She's made up, but you know someone like her. She's 31, she's been saving for eight years, and she's about to sign for a ₹72 lakh 2 BHK whose brochure says 1,050 sq ft. The registration says the carpet area is 720.

While I was building it I kept asking chatbots for help with the numbers: what's ₹72 lakh in crore, what's the price per sq ft of carpet, what's the EMI on the loan. The answers sounded confident. I had no idea whether they were right.

So I benchmarked it.

What I Benchmarked

Lakh, Crore and Carpet Area is 60 small, deterministic questions about the money and property maths that Indian buyers actually do. Every question has exactly one right answer, which I computed in Python, not by hand.

Group	Cases	What it tests	Example
Conversion	15	lakh, crore, LPA, percentages	"₹1.2 crore minus ₹45 lakh, in lakh?"
Area	15	carpet vs built-up vs super built-up, loading, gaj	"Super built-up 1,300 sq ft, carpet 910. Loading as a % of carpet?"
EMI and tax	15	reducing-balance EMI, stamp duty, GST, TDS, brokerage	"₹72,00,000 at 8.4% for 25 years. Monthly EMI?"
Format	15	writing numbers the Indian way	"Write 98765432 with lakh/crore commas" → 9,87,65,432

Each prompt asks the model to finish with ANSWER: <value>. Numbers are checked against the true value, with ±₹2 allowed on EMIs for rounding and an exact string match for the format group. The scorer also flags any response that slips into Western grouping (7,200,000 instead of 72,00,000), even when the final answer is right. I call that western comma drift.

Why this matters to me: hundreds of millions of people think in lakh and crore, and property listings in India are famous for blurring carpet and super built-up area. If a model is going to sit inside a home-buying assistant, these are its first questions, not edge cases.

Models Tested

I picked a spread you'd actually reach for in a product, across three providers and two price tiers:

Gemini 3.7 Flash (Google, fast tier)
Gemini 3.1 Pro Preview (Google, top tier)
Claude Sonnet 4.6 (Anthropic)
GPT-5.4 mini (OpenAI, small tier)

I also queued Claude Opus 5 and GPT-5.5, and Gemma 4 31B is still running as I publish this. More on why they aren't on the board below, because that turned into the most useful lesson of the whole exercise.

Findings

Model	Score	Conversion	Area	EMI and tax	Format	Western comma drift
Gemini 3.7 Flash	1.00	15/15	15/15	15/15	15/15	5%
Gemini 3.1 Pro Preview	0.98	15/15	15/15	14/15	15/15	10%
Claude Sonnet 4.6	0.95	14/15	15/15	13/15	15/15	12%
GPT-5.4 mini	0.92	15/15	15/15	10/15	15/15	10%

1. Carpet area didn't trip anyone. EMIs did.

I expected the area traps to be the bloodbath: loading percentages, "what share of super built-up is carpet", gaj to sq ft. Every model got all 15 right.

The failures piled up in the one place a calculator never fails, compound interest. Look at case e04, a ₹72,00,000 loan at 8.4% for 25 years:

	Monthly EMI
Correct	₹57,492
Claude Sonnet 4.6	₹57,487
Gemini 3.1 Pro Preview	₹57,482

GPT-5.4 mini was worse still. It missed 5 of the 15 EMI-and-tax questions while acing every other group.

None of them said "approximately". They gave a clean, specific, wrong number. ₹10 a month sounds trivial until you remember it's a 300-month loan and the number goes straight into a loan comparison. The models do the arithmetic in their heads, and (1 + r)^300 is exactly where mental arithmetic slips.

What this changed for me: I'd been treating EMI as "easy maths". For an LLM, it's the hardest kind: long, exact and unforgiving. In anything I ship now, the model explains the formula and a tool computes the EMI.

2. Models drift into Western commas even in an Indian question

Every question was framed in lakh and crore, and every model aced the explicit format test when asked directly. Yet 5–12% of answers still wrote something like 4,000,000 somewhere in their working. Gemini 3.7 Flash drifted least (5%), Claude Sonnet 4.6 most (12%), and GPT-5.4 mini and Gemini 3.1 Pro landed at 10%.

That doesn't make the answer wrong, but it's exactly the slip that makes "₹40 lakh" read as "₹4 crore" to a tired buyer at 11 PM.

3. Right value, wrong unit

Claude Sonnet 4.6 missed c14 ("down payment is 20% of a ₹95 lakh flat, in rupees") by answering ₹19 lakh. The maths is right, but I asked for rupees. Real users ask for a specific unit because they're pasting it into a form.

4. My benchmark lied to me first

My first run said Claude Opus 5 scored 0.08 and GPT-5.5 errored completely. I almost wrote a post about it.

Then I opened the run logs. Every failure was a 403 PermissionDeniedError: max estimated cost exceeds your available quota. I'd fired cases in parallel, the expensive models blew through the quota, and my scorer counted each errored call as a wrong answer. Opus 5's printed table actually showed 100% on every group it got to answer.

I fixed it: sequential calls, retries, and every run now prints answered: X of 60 before any score. I dropped the two most expensive models rather than publish a number I can't stand behind.

The lesson: a benchmark score is only as honest as its error handling. If a leaderboard can't tell you how many questions were actually answered, don't trust the leaderboard.

What I'd measure next

Give the models a calculator tool. Do the EMI errors vanish completely, or do models still skip the tool when they feel confident?
Regional units. Bigha, guntha and cents change meaning from state to state, which is a much nastier test than gaj.
State-specific stamp duty and Hinglish phrasing ("72 lakh ka flat, 20% down, EMI kitni?").