This is a submission for the Kaggle Benchmarking Challenge
What I Benchmarked
Most AI benchmarks measure global knowledge. But everyday life in Pakistan runs on things those tests rarely touch: the lakh/crore number system, Pakistan Standard Time (UTC+5), local geography, and Urdu script. A model can be excellent at English trivia and still stumble on "how many lakh are in 2 crore?" while sounding completely sure of itself.
So I built the Pakistan Local Context LLM Suite, a set of short questions with fixed, checkable answers in four categories:
- lakh/crore: converting between lakh, crore, arab, kharab and plain digits, including a monthly-to-annual salary conversion
- PKT time: UTC to Islamabad conversions, including a date rollover past midnight
- Geography and civics: provincial capitals, rivers, K2, the 1973 Constitution, the stock exchange
- Urdu: writing Urdu words and numerals in Urdu script, and the double meaning of "کل" (yesterday and tomorrow)
I chose fixed-answer questions on purpose. Scoring is done by code (regex matching on a normalized response), with no judge model, so every result is reproducible. Models were told to reply with only the final answer, and commas inside numbers (like 12,50,000) are stripped before checking, so formatting does not unfairly punish a correct answer. The benchmark is built with the kaggle-benchmarks library, one task per question, run over a DataFrame.
Dataset
import re, pandas as pd import kaggle_benchmarks as kbench def N(x): # matches a standalone number return rf"(?<![\d.]){re.escape(str(x))}(?![\d]|\.\d)" rows = [ # --- lakh/crore system --- ("lakh_crore","Convert 3.5 crore rupees into plain digits. Answer with digits only.", N(35000000)), ("lakh_crore","How many lakh are in 2 crore? Answer with a number only.", N(200)), ("lakh_crore","Express 12,50,000 in millions. Answer with a number only.", N(1.25)), ("lakh_crore","What is 7 lakh plus 8 lakh? Answer in the form 'X lakh'.", r"(?<![\d.])15\s*lakh|fifteen\s+lakh"), ("lakh_crore","How many zeros are in one arab? Answer with a number only.", N(9)), ("lakh_crore","One kharab equals how many arab? Answer with a number only.", N(100)), ("lakh_crore","A salary is Rs 85,000 per month. What is the annual salary in lakh rupees? Number only.", N(10.2)), ("lakh_crore","What is 2.5 lakh divided by 5, in plain digits only?", N(50000)), # --- time (PKT) --- ("time","Pakistan Standard Time is how many hours ahead of UTC? Number only.", N(5)), ("time","It is 9:00 AM in Islamabad. What time is it in UTC? Use HH:MM 24-hour format.", r"\b0?4:00"), ("time","It is 15:00 UTC. What time is it in Islamabad? Use HH:MM 24-hour format.", r"\b20:00"), ("time","It is 23:30 UTC on 14 August. What is the date in Islamabad? Answer as 'DD Month'.", r"\b15(th)?\s+(of\s+)?august"), # --- geography/civics --- ("geography","What is the capital city of Khyber Pakhtunkhwa?", r"peshawar"), ("geography","Which river is Tarbela Dam built on?", r"indus"), ("geography","Which mountain, the second highest in the world, lies on the Pakistan-China border?", r"k2|godwin|chogori"), ("geography","What is the capital of Gilgit-Baltistan?", r"gilgit"), ("geography","Which province has Quetta as its capital?", r"balochistan|baluchistan"), ("geography","In which year was Pakistan's current Constitution adopted?", r"1973"), ("geography","In which city is the Pakistan Stock Exchange headquartered?", r"karachi"), ("geography","Which province contains the Swat Valley?", r"khyber|kpk|nwfp"), # --- Urdu --- ("urdu","Write the Urdu word for 'book' in Urdu script.", "کتاب"), ("urdu","Write the Urdu word for 'water' in Urdu script.", "پانی"), ("urdu","Write the Urdu word for 'school' in Urdu script.", "اسکول"), ("urdu","Write the number 7 using Urdu-script numerals (just the digit).", "[۷٧]"), ("urdu","The Urdu word 'kal' (کل) has two meanings. List both in English, separated by a comma.", r"(?s)(?=.*yesterday)(?=.*tomorrow)"), ] df = pd.DataFrame(rows, columns=["category","question","pattern"]) df["qid"] = range(len(df)) print(df.shape, df.category.value_counts().to_dict()) Tasks
def norm(t): t = t.strip().lower() return re.sub(r"(?<=\d),(?=\d)", "", t) # remove commas inside numbers @kbench.task(store_task=False) def single_item(llm, qid, category, question, pattern) -> dict: response = llm.prompt(question + "\n\nReply with only the final answer, no explanation.") ok = re.search(pattern, norm(response)) is not None return {"qid": int(qid), "category": category, "question": question, "response": response, "is_correct": bool(ok)} @kbench.task(name="pk_local_context") def pk_task(llm, df) -> tuple[float, float]: """Pakistan local-context benchmark: lakh/crore, PKT time, geography, Urdu.""" with kbench.client.enable_cache(): runs = single_item.evaluate( stop_condition=lambda r: len(r) == df.shape[0], max_attempts=1, llm=[llm], evaluation_data=df, n_jobs=2, timeout=120, remove_run_files=True) ok = runs.as_dataframe().result.str.get("is_correct").astype(float) return float(ok.mean()), float(ok.std()) QUICK TEST
pk_task.run(kbench.llm, df.head(5)) models = ["google/gemini-2.5-flash", "google/gemini-2.5-pro", "meta/llama-3.1-70b"] # edit this frames = [] for m in models: try: r = single_item.evaluate(stop_condition=lambda x: len(x) == df.shape[0], max_attempts=1, llm=[kbench.llms[m]], evaluation_data=df, n_jobs=2, timeout=120, remove_run_files=True) x = pd.DataFrame(list(r.as_dataframe().result)); x["model"] = m frames.append(x) except Exception as e: print("skipped", m, e) res = pd.concat(frames) print(res.pivot_table(index="model", columns="category", values="is_correct", aggfunc="mean").round(2)) print(res.groupby("model").is_correct.mean().round(2)) res[~res.is_correct][["model","qid","question","response"]] # READ these failures THE FINAL RUN
pk_task.run(kbench.llm, df) %choose pk_task Models Tested
I ran the suite on five models through Kaggle Benchmarks, including Qwen 3 235B A22B Instruct, plus models from Google and OpenAI. I wanted a mix of providers and price points, because for a country like Pakistan, where API budgets are small, "which model is good enough, and at what cost?" matters as much as "which model is best?". Every model received identical prompts.
Findings
1. The benchmark is close to saturated.
On the Kaggle leaderboard, the models cluster near the top of the score scale. These questions were not hard enough to separate today's strong models. That is a result in itself: the basics of Pakistani numbering, time and geography are largely solved. My questions were short, single-step and had unambiguous answers.
2. Cost varied far more than accuracy.
On the score-versus-cost chart, the models sit at similar scores, but their cost per run differs by roughly an order of magnitude or more (reading the chart, from about $0.0003 for the cheapest to several thousandths of a dollar for the most expensive). A cheaper model, Qwen 3 235B A22B Instruct among them, landed in the "efficient" zone with a score comparable to much costlier models. For simple local-context tasks, paying more bought almost nothing.
3. What a near-perfect score does not tell you.
A high score on 25-ish short questions does not mean a model understands Pakistan. It means it handles short, well-formed, single-answer questions. Real use is messier: Roman Urdu typed on a phone, long tax documents, ambiguous dates, mixed Urdu-English sentences. None of that is in this version.
4. Scoring design matters as much as the questions.
Regex scoring is strict about format, so a correct answer written in an unexpected way can be marked wrong, while a lenient pattern can accept a sloppy answer. I handled the most common case (commas in numbers), but it is a reminder that a benchmark can end up measuring its own grading rules.
Limitations
- The question set is small, so tiny differences between models should not be over-read.
- Regex scoring cannot recognize every valid phrasing.
- Questions are single-turn and in clean text, not messy real-world input.
What I would measure next
- Harder, multi-step items: tax calculations, legal clauses, mixed Urdu-English documents.
- Roman Urdu, which is how many people actually type.
- Repeated runs of each question to measure consistency, not just accuracy.
- Cost per correct answer, since the cost chart suggests that is where the real differences are.
