hinglish - the romanized hindi-english code-mix typed across india every day - is not english. beam's own model card says it "was developed predominantly for English" (https://huggingface.co/reflection/Beam-1.0-501B-A23B). so here is a simple question: if a model gets the exact same problem in english and in hinglish, what changes?
that question became hinglishbench, a small evaluation i set the direction for: my AI assistant built it and ran it on my behalf, and every number below comes from those runs.
the setup: 82 GSM8K math problems, each existing in two language versions (english original, hinglish translation by beam, checked by an automated numeric-preservation rule; 18 of the original 100 failed that check and were excluded). three models - beam-1.0-501B-A23B, gemma-3-27B-it, qwen3-32B - got the identical zero-shot prompt in both languages:
Solve the problem. You may think briefly, then end your reply with a final line containing ONLY the numeric answer.
grading is exact match on the final line. each run records two things per answer: right or wrong, and how many completion tokens the API reported. note that second one carefully: tokens, not invoices. nothing in this post is a dollar figure.
three models, three different answers
| model | english | hinglish | median token ratio (HI/EN) |
|---|---|---|---|
| Beam-1.0-501B-A23B (effort=low) | 96.3% | 95.1% | 1.47x |
| Gemma-3-27B-it | 91.5% | 93.9% | 0.96x |
| Qwen3-32B | 90.2% | 82.9% | 1.00x |
each model paid differently in this run:
- beam paid in tokens. 96.3% english vs 95.1% hinglish - close, but with 82 pairs and one run per cell, read that as an observed difference, not proof the two are equivalent. meanwhile the median hinglish answer used 1.47x the completion tokens of the english answer to the same problem.
- qwen paid in accuracy. 82.9% vs 90.2% in this run, with no token increase (median ratio 1.00x).
- gemma showed no token increase or accuracy drop in this run. 93.9% vs 91.5%, median ratio 0.96x.
so the honest claim is narrow: a hinglish token tax appears in these runs, for beam. it is not a universal property of hinglish, and it is not a property of every model.
two more datasets, same pattern
these are additional checks, not independent replications - both extra sets are beam-translated or beam-authored - but the pattern repeats:
- 100 author-built tiered pairs: 1.58x median token ratio, accuracy 98.0% in both languages.
- 309 reverse-direction pairs (hinglish-first problems translated to english, so the effect is not limited to english-first items): 1.30x. gemma (0.99x) and qwen (0.94x) again show no tax there.
two prompt variants did not remove it in these runs: a "think in english" system prompt on 30 hinglish items changed accuracy by 0 points and increased tokens by 9.9%, and beam's reasoning_effort=medium showed a ratio of 1.39x against 1.47x at low effort. one sample per cell, so this data cannot estimate run-to-run variance. these results bound the easy explanations; they do not identify the cause.
the honest limits
read these before citing anything above:
- single run per cell, zero-shot, n=82. these are observed values, not proven effects; no significance claims.
- no human validation. all quality checks were automated or model-based: translations are model-generated (by beam itself) and verified by automated rules, and nothing was independently reviewed by a person.
- the median is exactly defined: the upper-middle element of the sorted per-pair ratios (analyze.py,
sorted(...)[n//2]), 1.4706 for beam, reported as 1.47x. - the extra sets share beam's authorship. the tiered set is beam-built, the reverse set is beam-authored and easier with saturated accuracy. they check the pattern; they cannot rule out translation or selection artifacts.
- GSM8K is english-US-context math; the translation preserves context (dollars stay dollars). romanized hinglish only - no devanagari.
why a token count matters at all
serving costs scale with output tokens at flat per-token pricing. so in these runs, beam's hinglish answers carry an output-token count about 1.5x the english one for the same problems - a difference invisible in english-only evals. that is a statement about tokens on 82 pairs, not a bill and not a market claim.
try your own language pair
the runner is python stdlib only, no dependencies. point it at your own HF token, swap in your language pair, and you get the same paired table for whatever you care about - tamil-english, spanish-english, anything.
- dataset + per-item predictions (CC BY-SA 4.0): https://huggingface.co/datasets/MaybeSomeone19/hinglishbench
- explorer, with every raw answer to read yourself: https://huggingface.co/spaces/MaybeSomeone19/hinglishbench
- code, charts, methodology: https://github.com/MaybeSomeone-arc18/hinglishbench
AI disclosure: this post was drafted with AI assistance, and the project behind it - code, translations, runs, analysis, charts - was built and run by my AI assistant on my behalf. i set the direction, and the decision to publish is mine. all numbers come from real runs; every raw output is in the dataset repo linked above, and the limits section is part of the result, not fine print.
