Five datasets, totally different formats. Five different answers to one simple question: what is one row?
- SWE-smith: a task with a bug put in it.
- SWE-Gym: a task.
- ToolACE: a made up dialogue.
- CommitPackFT: a before and after pair, no process at all.
- MegaBit-trajctories(mine): one full session that actually ran.
Put those in one table the lazy way and every number after is a lie.
So I built a method to compare things that share almost nothing. It is all CPU. No GPU, no training, no weeks. One script and an afternoon. Here it is, step by step, so you can use it on anything.
Step 1: Fix the unit before any number
Row zero is not a number. It is a question: what is one row?
Write the answer for every dataset before you count anything. If you skip this, every count after it compares apples to oranges to a rock.
And where a dataset has no process, write N/A. Not zero. Zero is an insult. N/A is the truth. That one choice is most of the fairness.
Step 2: Compare the box, not the effect
When it is just data, do not ask what it does to a model. Ask what is inside the box.
Model effects need GPUs and weeks. Box contents need a CPU and an afternoon. Box contents are also the only thing you can actually check today.
Step 3: Run the six buckets of checks
Size and cost: rows, repos, languages. Tokens with one tokenizer on the exact text you would train on. Download size, and can you load it in one line.
Process depth: assistant turns per row, tool calls per row, percent of rows with a real tool result.
Real execution: are tool outputs real? Look for exit codes, stderr, durations, real tracebacks. Does the card say how it ran, and can you spot check it.
Failure and recovery: percent of rows with a real failing run. Percent with a rejected edit. Percent where an error is followed by another action instead of the row just ending. Longest error into retry chain.
Target and proof: where did the task come from, a human commit, an injected bug, or a made up spec. Is there a reference to score against. Does it ship a verifier, hashes, a per row license and commit, and does the verifier actually pass.
Packaging and honesty: tool schemas, truncation policy and flags, split labels with reasons, partial runs flagged, limits disclosed.
Step 4: Normalize or it means nothing
Every count twice: per row, and per 1k tokens. A long row can fake a big count. Per token cannot cheat.
One tokenizer for all of them, or your token numbers mean five different things.
Pin the version, the hash, and the date for every dataset, including your own. Mine changed the day before I published. If I had not pinned mine, my table was stale by morning.
Step 5: Invent one metric, then let it hurt you
Pick the one thing you actually care about and make a number for it. Mine was recovery density: a real error followed by a real fix, per 10k tokens.
Then compute it for everyone, including yourself, and print whatever comes out.
SWE-Gym beat me on my own metric. I published it. That one loss buys trust for every row I win. If your invented metric cannot lose, it is not a metric. It is a slogan.
Step 6: Know the traps
Do not call a dataset fake if it executes for real. SWE-smith and SWE-Gym run real code. My difference from them is where tasks come from and how it is packaged, not whether it is real.
Synthetic by design is a different tool, not a fraud. ToolACE says so on its own card. Quote the exact card line, with a link. No vibes.
Analyze their data. Do not redistribute their rows. Put the scale row at the top, not hidden at the bottom.
Step 7: Publish the rows you lose
A table where you win every row is marketing. A table where you lose six rows is measurement.
Mine loses on size, repos, languages, retry chain length, published training proof, and contamination. All of it in the table, in the open.
This is the step everyone skips and the step that makes the rest believable.
What you ship
One table, five columns, about eighteen rows, N/A where it does not apply. The script. The raw output per dataset. A versions and dates table. No performance claims anywhere in the file.
Then when someone asks "why not just use the big one?", you send one link.
The table this method produced is here: I did this with my own dataset.
The data itself is free, MIT, no signup:
- GitHub: https://github.com/Sumama-Jameel/megabit-trajectories
- Kaggle: https://www.kaggle.com/datasets/sumamadev/megabit-trajectories
Use this on anything. Datasets, models, tools, two job offers. The unit question and the lose-rows rule work everywhere.
One question
Tell me a comparison you have seen that was really a sales deck. Which row did they hide?
I read AND REPLY to every comment.
