TL;DR

  • On November 2, 2026 the Hugging Face Terminal-Bench boards list 39 rows for version 2.0 and 42 rows for 2.1, and 0 of the 81 are marked verified. Both versions have 89 tasks.
  • Rank 1 of the 2.0 board (87.6) is a 2.1 score. The model card says "TerminalBench 2.1", the result file is named terminal_bench_2_1.yaml, and it is registered under the 2.0 dataset. The next row is 69.0, so the lead shown is 18.6 points. On the 2.1 board the same number would tie for rank 5.
  • That matters because the benchmark maintainers fixed 28 of the 89 tasks between versions, and their own release post reports a gain of up to 12.1% for one unchanged agent and model pair. A version shift of that size is larger than the whole spread of the top 9 rows on the 2.1 board (5.4 points).
  • Agent harness labels are patchy: 17 of 39 notes on the 2.0 board and 29 of 42 on the 2.1 board name one. Where the same model has a Terminus-2 row and a Claude Code row on 2.1, the gap is only 0.7 to 2.5 points and points both ways.
  • One task is worth 1.12 points. Most of the ordering at the top of either board is a few tasks, not a measured difference.

Top 8 rows of the Terminal-Bench 2.0 board, with the 2.1 score highlighted

What do the two Terminal-Bench boards on Hugging Face contain?

Terminal-Bench is a set of 89 command-line tasks run by an agent inside a container: compile this, fix that server, recover this file. The score is the share of tasks whose hidden tests pass. Hugging Face lists both harborframework/terminal-bench-2.0 and harborframework/terminal-bench-2.1 as benchmark datasets, so a model repo can publish a result in a .eval_results/*.yaml file and the board collects it.

I pulled both boards on November 2, 2026 (KST) from the undocumented JSON behind the leaderboard view. The 2.0 board has 39 rows from 37 repos. The 2.1 board has 42 rows from 36 repos. No row is verified.

The 2.0 dataset card is blunt about this. It opens with a warning: "Warning: The leaderboard above is unofficial." It points to the official leaderboard at tbench.ai, where entries are audited for configuration, show which agent harness was used, and have public trajectories. So the HF board is a collection of self-reports and third-party registrations, and I treat each row as a claim with a source link, not as a measurement the board produced.

Nine of the 39 rows on the 2.0 board cite the official tbench.ai 2.0 leaderboard as their source, and the notes say "agent: Terminus 2". Those are among the best documented rows on the page.

Why is a 2.1 score at rank 1 of the 2.0 board?

The top row of the 2.0 board is mindlab-research/Macaron-V1-Venti at 87.6. Three things about it line up:

  • The model card's benchmark table has a row named "TerminalBench 2.1" with 87.6, and the protocol table says "Harbor framework, Claude Code Agent Harness, 4h timeout" with pass@1.
  • The result file is .eval_results/terminal_bench_2_1.yaml.
  • Inside that file, the dataset id is harborframework/terminal-bench-2.0 and the task id is terminalbench_2.

So the card says 2.1, the filename says 2.1, and the registration says 2.0. I cannot tell from the files whether this was a slip when the YAML was written; the fix would be a one-line change of the dataset id. What I can say is that the board has no check for it, and the result is a lead of 18.6 points over the next row (69.0), which is larger than any other gap on the page.

The card itself is careful. It is built on GLM-5.2, marks imported comparison values with an asterisk, states that the coding and terminal rows use the Claude Code scaffold and not the production harness, and says "Whether reported gains come from specialization, routing, the harness, or their interaction is not resolved by any controlled experiment in this release." The issue is not the claim. It is where the claim is displayed.

On the 2.1 board, 87.6 would sit at rank 5, tied with XiaomiMiMo/MiMo-V2.6-Flash-RL, behind 90.6, 89.9, 88.2 and 87.9. On the 2.0 board it is first by a margin that no other row has.

How different are Terminal-Bench 2.0 and 2.1?

The maintainers' release post (May 6, 2026) says Terminal-Bench 2.1 fixes issues in 28 of the 89 tasks, about 31%. It sorts the problems into three groups:

  • External dependencies, 9 tasks. 2.0 pinned pre-built Docker images, but tasks that touch the internet drifted after release.
  • Resource mismatches, 8 tasks. Some tasks had too little time or compute for at least one valid approach to finish consistently.
  • Misspecification. In query-optimize the tests expected Spark SQL output while the instructions asked for PostgreSQL, so the task was rewritten to use PostgreSQL throughout.

The post adds that no task is unsolved on 2.1, and that most representative agent and model pairs score higher on 2.1. The largest gain it names is Claude Code with Opus 4.6, "12.1%". The post does not say whether that is points or relative, so I read it as an order of magnitude, not a constant. The dataset repository's README counts the changes slightly differently (26 modified tasks), so treat 26 to 28 as the range.

Two consequences follow. First, a 2.0 row and a 2.1 row are different tests, and the difference can be larger than the gaps people argue about. Second, a jump across model generations on different boards cannot be split into "better model" and "easier benchmark". GLM-5.1 shows 63.5 on the 2.0 board (Terminus-2). GLM-5.2 shows 81.0 on the 2.1 board (Terminus-2). That is 17.5 points, and the data cannot say how much of it is the model.

How big is each effect: one task, harness gaps, spread of the top 9, sampling interval, version shift, rank 1 lead

Does the agent harness decide the ranking?

Terminal-Bench scores belong to an agent and a model together. The most common agent on these boards is Terminus-2, the Terminal-Bench team's own agent. Others are Claude Code, a model vendor's own harness, or an in-house setup.

I counted how many row notes name a harness, agent, scaffold or framework at all:

Board Rows Notes naming a harness Notes silent
Terminal-Bench 2.0 39 17 (all Terminus) 22
Terminal-Bench 2.1 42 29 13

On 2.1 the 29 split into 11 Terminus, 8 Claude Code, 3 DeepSeek's own harness, and 7 others such as NexAU, an in-house setup or NVIDIA's evaluator. The 13 silent rows include ranks 2, 5 and 8. For those I cannot say which agent produced the number, only that the model card reports it.

Three rows describe themselves as best-of: GLM-5.2 at 82.7 ("Best reported harness"), Inkling-Small at 64.7 ("Best harness") and Inkling at 63.8 ("Best Harness. Effort=0.99"). Picking the best harness per model is a maximum over several runs of a noisy test, so it is biased upward compared with a row that used one fixed harness. One more row, Qwen3.8-2.4T at 86.6, is reported "under the Qwen3.8-Max product tier ... may not be identical to the raw open checkpoint alone", which is a different artifact from the weights the repo name suggests.

How large is the harness effect on this board? Six repos have both a Terminus-2 row and a Claude Code row on 2.1, so I can compare directly.

Model Terminus-2 Claude Code Terminus-2 minus Claude Code
Ornith-1.5-397B 86.1 85.2 +0.9
GLM-5.2 81.0 82.7 -1.7
Ornith-1.0-397B 77.5 78.2 -0.7
Ornith-1.0-35B 64.2 62.8 +1.4
Ornith-1.5-9B 46.2 47.0 -0.8
Ornith-1.0-9B 43.1 40.6 +2.5

The gaps run from 0.7 to 2.5 points with a median of 1.15, and the sign is split three and three. Five of the six pairs come from one lab with the same sampling settings and "avg of 5 runs", so treat this as six pairs, not six independent labs. The honest reading is that here the harness choice moved scores by one or two tasks and in no consistent direction.

That is not a general law. On the 2.0 board, GLM-5.1 has 69.0 under Claude Code and 63.5 under Terminus-2, a 5.5 point gap. The harness effect can be small or large depending on the model, and a board that does not record the harness cannot tell you which case you are in.

Terminus-2 minus Claude Code for six same-model pairs on the 2.1 board

How much can 89 tasks separate?

One task is 1.12 points. The top 9 rows of the 2.1 board run from 90.6 to 85.2, a spread of 5.4 points, or about 4.8 tasks.

For scale, take a model that solves 85% of tasks and treat one run as 89 independent draws. The standard error is 3.8 points and the 95% interval is about plus or minus 7.4. At 50% the half-width is 10.4. That calculation is a ceiling for unpaired comparisons: every model faces the same 89 tasks, so the real noise in a head-to-head gap is smaller, but the board has no per-task results, so I cannot compute by how much. Averaging 5 runs, as several rows do, removes attempt-to-attempt variation but not the fact that the 89 tasks are a sample.

So 90.6 against 88.2 against 87.9 is not an ordering the data establishes. It is a cluster. The same logic applies to the near-tie at the top of the 2.0 board once the version-mismatched row is set aside: 69.0, 69.0, 68.4, 67.9, 66.7 are within 2.3 points.

Which rows repeat a number from another repo?

Four rows on the 2.0 board, from three repos, cite a different Hugging Face repo as their source.

  • INCModel3/GLM-5.1-MXFP4-Mixed-CT-AutoRound shows 69.0 and 63.5, the same two values as zai-org/GLM-5.1, with that repo as the source. Its card describes an MXFP4 quantization made with auto-round, and I found no Terminal-Bench line in it. A quantized checkpoint can score differently from its base, and these rows do not say it was measured. My October 31 post on the SWE-Bench Pro board found the same repo doing the same thing.
  • RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 shows 31.0 with NVIDIA's repo as the source, the same value as the original.
  • MuVeraAI/Laguna-XS.2 shows 30.1 and cites poolside/Laguna-XS.2, whose own row is 35.7. A 5.6 point disagreement with the cited source and no note about why.

The contrast is on the 2.1 board: NVIDIA's BF16 and NVFP4 Nemotron-3-Ultra rows show 56.4 and 53.9. When a quantized checkpoint is evaluated, the number moves, and the two rows say so.

How do I check these things on any board?

The leaderboard JSON endpoint is undocumented and may change, so treat this as a snapshot script. It reproduces the counts above.

import json, re, collections, urllib.request def board(repo): # undocumented JSON behind the leaderboard view; may change  url = f"https://huggingface.co/api/datasets/{repo}/leaderboard" req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"}) return json.load(urllib.request.urlopen(req, timeout=60)) B20 = board("harborframework/terminal-bench-2.0") B21 = board("harborframework/terminal-bench-2.1") print(len(B20), "rows on 2.0;", len(B21), "rows on 2.1;", sum(r["verified"] for r in B20 + B21), "verified") # 1. a result file whose name says 2.1 but sits on the 2.0 board for r in B20: if re.search(r"2[_.]1", r["filename"]): print("version mismatch:", r["rank"], r["modelId"], r["value"], r["filename"]) # 2. how many notes name an agent harness at all HARNESS = re.compile(r"terminus|claude code|harness|agent|scaffold|codex|openhands|framework", re.I) for name, b in (("2.0", B20), ("2.1", B21)): named = sum(bool(HARNESS.search(r.get("notes") or "")) for r in b) print(f"{name}: harness named in {named} of {len(b)} notes") # 3. same repo, two harnesses, on the 2.1 board by = collections.defaultdict(dict) for r in B21: n = r.get("notes") or "" key = "terminus" if re.search("terminus", n, re.I) else "claude_code" if re.search("claude code", n, re.I) else None if key: by[r["modelId"]][key] = r["value"] for m, v in by.items(): if len(v) == 2: print(f"{m}: Terminus-2 {v['terminus']} vs Claude Code {v['claude_code']} ({v['terminus'] - v['claude_code']:+.1f})") # 4. rows whose cited source is a different Hugging Face repo for r in B20: m = re.match(r"https://huggingface\.co/([^/]+/[^/]+)", r["source"]["url"]) if m and m.group(1) != r["modelId"]: print("source is", m.group(1), "->", r["modelId"], r["value"])

On November 2, 2026 it prints:

39 rows on 2.0; 42 rows on 2.1; 0 verified version mismatch: 1 mindlab-research/Macaron-V1-Venti 87.6 .eval_results/terminal_bench_2_1.yaml 2.0: harness named in 17 of 39 notes 2.1: harness named in 29 of 42 notes ornith-ai/Ornith-1.5-397B: Terminus-2 86.1 vs Claude Code 85.2 (+0.9) zai-org/GLM-5.2: Terminus-2 81 vs Claude Code 82.7 (-1.7) ornith-ai/Ornith-1.0-397B: Terminus-2 77.5 vs Claude Code 78.2 (-0.7) ornith-ai/Ornith-1.0-35B: Terminus-2 64.2 vs Claude Code 62.8 (+1.4) ornith-ai/Ornith-1.5-9B: Terminus-2 46.2 vs Claude Code 47 (-0.8) ornith-ai/Ornith-1.0-9B: Terminus-2 43.1 vs Claude Code 40.6 (+2.5) source is zai-org/GLM-5.1 -> INCModel3/GLM-5.1-MXFP4-Mixed-CT-AutoRound 69 source is zai-org/GLM-5.1 -> INCModel3/GLM-5.1-MXFP4-Mixed-CT-AutoRound 63.5 source is nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 -> RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 31 source is poolside/Laguna-XS.2 -> MuVeraAI/Laguna-XS.2 30.1

The harness classifier reads free text, so it is a lower bound on rows that name one, which is the same limit a reader faces.

What should a reader do with a Terminal-Bench row?

  1. Check the version before the number. The filename, the model card and the dataset id can disagree, as the rank 1 row shows.
  2. Never compare a 2.0 row with a 2.1 row unless the same agent and model pair was run on both. Of the 89 tasks, 28 changed.
  3. Find the harness. If the note is silent, open the card. If the card says "best harness", it is a maximum, not a typical run.
  4. Treat a few points as nothing. One task is 1.12 points, and a single run has a sampling interval of several points.
  5. Look for quantized and mirrored repos whose row cites another repo. A copied number is the base model's score.
  6. Use the official tbench.ai leaderboard for audited entries, and use the HF board to find what to look at, not to rank.

FAQ

Is Terminal-Bench 2.1 harder or easier than 2.0?
The release post says most representative agent and model pairs improve on 2.1 after 28 of 89 tasks were fixed, and that no task is unsolved. In that sense it is easier on average, but the shift differs by agent, model and task.

Is the 87.6 score at rank 1 of the 2.0 board wrong?
Nothing here shows the score is wrong. Its card documents the protocol and labels it Terminal-Bench 2.1. The problem is that it is registered under the 2.0 dataset, so the 2.0 board shows it as a 2.0 result.

Does the harness matter on Terminal-Bench?
Sometimes. Six same-model pairs on the 2.1 board differ by 0.7 to 2.5 points with no consistent sign, while GLM-5.1 differs by 5.5 points on the 2.0 board. There is no single constant to subtract.

How many Terminal-Bench scores on Hugging Face are verified?
None. Across the 2.0 and 2.1 boards, all 81 rows show verified as false.

Which leaderboard is official?
The 2.0 dataset card says the HF leaderboard is unofficial and names tbench.ai as the official one, where entries are audited and the harness is shown.

Can I compare a Terminal-Bench 2.0 score with a 2.1 score?
Only if the same agent and model pair was run on both versions. Otherwise the difference mixes the model, the harness and the 28 changed tasks.

Further reading

Sources

All counts in this article are my own measurements from the board JSON, dataset cards and the cited model cards on November 2, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. The classification of notes is mine, based on free text, and the board endpoint is undocumented and may change.