Darwin-27B-ZTC-v5 Tops the Typed Decisions Leaderboard with a 0.81 Accuracy Score

TL;DR: VIDRAFT's Darwin-27B-ZTC-v5 has reclaimed the #1 spot on the Hugging Face-official LocalLLaMA/typed-decisions benchmark, scoring 0.81 accuracy — the highest recorded on this leaderboard to date. Typed Decisions is a structured, probabilistic classification benchmark covering real-world agentic workflows, making this result particularly relevant to engineers building decision-automation and AI agent systems.

What it is

Darwin-27B-ZTC-v5 is the latest model release from VIDRAFT (비드래프트), a Korean Pre-AGI AI startup. It is evaluated under the Hugging Face-official LocalLLaMA/typed-decisions benchmark, which measures a model's ability to make calibrated, structured decisions across multi-dimensional classification tasks derived from enterprise workflow scenarios.

The typed-decisions dataset itself is worth understanding:

• License: Apache-2.0
• Size: 1K–10K samples
• Modalities: Tabular + Text
• Task type: Text Classification (structured decisions)
• Tags: structured-decisions, calibration, probabilistic-classification, system-one, workflow-evaluation
• Language: English
• Subsets: agent_trace_observability, customer_service, invoice_processing, security_incidents — each with 400 rows, split into train (300) and test (100)

Each sample in the dataset includes rich, multi-label output fields: outcome, action, needs_review, risk, and urgency — each with a predicted label, a confidence score, and a full probability distribution. This makes it a calibration-aware benchmark, not just an accuracy benchmark.

How it works

At a conceptual level, Typed Decisions evaluates whether a model can simultaneously classify multiple decision dimensions from a structured workflow state — essentially simulating what an intelligent agent would need to do when observing a running process and deciding how to respond.

Each benchmark sample presents:

• A workflow state description (a structured text blob describing what an agent or system is doing)
• A set of structured questions the model must answer (e.g., "What is the risk level?" or "Does this trace need human review?")
• Gold labels with supporting rationale and factor explanations

The model must output not just a label but a probability distribution across label classes, enabling assessment of calibration quality alongside raw accuracy. For example, the needs_review__probability_true field captures the model's confidence that a human should intervene — a nuanced signal beyond a binary yes/no.

This design mimics System-1 workflow evaluation: fast, structured judgment under realistic enterprise conditions (data erasure queues, security incident triage, invoice routing, customer service escalation, and agent trace observability).

Darwin-27B-ZTC-v5's previous iteration, Darwin-27B-ZTC, already held the top position with a 0.74 score. The v5 release achieves a +0.07 improvement in accuracy on the same leaderboard, representing a meaningful gain in this tightly contested benchmark space.

Benchmarks & results

From the public LocalLLaMA/typed-decisions leaderboard (as of 2026-10-16):

Notable observations:

• Darwin-27B-ZTC-v5 leads by +0.07 over the next closest competitors
• The benchmark is filtered to base models only (no fine-tuning with task-specific supervision declared separately)
• Competing models range from 421M to multi-billion parameters, making Darwin-27B-ZTC-v5's score especially notable given the parameter efficiency context

How to try it

The LocalLLaMA/typed-decisions dataset is publicly available on Hugging Face and can be accessed directly:

# Load the dataset via the Hugging Face Datasets library pip install datasets python -c " from datasets import load_dataset ds = load_dataset('LocalLLaMA/typed-decisions', 'customer_service') print(ds['test'][0]) " 

You can also explore it interactively via Data Studio on the dataset page, or query it with the built-in SQL Console available on the Hugging Face dataset viewer.

For the Darwin-27B-ZTC-v5 model itself: access details and availability should be confirmed directly through VIDRAFT's official channels. The leaderboard entry links to evaluation results via the FINAL-Bench organization on Hugging Face — check there for model card and availability updates.

FAQ

Q: Is Typed Decisions a calibration benchmark or just an accuracy benchmark?
A: Both. Each prediction includes a full probability distribution per label class (e.g., outcome__probabilities, risk__score, urgency__score), meaning models are evaluated on how well-calibrated their confidence is — not just whether the top label is correct. This makes it more demanding than flat classification benchmarks.

Q: What enterprise use cases does this benchmark cover?
A: The four subsets map directly to common enterprise automation scenarios: customer service triage, invoice processing routing, security incident classification, and AI agent trace observability (deciding whether an autonomous agent action needs human review). These are practical, high-stakes decision domains.

Q: How does Darwin-27B-ZTC-v5 differ from the earlier Darwin-27B-ZTC?
A: Based on the public leaderboard data, v5 achieves a 0.81 accuracy score versus 0.74 for the prior version — a +0.07 gain. Internal architectural or training differences are not publicly disclosed.

Originally reported by FINAL-Bench (2026-10-16) — source article.