Two models, two rankings

A frozen GPT-5.6 Sol and a frozen Claude Fable 5 each gain about 35 points of held-out score by rewriting their own tools and notes. Sol levels off after about $120 of learning cost; Fable after about $610. Sol gets the far better plasticity score, 298 points per $1,000 against 57, but Fable ends with the higher score.

So which one should you deploy? The Agent Plasticity paper is careful not to answer that, and its headline number can't. Its learning curves can, though, and three other recent papers turn them into a strategy: let a strong model improve its harness, use that harness to hand work down to cheaper models, and on open weights, consolidate what has stabilized into the weights.

One detail from the paper's live games sets the stakes. At checkpoint 0, Fable reasoned through a chess game move by move with about 102,000 output tokens. At checkpoint 20, it loaded the engine it had written during training and used 2,000 to 7,000. The agent got better and cheaper to run at once, and the plasticity metric counts only the first.

What agent plasticity measures

Agent plasticity is held-out gain per dollar of learning cost, with model weights frozen. All learning goes into a persistent inventory of Python tools, skills and notes that each fresh-context episode inherits. Held-out games never reach the reflection step, so gains there are evidence that the artifacts generalize.

The headline number, P^sat, fits a saturating curve to held-out score against cumulative learning cost. It divides the fitted gain at saturation (the first checkpoint reaching 90% of the fitted rise) by the cost of getting there. Learning cost includes rejected edits; held-out evaluation and deployment costs are excluded.

The authors scope the metric narrowly: plasticity "does not define self-improvement itself." It measures one property, how efficiently capability is acquired, and they note it depends on provider prices and on the saturation fit. Everything below builds on that scoping rather than arguing with it.

The question plasticity doesn't answer

A deployment has a quality bar τ, and a model whose ceiling sits below τ is useless however efficiently it learns. In Chess (Hard), Sol climbs from near 0% to 36.9% on held-out games. For a team that needs 70%, that curve is irrelevant.

Score points are also not equally valuable across the range. Going from 70% to 85% halves the error rate; going from 0% to 15% barely dents it. Relative error reduction is a fairer measure of how much a model learned:

Chess (Hard), held-out ID Start Mean of ckpts 16–20 Gain (points) Error removed
Claude Fable 5 37.5% 73.3% 35.8 57%
GPT-5.6 Sol ~0% 36.9% ~36.9 37%

On points the two gained about the same; on error removed, Fable learned more. Log-odds would be more principled but diverge at Sol's 0% start, which is the headroom problem in another form. These are Chess (Hard) numbers; the 298 vs 57 plasticity figures average chess, Go and Hex.

Cost-to-threshold: a deployment metric from the same curves

For deployment, read each model's fitted curve at your own threshold τ, not at its own saturation point. Call the learning spend at which model m reliably reaches τ its cost-to-threshold, C_m(τ). Then pick the model with the lowest total cost over N deployed tasks, among models whose ceiling clears τ:

\min_m \; C_m(\tau) + N\left(c^{\text{act}}_m + p^{\text{fail}}_m \, c^{\text{review}}\right) \quad \text{subject to} \quad S_m^{\infty} \ge \tau

Here c^act is the per-task acting cost after learning, p^fail the failure rate at τ, and c^review the cost of catching one failure. Three things change relative to P^sat:

  • Feasibility comes first. A model that cannot reach τ is excluded, whatever its efficiency.
  • Operating cost is in the objective. Artifacts can cut per-task cost, as Fable's engine did, or raise it when a bloated inventory is reread every episode.
  • Volume decides. At small N, pick the best starting score per acting dollar. At large N, ceilings and learning slopes dominate.

How long to keep learning is a marginal question, not an average one. Keep investing while N·v·S′(C) exceeds 1 + N·c′_act(C), where v is the value of one score point. P^sat is an average return, gain divided by cost.

One definition needs care, because learning curves are not monotone. Claude Opus 5's Go score collapsed after a single bad tool edit at its final checkpoint. I define C_m(τ) as the cost after which the lower confidence bound of held-out score stays at or above τ for the rest of the run, and report "not reached" otherwise. Crossing τ once and falling back is not reaching it.

Units: what dollars, GPU-hours, FLOPs and time each answer

Dollars are the right unit for a deployment decision and the wrong one for a scientific claim. Prices move with vendor strategy, so an agent's plasticity changes when a price list does. Raw token counts are no fix: input, cached, output and reasoning tokens cost different amounts, and weighting them equally is just another price list.

Tokens also aren't comparable across models. Tokenizers differ, and a token from a large dense model is far more compute than one from a small MoE.

Unit Answers Caveat
Dollars at a dated market price Comparison with published results Price open-weight models at a hosted per-token rate to share the paper's unit
GPU-hours × your rate at measured utilization What learning actually costs you Depends on batching, quantization and utilization
FLOPs (about 2·N_active per token, plus attention) Learning efficiency as a scientific claim Needs known parameter counts; undercounts bandwidth-bound decode
Wall-clock time-to-threshold Whether learning finishes in time to matter Parallelism makes it a poor cost proxy; report it separately

Open weights change more than the price. Marginal rollouts on owned hardware are cheap, which buys more lineages and larger validation sets, the two places where variance hurts these papers most. A harness injected as a fixed prompt prefix is nearly free to reread under prefix caching in vLLM or SGLang. And open weights allow a final step closed models can't take: moving stable harness knowledge into the weights.

Making harness learning compound

WikiSkill adds a persistent knowledge layer between raw traces and executable skills. An inference agent runs tasks with the current skills, a wiki maintainer turns traces into pattern pages, a skill proposer edits one skill at a time, and a validation gate accepts or rolls back the edit. The wiki is never rolled back, so a rejected edit still leaves its lesson behind.

Within the Qwen family, average gains over no skills grow with size: +12.3 points at 4B, +17.5 at 9B and +23.9 at 27B. Yet Qwen-3.5-9B with evolved skills (47.4%) beats Qwen-3.6-27B without them (39.4%). Removing the wiki from the skill proposer drops Gemini-3.5-Flash's average from 63.7% to 48.7%.

Skills also transfer across models, sometimes better than self-evolved ones. On ALFWorld, Qwen-3.5-9B scores 63.4% with its own skill and 70.2% with one evolved by Qwen-3.6-27B. The authors read this as evidence that discovering skills and executing them are separate capabilities.

Transfer cuts both ways, and the failures are the useful part:

  • Weak author, strong user. Qwen-3.5-4B's SpreadSheet skills drop Gemini-3.5-Flash from 50.5% to 18.1%. They encode low-level workarounds that stop the stronger model from writing end-to-end scripts.
  • Strong procedure, weak executor. On OfficeQA, Qwen-3.5-4B fails to follow multi-step long-context search procedures and falls back to its default reading behavior.

The working rule: a skill fits the capabilities of the model whose failures it was built from. That rule drives the delegation strategy below.

Progressive delegation: start strong, hand work down

Start each task on the strongest model you can afford, and let the harness it builds carry the work down to cheaper models, one tested promotion at a time. A ladder might run Claude Fable → Opus → Sonnet → Haiku → an open-weight model on your own GPUs. That ladder is my extrapolation: the transfer evidence above comes from Qwen, Gemma and Gemini, not Claude tiers.

[embed: node/9eb4f389-c807]

Each task moves down only after its re-fitted harness passes every regression suite, and climbs back when its failure rate rises.

Four rules make the ladder work.

  1. Promote per task type, not per model. Extraction and routing may reach the bottom rung; long-context synthesis may never leave the top. Where each task stalls is itself the measurement.
  2. Promotion means re-fitting the harness. Run the cheaper model on the task, collect its failures, and let the strong model rewrite the harness from them. The student generates the traces and the teacher does the diagnosis, so skills fit the model that will run them.
  3. Gate on regression, with enough data. A promoted tier must pass every earlier test suite, not only the current one. Validation sets must be big enough to gate: Gemini-3.5-Flash scored 100% on WikiSkill's ALFWorld validation split and 85.9% on test.
  4. Keep a way back up. Vendors update models and task distributions drift. Escalate a task back up the ladder when its failure rate crosses the gate.

Moving a task from tier k to the cheaper tier k+1 pays when the per-task savings over the deployment volume exceed the costs it adds:

N\left(c_k - c_{k+1}\right) \;>\; C^{\text{refit}} + C^{\text{test}} + N\left(p^{\text{fail}}_{k+1} - p^{\text{fail}}_{k}\right) c^{\text{review}}

High-volume tasks pay back quickly; rare ones stay on the strong model.

This resembles model cascades and routers such as FrugalGPT and RouteLLM, which send each query to the cheapest model likely to succeed. The difference is time. Here the routing thresholds move because the harness improves, so a model's place on the ladder is not a fixed property of the model.

Consolidation: from harness to weights

On open weights, the harness can be a staging area for weight updates. Learn in files, where edits are fast, auditable and reversible; once a skill has stayed stable across promotions, distill it into the model. This is the wake–sleep pattern of Language Models Need Sleep, with the fast memory moved out of the architecture and into the harness.

That paper moves knowledge from fast, temporary components into a new low-rank expert in a slower block. The new expert is trained to imitate a teacher holding old and new knowledge together, so old behavior is the target rather than collateral damage. Its Dreaming stage builds on SEAL, and the method is built on the authors' own architecture and training loop rather than an off-the-shelf model.

The harness version is context distillation. The teacher is the small model with the harness in its prompt; the student is the same model without it, plus a new LoRA trained to match the teacher's outputs. The LoRA becomes a cache of the wiki, and you can test whether the cache still agrees with its source.

SEAL generates the training data for that step, and two of its properties fit this design:

  • It needs a downstream evaluation for every context. Its authors name this as what keeps it from scaling to unlabeled data. A harness loop always has one: the validation gate or the held-out tasks.
  • Its authors already describe a teacher–student variant. A separate teacher is trained by RL to write edits that maximize the student's improvement. That is the delegation ladder applied to weights.

Two problems remain open. SEAL reports that knowledge from earlier edits gradually degrades as sequential self-edits accumulate. And each SEAL reward requires fine-tuning and evaluating a model, about 30–45 seconds per candidate edit.

Consolidating from a harness rather than replacing it gives a rollback: if a LoRA degrades anything, drop it and the harness still works. Consolidation then becomes one more promotion. "Model plus LoRA plus a thinner harness" must beat "model plus full harness" on the same regression suite, at lower cost per task.

Professional work: where verification costs the most

In professional work, verification cost favors the stronger model. Chess has a cheap, strong verifier in an engine. In legal or medical work the verifier is an expert's time, so review cost scales with the failure rate, and a better model at a higher token price can be cheaper in total.

Verification cost also limits plasticity itself, because feedback is what the harness learns from. The plasticity paper's environments form a gradient: chess gives a per-move engine verdict, Go and Hex get lighter diagnostics, and NetHack has no per-action oracle. In NetHack only one of six lineages improved reliably, and Sol, the most plastic model on board games, fell from a mean score of about 570 to 420.

That suggests a feedback ladder for any domain:

Rung Reflection sees Tests
0 Nothing beyond validator checks that tools run Priors plus a compute budget
1 Trajectories with outcomes stripped Learning from process alone
2 Trajectories plus the final outcome Learning from sparse results
3 Plus per-step expert annotations The plasticity paper's chess setup

Most professional work sits at rung 2: you learn whether the brief won, rarely which paragraph lost it. The gap between rungs 2 and 3 estimates how much measured plasticity survives outside games. Until someone measures it, read game-based plasticity as an upper bound for professional domains.

This is also why I'd call these loops autonomous between gates, not autonomous overall. Every gate here is a validation set with ground truth that a person built.

Two experiments worth running

Both experiments fit on open weights and a modest GPU budget.

1. The feedback ladder. Run the plasticity protocol at rungs 0 to 3 with budget-matched reflection. Rung 0 is the control the plasticity authors name as missing: it separates learning from feedback from simply thinking harder with more compute. A related counterfactual design, which reruns from a branch point with the agent's experience erased, appears in recent work on long-horizon R&D agents.

Rung 1 is hard to build in chess, since a checkmate is visible in the final position. Truncating games works but changes what reflection sees, so report it as a limit.

2. Harness-to-LoRA consolidation. Take a WikiSkill-style evolved skill for a small Qwen model and context-distill it into a LoRA. Measure three things:

  • Task score with the skill removed from the prompt, against the skill-in-prompt baseline.
  • Score on an unrelated benchmark before and after, as a forgetting check.
  • Prompt tokens saved per call, and the volume N at which distillation pays for itself.

A third arm would test the student-plays, teacher-diagnoses design: a small model generates the traces while a larger model maintains the wiki and proposes skills. As far as I can tell from WikiSkill's main text, its roles are never split that way.

Caveats

  • The plasticity results come from games with automatic scoring and small evaluation sets. Claude Fable 5, the highest-scoring model, has one run in Chess (Hard); most others have three.
  • P^sat and every dollar figure here depend on provider prices at the time of the study.
  • WikiSkill's validation sets are small enough that a perfect validation score coexisted with 85.9% on test.
  • The Claude-tier delegation ladder is an extrapolation; the transfer evidence comes from Qwen, Gemma and Gemini.
  • Consolidation from a harness is a proposal. SEAL shows forgetting under sequential edits, and retention across many consolidation cycles at production scale is unshown.
  • The cost formulas are decision aids. Their inputs, especially review cost and failure rate at τ, must be measured per deployment.

References