A few weeks ago I set out to explore decision models in more depth. At first, mainly for lack of time, I used them only for classification. That is what they are designed for, and it is already a valuable use. Still, I suspected they could do more, although I could not yet see how.
So I started experimenting, not to find a breakthrough application, but to understand their limits and their potential. This article presents the first experiment; others will follow. I hope it encourages others to experiment and find useful applications of their own. Constructive feedback is always welcome.
The goal is to experiment with an approach, not to build the best code. Issue labelling is only a convenient test case. None of the versions described here is tuned for accuracy, and a well-tuned ML model could outperform all of them. The numbers are meant to show that the approach works, not to rank decision models against ML.
ssantomauro / self-evolving-triage
Software that rewrites its own rules: a decision model (Levanto Sage 2) and an LLM (Claude) evolve an issue-triage bot from maintainers' behaviour
Self-evolving triage
Software that rewrites its own rules from how people use it, with no developer changing the code.
A decision model (Levanto Sage 2) makes every decision. An LLM (Claude Opus 5.5) reads the mistakes and proposes better rules in plain English. The software tests each proposal on its own recent history and keeps it only if it works.
Goal: experiment with an approach, not build the best code. This repo tests whether a decision model and an LLM can work together so that software improves itself from how people use it. The code exists only to run that experiment and make the result visible.
The experiment builds that loop and lets it run on real data: an issue-triage bot that labels new GitHub issues replayed over the history of five public repositories. The write-up with diagrams and every rewrite proposal is in docs/index.html.
Out of
…From hard-coded rules to rules as data
Much of the logic in a typical product is a set of hard-coded if statements. When user behaviour shows that a rule is wrong, a developer has to change the code and release it. The software improves only when a developer changes it.
Decision models such as Levanto Sage 2 offer a different option. They take content and a question and return a structured answer with a probability, in milliseconds and at a very low cost per call. They can be used wherever the code would otherwise contain an if.
The key point is that the rules a decision model follows are plain-language instructions and definitions. They are data, and data can be updated by a program.
The experiment tests the following setup:
- a decision model makes every decision, quickly and at low cost
- an LLM analyses the incorrect decisions and proposes better instructions
- a validation step on recent history decides whether the new instructions are adopted
If this works, the result is software that improves from how people use it, without changes to its code.
The test case: labelling GitHub issues
The experiment needed a feature where users produce feedback as part of their normal work. Issue triage fits this well: a new issue arrives, the software suggests labels, and the maintainers later apply the labels they consider correct. Every issue therefore shows whether the software's decision was right.
I replayed the history of five public repositories: pandas, black, pydantic, pip and poetry. This amounts to 6,643 issues in chronological order, with about 12 content labels per repository. Each repository has its own labelling conventions, which are not explained to the software.
The replay follows one rule: each version decides first, and only then sees the maintainers' labels.
Four ways to build the same feature
I implemented the same feature four times. Every version receives the same issues in the same order and must answer before the maintainers' labels are revealed. The versions differ in what makes the decision and in what changes over time.
| Version | What makes the decision | What changes over time |
|---|---|---|
| A. Rules | A regular-expression table, as in most auto-labelers | Nothing, unless a developer edits it |
| B. Classic ML | TF-IDF and logistic regression | Its weights, retrained every 50 issues |
| C. Sage, fixed | A Sage 2 tags decision based on label names and descriptions | Nothing; this measures the decision model alone |
| D. Sage + learner | The same Sage decision as C | Its own rules |
Version D is the focus of the experiment. Three parts of it evolve:
- Memory: each issue is sent together with a few similar past issues and their labels.
- Cutoffs: each label has its own probability threshold, tuned on the repository's history.
- Rules: every 150 issues, Claude Opus 5.5 analyses recent mistakes and proposes new instructions and label definitions.
The rewrite step, in simplified form:
every 150 issues: proposal = claude.propose(current_spec, recent_mistakes) if score(proposal, last_60_issues) >= score(current_spec, last_60_issues) + 0.01: current_spec = proposal # the software has changed its own rules log(proposal, scores) # every change is recorded and auditable Claude does not label any issue. Sage makes all 6,643 decisions for each version, while Claude is called only to analyse patterns in the mistakes: 40 times over the whole experiment.
C and D share the same Sage request; D adds about 90 lines of code.
Results
The relevant comparison is C against D: same model, same issues, same starting instructions. The only difference is the learning loop.
Results are measured as label F1, a score from 0 to 1 that shows how often the suggested labels match the maintainers' labels. Both wrong labels and missed labels lower it. F1 is a grade on past decisions, not Sage's probability. The validation step uses the same score.

Label F1 for each block of 100 issues, all five repositories together. Later blocks include fewer repositories, since the histories have different lengths.
C and D start at the same level. C remains flat, while D improves after its first rewrite at issue 160 and maintains the gain.
| Version | F1, all issues | First 50 issues | New labels | Changed by |
|---|---|---|---|---|
| A. Rules | 0.469 | 0.503 | 0.229 | a developer |
| B. Classic ML | 0.760 | 0.552 | 0.068 | retraining |
| C. Sage, fixed | 0.661 | 0.650 | 0.383 | no one |
| D. Sage + learner | 0.778 | 0.663 | 0.619 | itself |
Classic ML performs close to D. This is expected: issue labelling offers thousands of examples and a fixed set of categories, the conditions in which ML performs best. D's advantages lie elsewhere:
- It is useful from the first issue. Sage starts from label names and descriptions, so the first 50 issues of each repository are already labelled reasonably well. ML has no training data at that point.
- It handles new labels. On the first 15 uses of labels that the maintainers introduced during the period, D reaches 0.62 F1, against 0.07 for ML.
How the rules evolved
Every 150 issues, D requested new rules from Claude and tested them. Each dot represents one proposal:
Of 40 proposals, 12 were accepted and 28 rejected. Every repository accepted its first proposal. Later proposals were mostly rejected, as the rules were already well adapted. The table summarises a selection of proposals, with the main change each one made to the instructions; two more are described in detail below.
| Repository | Issue | Outcome | F1, current → proposed | Main change proposed by Claude |
|---|---|---|---|---|
| pandas | 160 | Accepted | 0.80 → 0.84 | Use the title prefix for the type (BUG → Bug, ENH → Enhancement). Mentioning dates, NaN or CSV in an example does not justify a topic label. |
| poetry | 160 | Accepted | 0.64 → 0.92 | Apply one kind label that matches the issue template. Area labels are rarely applied at triage. |
| pydantic | 800 | Accepted | 0.81 → 0.87 | Bug-template reports default to bug V2. Never add topic labels to ordinary bugs. |
| black | 1120 | Accepted | 0.77 → 0.83 | Crashes, parse errors and CI failures get T: bug alone. Requests for new options get T: enhancement, even when filed with the bug template. |
| pip | 960 | Rejected | 0.62 → 0.58 | Choose the type by subject, not by template. This would have reversed the rule pip had adopted at issue 160. |
| pydantic | 1120 | Rejected | 0.88 → 0.79 | Add topic labels for JSON schema, serialization and generics issues. This would have reversed the rule adopted at issue 800. |
| black | 1280 | Rejected | 0.75 → 0.65 | Add C: crash to any error on valid code, although the current rules kept C: labels rare. |
An accepted proposal: pip and its issue templates
For its first 160 issues, the pip bot used the generic starting instructions:
This is a new GitHub issue filed in the pypa/pip repository. Decide which labels the maintainers would apply to it. A label applies only when the issue is clearly about that topic.
The bot over-labelled. It frequently added type: enhancement alongside type: feature request, and applied area labels whenever an issue mentioned the topic. Based on these mistakes, Claude proposed the following instructions:
Maintainers label sparingly: usually exactly one 'type:' label, chosen mainly by issue template. 'Description / Expected behavior / How to Reproduce' issues get 'type: bug', even if the user may be at fault. 'What's the problem this feature will solve?' issues get 'type: feature request'. Add a 'C:' or 'project:' label only when that area is clearly the core subject.
The bot was never told that pip's labels follow its issue templates; it inferred this from the maintainers' behaviour. On the 60 held-out issues, the new rules scored 0.78 F1 against 0.58 for the existing ones, so they were adopted. The effect is visible on the issues that followed:
| Issue | Maintainers | C, original rules | D, rewritten rules |
|---|---|---|---|
| #11460 Support downloading pyproject.toml build deps with parameter | type: feature request | type: feature request type: enhancement | type: feature request |
| #11461 Unclear warnings when extras_require causes unsatisfiable dependencies | type: feature request | type: feature request type: enhancementC: error messagesC: dependency resolution | type: feature request (+ C: dependency resolution marked as uncertain) |
Struck-through labels are incorrect. "Uncertain" means Sage returned null for that label, so it would be passed to a person instead of being applied.
A rejected proposal: poetry and area labels
By issue 160, the poetry bot had learned that its maintainers rarely apply area labels at triage. Around issue 960, Claude analysed a set of mistakes in which some area labels had been used, and proposed reintroducing them:
| Current rules | Claude's proposal |
|---|---|
| "Area labels are applied very rarely at triage; default to NO area label even when the topic seems related (solver, venv, cli, deps, sources, etc.)." | "Then add area labels for the subsystem where the failure clearly lies; multiple areas may apply (e.g., solver+sources+vcs)." |
In the following 60 issues the maintainers rarely used area labels, so the proposal would have added incorrect labels to most of them. The validation step scored it 0.60 F1 against 0.92, and the current rules were retained.
This validation step is what makes it acceptable to run the loop without human review. The LLM can propose a poor change; the software validates every proposal against its own data before adopting it. Seven of the 28 rejected proposals would have reduced the score by more than 0.1.
All 40 proposals, with the full text of every rule changeEvery proposal, accepted or rejected, is logged with both scores and the complete instructions and label definitions in
results/runs/D_Sage_plus_learner/, one .state.json file per repository.Cost
- Sage: about $5.50 for all decisions in the experiment, around 16,900 including the pilot run and every proposal test, or roughly $0.0003 per decision.
- Claude Opus 5.5: 40 calls, one per proposal.
- Infrastructure: none. No training pipeline, no labelled dataset to prepare and no model to host. The learner is about 140 lines of Python.
Because each decision is inexpensive, the software can afford to re-evaluate past issues. This is how the validation step works: before adopting new rules, it replays the 60 most recent issues with both sets of rules and compares the results.
Other applications
Issue labelling is only a test case. The approach requires three conditions:
- a decision the software makes frequently
- a signal that users produce in their normal work and that shows whether the decision was correct
- a decision controlled by its input (instructions, label definitions, examples), so that an LLM can improve it by reasoning about the mistakes
Here are a few products that meet all three and could benefit from this approach. Suggestions for other products and tools are welcome in the comments.
| Product | Decision made by the model | Signal produced by users | What the software rewrites |
|---|---|---|---|
| Support desk | Which team receives a ticket | Which team resolves it | Routing rules per team |
| CRM | Whether a lead should be contacted today | Deals won and lost | The definition of a qualified lead |
| Onboarding | Which next step to show a user | Where users drop off | The rules for each step |
| Moderation | Remove, warn or allow | Successful appeals | The policy definitions |
| Expense management | Approve or send to review | Decisions reversed by finance | The approval policy |
| Urgent or not urgent | Which messages the user opens first | The user's definition of urgent |
In each case the work is divided in the same way. The decision model makes thousands of fast, low-cost decisions. The LLM makes a few slower analyses of what the patterns mean. A validation step decides which changes are adopted. Because the rules are text, every change can be read, explained and reverted.
Sage 2 and Claude are the models used in this run; the approach does not depend on them. The code can be adjusted to use other LLMs and decision models: a new client and request mapping for the decision model, and a different API call in the rewrite step for the LLM. The learner and the validation step remain unchanged.
Limitations
- The loop depends on the quality of its signal. A repository whose maintainers do not label issues provides nothing to learn from.
- The validation step uses 60 recent issues. This is enough to catch clearly poor proposals such as the poetry example, but close cases can be decided either way.
- This is a replay of historical data, not a live product, and five Python repositories are a limited sample.
Reproducing the experiment
The repository contains the four versions, the replay harness and every decision made by each version.
make install && make fetch # download the issue histories make classic # versions A and B, no API costs export SAGE_API_KEY=... # plus ANTHROPIC_API_KEY for version D make sage && make report Every API response is cached, so an interrupted run resumes where it stopped and regenerating the report costs nothing.
This is the first of several experiments I plan to run with decision models. I would be interested to hear where this approach could apply in the products you work on, and about any similar experiments, with Sage or other decision models.



