OpenJev and keyword rules tied on 18 issues. The labels needed a closer look.

On a small set of public Ruff issues, OpenJev and a keyword rule each matched 11 of 18 saved labels. The useful finding came from reading the reports behind that score: a repository label can describe the eventual fix rather than the request a model is being asked to classify.

AI disclosure: An AI agent ran the local checks and drafted this article from their saved outputs. The measurements below come from executed code, not generated example results.

The question

Can a small local model sort incoming issues into bug, documentation and feature requests better than simple rules?

This test used mjdileep/OpenJev, an independent Jev-style project. It is not an official open release of TypeSafe's Jev. The model was Qwen3.5-0.8B, quantized to 4-bit, running through MLX on an Apple M3 Pro with 18 GiB of system memory and Python 3.12.11. No hosted model was tested.

The saved data contained 24 public Ruff reports: six development examples and 18 evaluation examples. Each evaluation class had six reports. The reference categories came from the upstream labels bug, documentation and wish, with wish mapped to feature.

The sample came from GitHub's returned order within each label, followed by a fixed shuffle. It included closed reports. It was not a random sample of new requests.

The rules checked the title for documentation words first, then words such as “feature”, “request”, “add” and “support”; everything else became bug. The model received the title and up to 1,600 body characters. It did not receive the label. That input difference matters: this compares two recipes, rather than isolating the effect of the model.

The result

The prompt and rules were kept fixed for the evaluation. No model training was performed.

The methods made different mistakes. Nine reports matched both methods, two matched only the rules, two matched only OpenJev, and five matched neither.

A second run used the same saved texts, prompt and model revision. All 18 OpenJev choices stayed the same. Checks also verified input hashes, evaluation membership, reported totals and the relationship between candidate scores and selected answers.

OpenJev's median request time in the repeat was 0.513 seconds, excluding model loading and warm-up. Cached loading took 2.153 seconds and warm-up took 0.217 seconds. These are single-machine observations from one repeat, not a general speed benchmark. The rule lookup was much faster in this setup.

Why 11 out of 18 is not an accuracy claim

Consider Ruff #28493. The report describes a supposedly safe fix creating a type-checker error. The maintainer explains that Ruff's safety definition concerns runtime behaviour and comments, and accepts documenting the limitation. The documentation label makes sense for that resolution. A classifier reading only the original report could reasonably choose bug.

Ruff #28248 describes missing namespace-package diagnostics. The discussion distinguishes preview behaviour and asks for clearer documentation. Again, the final label is not an unambiguous description of the original request.

The mapping from wish to feature also loses information. Ruff #22535 has that label but describes unwanted debug output, with a maintainer accepting a fix.

These examples were inspected after the evaluation. They were not used to change the reference answers or improve the reported score. They show why the measured quantity is agreement with repository labels, not independently checked classification accuracy.

A normalized score is not a confidence estimate

OpenJev scores each candidate using the language model's next-token support for “yes”. For a choice, it divides each support by their sum and selects the largest. Its architecture notes explicitly say the scores are uncalibrated.

For one saved report, the three supports were approximately:

supports = {"bug": 0.737, "documentation": 0.638, "feature": 0.673} total = sum(supports.values()) shares = {name: value / total for name, value in supports.items()} # Approximately: bug=0.360, documentation=0.312, feature=0.329 

Those shares add to one because of the arithmetic. That does not establish a 36% chance that bug is correct. Using them to decide which requests can skip human review would require separate evidence.

The decision

This test provides no reason to replace the simple rules for this toy task. It also says nothing about replacing a larger model: none was tested. The sample is small, comes from one repository, uses imperfect reference labels and may contain reports seen during model training.

Before extending this comparison, the target needs a precise definition: classify what the reporter asks for, or predict the maintainer's eventual action. Those are different tasks. New reports would need answers assigned under that definition before another model run, with ambiguous cases kept visible and matched inputs for both methods.

For this experiment, checking what the labels meant was more useful than adding another model to the comparison.