I picked the skill with more stars. Its repo had 46 skills tied for first place.

Tuesday afternoon. Two skills, same job: turn a messy commit history into release notes. The first one lives in a repo you've heard of — thousands of stars, active maintainers, the README is a wall of badges. The second one is by a solo author you've never seen before, with a fraction of the attention. You don't deliberate long. You pick the stars. It's not even a decision, it's a reflex — the same reflex that made you pick the popular library over the unknown one for the last decade. You install it, it works, you move on with your day.

Your colleague, the one who watches you work, raises an eyebrow. "Why that one?" he asks. "It's got more stars," you say, as if that closes the argument. He doesn't push it, but you catch the look — the one that says "everyone uses it" is not the same as "it fits you". You file the look away and go back to work.

Then, that evening, you open the directory to browse for something else — and you notice something about the popular repo. Its skills are all over the first page of results. You click into the ranking to see which one is best, and that's when you see it:

Forty-six skills from that repo, tied for first place.

Not "ranked in the top 46". Tied. Identical score. The system's honest answer to "which of these should I install?" was a list of 46 things, in no particular order. The stars told you the repo was worth attention. They told you nothing about which file inside it was worth your time.

You click through a few of the 46 out of pure curiosity. There are genuinely excellent skills in there — the kind with worked examples and stop-and-ask rules. And mixed in with them, files that are basically a title plus a paragraph of intent. The ranking can't tell them apart. It's not that the ranking is broken in a spectacular way; it's that it collapsed, and a collapsed ranking is worse than no ranking, because it looks authoritative while telling you nothing.

You close the tab and sit with it for a minute. The colleague's look from this afternoon comes back to you — "everyone uses it" is not the same as "it fits you". He was right, and the 46-way tie was the proof. I hadn't picked a skill at all that afternoon. I'd picked a repo. The distinction felt pedantic until the moment it produced a wall of 46 identical scores — and I realized I would have been equally unable to choose between them myself, because I was using exactly the same signal the broken ranking was using.

That moment changed how I pick skills. It also sent me down a rabbit hole into how that ranking is actually built — and what I found is that the ranking you see on any directory page is a design artifact, not a judgment from on high. It has a history. It broke twice, in instructive ways. Understanding those two breakages is the best skill-picking tool I've found, because it tells you exactly what a score can and cannot tell you.

Why stars can't answer the only question you have

Let's be fair to stars first. They're a real signal. A repo with thousands of stars has been looked at by thousands of people; it has survived contact with real users; it probably has maintainers who care. None of that is nothing.

But the question you're actually trying to answer when you pick a skill is not "is this repo good?". It's "is this file good — for my project, my stack, my habits?". And stars answer the first question, not the second. A star count is attached to a repository, and a repository can contain a hundred skills. The stars are a property of the collection; the decision is about the item. When the collection is good, the star signal actively misleads you about the items — because it makes all of them look equally good, including the ones that are a title plus a paragraph of intent.

That's exactly what the 46-way tie was. The repo wasn't the problem — it's a genuinely good repo, full of genuinely good work, and I want to be clear about that, because the failure was the ranking's, not the authors'. The problem was that the ranking couldn't tell the 46 apart, because it wasn't reading them.

Version 0: the metadata heuristic, and why it collapses

Here's how the first version of that ranking worked. It scored every skill on eight metadata signals — stars, whether the owner is a known-good organization, license presence, freshness, naming quality, structural soundness, and a few more. Cheap, deterministic, no inference cost. If you're building a directory of anything, this is the obvious place to start: it runs in milliseconds, it never hallucinates, and it mostly agrees with your gut.

It has one flaw that isn't visible until you look at the output distribution: none of the eight dimensions reads the skill's content. All eight look at the outside of the file — who made it, when, how it's packaged.

That's survivable when you're comparing skills across different repos. A skill from a famous organization with a fresh timestamp and a proper license will legitimately outscore a skill from a dead repo with no license. Fine. But within a repo, it's fatal — because for two skills in the same repository, the stars are identical, the owner is identical, the license situation is identical, the freshness is nearly identical. Of the eight dimensions, four or five are constant by construction. They're the same repo. Of course they are.

The measured result, on one large publisher, was stark:

3,432 skills. 31 distinct score values. 46 tied for first place. 808 tied at 75.

A user asks "which one should I install?" and the honest answer the system can produce is a list of 46 things tied for first, and a pile of 808 more tied just below them. The ranking had collapsed into a flat wall of numbers that all mean "fine, probably".

This is the point where most people give up and go back to picking by stars — which is the same collapse with extra steps. But the 46-way tie tells you something useful: the only thing that can separate two files with identical metadata is the content of the files. Stars can't read. Metadata can't read. The whole reason to bring a language model into a ranking is not because LLMs are trendy — it's because only reading the text can distinguish two files whose outsides are the same. That's the entire justification, and it's a strong one.

The part where I recognize myself

The 46-way tie was embarrassing for the ranking. It was also embarrassing for me, because I'd been doing the same thing by hand for years. My bookmarks folder is a graveyard of "best of" lists — every one of them sorted by the same eight metadata signals: who made it, how famous it is, how popular it looks. I'd never once sorted a list by actually reading the things on it. Popularity is a proxy for quality, and proxies are fine until they disagree with the thing they proxy for — which is exactly the moment a 46-way tie happens.

And there's a subtler version of the same trap I keep falling into: I don't just pick by stars, I pick by the shape of popularity. A wall of badges. A fancy logo. A well-designed README with screenshots. All of that is metadata about the outside of the project, and none of it tells me whether the file inside will change what my agent does. The 46-way tie just made the collapse visible for once — instead of hiding behind my own vibes.

The deeper lesson took longer to land. A proxy isn't a shortcut that's slightly worse than the real thing; it's a different thing entirely that happens to correlate. Stars correlate with quality the way reputation correlates with competence — enough to be useful, never enough to be decisive. The moment the proxy and the real thing disagree — the moment a famous repo contains 46 files of wildly different quality — the proxy doesn't just fail, it lies with confidence, because it has no idea there's anything it's not measuring. The 46-way tie wasn't an error in the ranking. It was the ranking being honest about what it knew: nothing about the contents.

So I was primed to believe the fix was simple: make the system read. Send the skill to a model, get a quality score, done. That's what Version 1 did. And Version 1 was worse than the heuristic it replaced.

Version 1: ask the model for a score. It gives you 85.

The obvious implementation: send the skill, ask for a 0–100 quality score, store it. The model reads the content — problem solved, right?

On a 200-skill sample, the results came back:

24 distinct values. 54 tied at 85. 38 tied at 75. Those two values ate 46% of the sample.

For comparison, the dumb metadata heuristic managed 37 distinct values on the same sample, with a maximum tie of 23. The LLM had produced a flatter distribution than the thing it was replacing. We had spent inference budget to make the ranking worse — not by a little, but measurably, on the exact metric that matters: can the ranking tell two skills apart?

The team's first instinct was the usual one: tune the prompt. "Be harsher." "Use the full range." "Grade like a tough teacher." Someone even suggested few-shot examples of what an 20 and a 95 look like. It all felt productive, and none of it would have worked, because the problem wasn't the instruction — it was the question shape. The cause is well known once you see it, and it's not mysterious. Ask a model for a single overall score and it snaps to the round, socially safe numbers. 85. 90. 75. It is not going to say 73 — 73 is a weird thing to say, and models hate saying weird things. They've been trained to be agreeable, and "this is an 85" is the most agreeable possible answer to "how good is this?". The result is that the model was discriminating less than the heuristic — every item came back "good, roughly 85", which is exactly as useful as the 46-way tie, with extra latency.

The distribution chart told the whole story at a glance, the way these things always do. One bar at 85, tall enough to blot out the axis label. A smaller bar at 75. A long tail of nothing. A useful scoring distribution has shape — it's lumpy, it has outliers, it has a long tail in both directions. Ours was a single spike with shoulders. And the worst part was how easy it was to miss: nobody looks at the distribution of a scoring system on day one. You run a sample, the scores look reasonable, every individual number seems fine, and you ship. The distribution is where the truth lives, and it's the last thing anyone checks.

This is the failure mode I'd bet you've hit if you've ever tried to use an LLM as a judge: everything comes back an 85. And the instinct when that happens is to tune the prompt — "be harsher", "use the full range", "grade like a tough teacher". You can burn days on that, and the model will politely agree and keep producing 85s, because the problem isn't the instruction, it's the question shape.

The fix: never ask for the total

The rewrite that finally worked changed one thing, and it wasn't the model, the prompt length, or the temperature. It changed what the model is asked to produce. Instead of one 0–100 score, the judge now scores four dimensions independently, 0–25 each, and the code sums them:

• specificity — is anything here falsifiable, or is it all "handle errors appropriately"?
• actionability — is there a next action, or just a value?
• completeness — does it survive the unhappy path?
• distinctiveness — could this be about anything?

Same model, same single call, same cost. But the model never sees a total and never emits one — the prompt explicitly forbids computing a total, and forbids letting one dimension drag the others.

Why this works: even if the model snaps to round numbers within each dimension — and it will, it always does — four semi-independent snaps combine into a spread. An 85 becomes 20 + 25 + 20 + 20, and the next skill's 85 becomes 15 + 25 + 15 + 25, and suddenly the ranking has hundreds of reachable values instead of two dozen. You get discrimination not by making the model more calibrated — you will never win that fight — but by never asking it the question it's bad at.

If you take one thing from this whole story: an LLM is much better at "how specific is this, 0–25" than at "how good is this, 0–100". Decompose, then add in code. The arithmetic is the part the model is actually good at, which is why you do it yourself.

The decomposition has a second benefit I didn't expect, and it's the one that matters for you as a reader: the four numbers tell you why. Two skills can have the same total and be completely different. One scores 20/25/20/20 — specific, actionable, complete, a bit generic. Another scores 15/25/15/25 — sharply distinct, but thinner on specifics and unhappy paths. Same total, different files, different risks. A single score can't tell you that; a breakdown can. When I finally understood that, I stopped reading scores and started reading profiles.

Version 2's bug: we were only reading the first 2,000 characters

The first working version had a second bug hiding inside it, and this one is the one that should scare anyone who's ever trusted a summary. The judge truncated skill bodies at 2,000 characters before sending them to the model. Truncation is the kind of engineering decision you make once and never revisit — it's a cost optimization, invisible in the code, and it feels completely reasonable at the time.

Then someone measured the corpus: the median body length is 6,003 characters.

Only 13.2% of skills were being read in full. 86.8% of judgments were made on an opening fragment — and openings are titles, badges, language-selection menus, and boilerplate.

Here's the one that made it concrete. There's a skill in the corpus called money-social. Its first 2,000 characters are the H1 — # Money Social — Social Media & Community Automation — plus a language picker. The actual content starts after that. And the actual content turns out to be a content-strategy handbook that automates nothing.

The judge, reading only the opening, wrote: "Automates social media content creation, scheduling..." Confidently. From the title. A real agent reading the whole file finds a document that doesn't automate anything. A real human, given the truncated judgment, would have installed it expecting automation and gotten a handbook. That's not a subtle error — that's the verdict being the opposite of the truth, produced with perfect confidence, at scale. And it's exactly the kind of error you can't see from inside the pipeline, because the pipeline only ever shows you the output, and the output reads fine.

The brutal punchline: the truncation wasn't even buying us much. The model we judge with has a million-token context. Raising the cap to 16,000 characters covers ~90% of skills in full, and the input cost to re-judge the entire corpus came to roughly $1.50.

We were producing systematically wrong verdicts to save a dollar fifty. Let me say that again, because it's the whole lesson: the cost of the wrong judgment was invisible, and the cost of the fix was trivial — and the wrong judgment still shipped, because nobody had looked at what the truncation was actually doing to the output. If you have a pipeline that summarizes or scores anything, go check your truncation limit right now. I'll wait.

Three implementation details that mattered more than expected

While I was in the guts of this, three smaller details kept showing up — the kind of thing that looks like bookkeeping and turns out to be load-bearing.

Idempotency keyed on content hash. Re-judging thousands of skills on every run is a cost you notice. If the SHA of the file hasn't changed, don't re-judge it. This sounds trivial until the day you realize your batch job has been re-billing the same content for a month.

Never stamp judged_at on failure. A single skill erroring shouldn't take down the batch — but if you record the timestamp anyway, the next run treats it as done and skips it forever. The failure becomes permanent and silent. One malformed file, and a skill silently never gets scored again. We hit this one the boring way: a skill with a weird encoding error, a timestamp written anyway, and two weeks later someone noticed it had never been re-scored. The rule that saves you: the timestamp is written only on success.

response_format: json_object guarantees valid JSON, not the right shape. It will happily hand you {} — valid, useless. Validate the shape separately, and treat a wrong shape as a failure, not a parse error to shrug at.

None of these is glamorous. All three of them are the difference between a scoring system that degrades gracefully and one that quietly lies.

What this still doesn't fix

Now the part most write-ups skip, and the part that matters most for how you read a score: the limits.

• The judge reads the skill. It does not run it. A skill that describes an excellent workflow and executes badly scores well. Nothing in this pipeline catches that — the only test for "does it actually work" is running it, which no directory-scale judge does.
• ~10% of skills are still truncated at 16,000 characters. Long ones are judged on a prefix — the same failure mode as before, just rarer. The 2,000-character bug is fixed; the category of bug is not.
• Four dimensions is a choice, not a discovery. They produce a usable spread. There is no evidence they're the right four dimensions — only that they're better than one. A different set might be better still, and the only way to find out is to measure, again.
• LLM judgments aren't stable across model versions. Change the model and the ranking shifts under you. Anyone telling you their AI scoring is objective hasn't re-run it after an upgrade. The ranking is a snapshot, not a truth.

And one more, which is the deepest one: the whole system still measures the text, not the behavior. The metadata version couldn't read the text. The LLM version reads the text but still can't run it. The thing that would actually tell you whether a skill is good — running it against a real task in a real repo — remains the one thing no directory can do for you. Which is exactly why the final decision stays with you, no matter how sophisticated the ranking gets.

How I pick skills now

Back to Tuesday. I still check stars first — habits are habits — but I decide last, and the order of operations changed.

Last weekend I needed a skill for extracting tables out of PDFs. Three candidates, all plausible, all with reasonable star counts. The old me would have picked the most popular and moved on. Instead I opened each one's score breakdown the way I'd read a nutrition label. The eight metadata dimensions told me about the outside of the file: who made it, how fresh it is, how it's packaged. The four content dimensions told me about the inside: whether it's specific, whether it has actual steps, whether it survives the unhappy path, whether it's about anything in particular. One of the three — the middle one, popularity-wise — had a distinctive profile: high distinctiveness, lower completeness. That told me exactly what I'd find inside: a sharp, opinionated skill that probably hadn't thought hard about what happens when the PDF is scanned. I installed it anyway, knowing the gap, and I was right on both counts: it was sharp, and the first scanned PDF made it ask me what to do. That was the skill working exactly as advertised.

Second, I skip the ranking's verdict and read the skill's own description and trigger. The whole point of the four-dimension design is that the score tells you why a skill is good or not — the dimensions are the reasons. If a skill scores high on completeness and low on distinctiveness, that tells me something concrete about what I'll find inside, before I open the file. And if it scores low on actionability, I don't even open the file — the one thing a skill must do is change what the agent does next, and a low actionability score says it doesn't.

Third — and this is the rule the 46-way tie taught me — when two candidates look equal on the outside, I stop comparing their outsides. I read their bodies. That's the only move that ever resolves the tie, whether the tie is 2 skills or 46.

What the ranking is really for

Here's the thing the whole rabbit hole eventually added up to. A skill directory has a real problem underneath all the tooling: the supply is not the bottleneck — the sorting is. There are tens of thousands of community skills. Nobody needs another list of them; anyone can scrape GitHub. What people need is to know which of the 46 things tied for first is worth installing — and the only way to answer that is to read them. That's what the ranking is for: not to pronounce judgment from on high, but to do the reading you can't do at scale, and hand you the reasons so you can do the deciding.

The number of skills in the index is not the point, and any directory that sells you on its count is selling you the wrong thing. The value is curation — the reading, the breakdown, the reasons. A score you can't decompose is a score you can't trust, and a directory that won't show you its work is asking you to take the 46-way tie on faith.

The Qumge directory is at qumge.com/en/skills, and it shows you both halves of the breakdown — the metadata dimensions and the content dimensions — before you install. When you find one you want:

npx skills add <owner/repo> 

One line, works across Claude Code, Cursor, Copilot, Windsurf and Cline.

And if you're building your own judge — for skills, for repos, for anything — take the two failure modes with you. Version 0 collapsed because it couldn't read content. Version 1 collapsed because it asked the model the one question it's bad at. Version 2 nearly shipped systematically wrong verdicts because of a truncation limit nobody had looked at. All three were invisible in the code and visible only in the output distribution — which is why the first thing you should ever do with a judge is look at how its scores are distributed, before you trust a single one of them.

The 46-way tie looked like a bug. It turned out to be the most useful thing the ranking ever told me: that stars and scores are proxies, and the only way to choose well is to read. The ranking reads at scale so you can read the last two pages yourself.