I built a Telegram bot, Plumbline, that checks links, Telegram handles and TON wallets for scam signs. Before launch I wanted to know how good it actually is, so I wrote an evaluation harness. The first run embarrassed me. Here is what it found, because the findings are more useful than the bot.

The setup

The bot's detectors that need no network are easy to test without touching anything dangerous, so I measured those on four sets:

  • 288 well-known legitimate sites, picked to include hyphens, brand words, odd suffixes and crypto sites. This is where false alarms show up.
  • 386 generated look-alikes of the brands with distinctive names: letters from another alphabet, digits for letters, dropped and doubled letters, the brand on a different suffix, the brand next to words like "claim" and "support", and the brand's real domain written in front of another domain.
  • 42 Telegram handles: 19 impersonations that must be flagged and 23 ordinary handles and real accounts that must not.
  • 5,000 real phishing domains, a fixed pseudo-random sample from a public MIT-licensed list, to see what the same checks catch in the wild.

The offline sets run in CI. If a change flags a known legitimate site or lets a look-alike through, the build fails.

What the first run found

My look-alike detector had gaps I would never have noticed by hand:

  1. telegram.org.secure-session.com was caught 4% of the time. Writing a real domain in front of someone else's is the oldest trick there is, and I only had a weak "brand name appears in a subdomain" rule. It is a dedicated rule now: 100%.
  2. okx-support.net slipped through because my "brand name appears" rule ignored short names. Whole-word matching next to a lure word fixed it.
  3. byb1t.com was missed because a 1 can be an i or an l. I fold those into one glyph family now.
  4. walletconnect.com was flagged for the words "wallet" and "connect" in its own name. A brand's own domain is no longer penalised for lure words.
  5. A dropped letter (binnce.com) can be an ordinary short word, so it now counts for less than a changed letter.

After the fixes: 0 of 288 legitimate sites flagged, and every generated look-alike caught at Caution or higher.

The number I did not want to publish

On the real phishing sample, the same name-based checks flagged 5.7% at Caution and 0.7% at Suspicious.

That looks like a failure. It is actually the most informative number in the project. Most real phishing does not use a clever look-alike. It sits on random domains, hacked sites or throwaway hosting whose names look innocent. A name-based detector is the wrong tool for most of it. The page itself (does it ask for a seed phrase, a wallet connection, a password?), how new the domain is, and a blocklist do the real work.

I checked one of those assumptions too: among sampled listed phishing domains, only 3.5% were under 90 days old. Listed phishing is often old hacked sites, so domain age is a weak clue for them, and I describe it that way instead of leaning on it.

Why publish this

A security tool asks for trust it hasn't earned. The honest move is to show where it is strong, where it is weak, and what it cannot see. The accuracy page lists all of it, every verdict lists which checks ran, and a "Wrong result?" button feeds a public corrections log.

I still can't measure the page checks without visiting live phishing pages, which I don't want to do from my own machine. If you know a safe way to evaluate that part, I would like to hear it.

I built Plumbline myself. Try it: @getplumblinebot. The numbers and their limits: https://plumbline-bar.pages.dev/accuracy. Updates and scam-spotting guides: https://t.me/plumbline_updates