TL;DR — We released ja-phish-guard-v1, a Japanese text classifier that scores phishing, fraud, and spam risk in emails and messages. On its sealed test it detected 1,640 of 1,640 attacks with zero false positives — and that perfect score is exactly why this article leads with the caveats: the test is synthetic, and real mail will score lower. It is an auxiliary risk scorer, not a replacement for DMARC, URL inspection, or sender authentication. Apache 2.0, ONNX included, ~28 ms per message on CPU.
- What it does: the model classifies a message into five classes (phishing / fraud / spam / benign_hard / benign) and returns a risk score — the summed probability of the three attack classes. Text in, score out, on your own CPU.
- The sealed-test numbers, stated precisely: 2,990 held-out items never used for training or threshold selection. Detection rate 1.0 (1,640/1,640), false-positive rate 0.0 at a 0.5 risk threshold (0.0022 at a lower threshold fixed on development data), per-class accuracy 1.0 across all five classes, risk-score AUC 1.0. Attack families deliberately withheld from training — SMS auth-code lures, rent scams — were also detected at 1.0.
- Why those numbers are an upper bound, not a promise: the training data and the sealed test are both synthetic, generated by the same process. The "unseen families" are unseen templates, not unseen reality. And on development data, the fraud class already shows the crack: precision 0.7483, well below the other classes. Real inboxes will be worse than this test. Plan accordingly.
Why a text scorer at all
Phishing defense is mostly not a text problem: sender authentication, URL inspection, and attachment scanning catch more, more reliably. But there is a residual layer those checks do not cover — the message that passes authentication, links somewhere new, and relies entirely on its wording to panic the reader into acting. A text scorer is a signal for that layer: one more input to a risk decision, cheap enough to run on everything.
That is the positioning, in the model's own card and here: auxiliary risk scorer. It reads only the text. It does not replace DMARC, URL inspection, or attachment checks, and it should never be the only gate.
The model
- ja-phish-guard-v1 — 5-class text classification (phishing / fraud / spam / benign_hard / benign) with a summed risk score
- Base model: tohoku-nlp/bert-base-japanese-v3
- Apache 2.0, ONNX (fp32) included. Speed: ONNX on a Core i7-11700F averaged 27.6 ms per message
- All training data is synthetic. No real emails, no real personal information, and no real brand names — messages were built from role names and fictional brands, organized by attack family. Some benign text comes from the Japanese subset of FineWeb-2 (ODC-By 1.0)
- Generalization was tested structurally: some attack families were kept out of training entirely and appear only in the development and sealed sets
Use it
from transformers import AutoModelForSequenceClassification, AutoTokenizer import torch repo = "japan-opendata/ja-phish-guard-v1" tok = AutoTokenizer.from_pretrained(repo) model = AutoModelForSequenceClassification.from_pretrained(repo) model.eval() text = "アカウントが停止されます。24時間以内に本人確認を完了してください" enc = tok(text, return_tensors="pt", truncation=True, max_length=512) with torch.no_grad(): probs = model(**enc).logits.softmax(-1)[0] id2label = model.config.id2label scores = {id2label[i]: round(float(p), 4) for i, p in enumerate(probs)} risk = sum(float(probs[i]) for i, lab in id2label.items() if lab in ("phishing", "fraud", "spam")) print(scores) print("risk_score:", round(risk, 4)) On the sealed test, a 0.5 risk threshold gave a 0.0 false-positive rate at 1.0 detection — read the limitations before importing that threshold into production.
Measured results (sealed test, 2,990 items)
- Detection rate (3 attack classes): 1.0 — 1,640 / 1,640
- False-positive rate: 0.0 at risk threshold 0.5; 0.0022 at the lower threshold fixed on development data under a 1% false-positive constraint
- Per-class accuracy: 1.0 for all five classes
- Withheld attack families (SMS auth-code type, rent-scam type): detection 1.0
- Risk-score AUC: 1.0
What didn't work / Limitations
- The test is synthetic, and so is the training data. Perfect scores here mean the model learned the generator's families completely — including the withheld ones, which are different templates from the same generator, not genuinely new tricks from the wild. On real mail, expect these numbers to come down. By how much is unmeasured, and we will not guess.
- Fraud wording is the weak class. On development data, fraud precision was 0.7483 against near-perfect scores elsewhere. Fraud phrasing overlaps with legitimate urgency — payment reminders, account notices — and the model feels it.
- Real brand names are unverified territory. Training used fictional brands only, so messages impersonating real banks and carriers — the most common real phishing — were never in the training distribution. Performance there is unknown, not assumed.
- Text only. No sender authentication, no URL resolution, no attachment analysis. A message can score low here and still be dangerous for reasons outside the text.
This is the fifth in our line of small, single-purpose Japanese models — PII detection, prompt-injection screening, punctuation restoration, document classification, and now message risk scoring. Same rules each time: free, ONNX included, measured on a sealed or public test set, and the caveats printed where the headline numbers are.
If you run mail or messaging infrastructure: where would a text risk score actually change a decision in your pipeline — quarantine, banner warnings, or just logging?
- Model on Hugging Face: https://huggingface.co/japan-opendata/ja-phish-guard-v1
- Code (the model series): https://github.com/japan-opendata-lab/ja-small-models
- Model page with usage notes: https://jpopendata.jp/models/ja-phish-guard-v1/
Get the next release notes and our Kaggle experiment logs by email: https://jpopendata.jp/ (newsletter signup in the footer).

![Cargo UAV Autonomy and Flight AI — Ch.06: Route Planning and Airspace Integration [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F54s7rrr8fw0ipjpci49u.png)
![Cargo UAV Autonomy and Flight AI — Ch.05: Perception and Sensor Fusion for UAV [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faemm4fj7d911fwx0irut.png)
![Cargo UAV Autonomy and Flight AI — Ch.04: Autonomous Navigation for UAV [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F98idhn116ojz4pviet9j.png)
![Cargo UAV Autonomy and Flight AI — Ch.03: Flight Control Software [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fminle8x2lymdhmeo5fbt.png)