TL;DR — We released ja-nli-v1, a small Japanese natural language inference model: given a premise and a hypothesis, it returns probabilities for entailment / neutral / contradiction. On a sealed test of 5,000 pairs it scores accuracy and Macro F1 of 0.9612; on a 600-pair human-verified set, 0.9567, with entailment↔contradiction confusions at 3 (0.5%). Two multilingual NLI models used zero-shot score 0.79–0.81 on the same sealed set. A premise-shuffle diagnostic confirms the model actually reads the premise: swap in an unrelated one and accuracy falls to ~0.38. Trained without any existing NLI dataset — licensing, not preference. Apache 2.0, ONNX included, ~109 ms per pair on CPU.
Model page (try the demo from there): https://huggingface.co/japan-opendata/ja-nli-v1
Browser demo: https://huggingface.co/spaces/japan-opendata/ja-nli-demo
- A verification part, meant to sit next to generation. Does the RAG answer contradict its source passage? Does the claim match the search result it cites? Those are premise–hypothesis questions, and in Japanese they usually get answered by a large multilingual model or not at all. This is the small, local, CPU-priced version.
- The sanity checks are the feature. Shuffle the premise for an unrelated one and accuracy drops to ~0.38, with most outputs collapsing to neutral — the model is lost without the premise, which is what you want. A hypothesis-only probe (can it guess the label from the hypothesis alone?) was driven down to ~0.63 on the human-verified set by diversifying the training data, so the shortcut is weak. An NLI model that doesn't read its premise is a label guesser; we measured that ours does.
- No existing NLI data was trained on. JSNLI, JNLI, JSICK, JaNLI, XNLI, ANLI — all excluded on license terms. The training pairs are synthetic, built from seed corpora limited to public-domain-licensed government documents (PDL 1.0) and MIT / Apache-2.0 Japanese OSS documentation, generated by an open-weight LLM and blind-judged by a different model. The zero-shot baselines above were measured for reference only.
Use it
from transformers import AutoModelForSequenceClassification, AutoTokenizer repo = "japan-opendata/ja-nli-v1" tok = AutoTokenizer.from_pretrained(repo) model = AutoModelForSequenceClassification.from_pretrained(repo) model.eval() # Give it a premise and a hypothesis; it returns probabilities for # entailment / neutral / contradiction (mapping in inference_config.json). # Max length 1,024 tokens; truncation applies to the premise side only. # Full reference implementation (PyTorch and ONNX): examples/infer_nli.py The ONNX build (fp32, opset 17) matched PyTorch predictions exactly on 200 development and 200 sealed items. Speed: ~109 ms per pair on a Core i7-11700F, measured on inputs that include longer premises. No INT8 build is shipped.
Measured results
Sealed test (5,000 pairs, separated from training, single final run):
- Accuracy 0.9612, Macro F1 0.9612
- Zero-shot comparison, same conditions: two multilingual mDeBERTa-family NLI models at 0.7920 and 0.8094
Human-verified evaluation (600 pairs):
- Accuracy 0.9567; entailment↔contradiction confusions: 3 (0.5%)
- The same zero-shot models: 0.8433 each
Diagnostics:
- Premise shuffled to an unrelated text: accuracy falls to ~0.38, most predictions collapse to neutral
- Hypothesis-only probe: ~0.63 on the human-verified set (kept low by design)
Training data
Seed corpora: PDL 1.0 public documents, and Japanese OSS documentation with confirmed MIT / Apache-2.0 licenses — nothing else. Pairs were generated by an open-weight LLM and verified by blind judgment from a separate model. A random 200-pair audit found label validity 95.0% and Japanese naturalness 94.5%.
What didn't work / Limitations
- The evaluation ground is, again, ours. Both test sets are synthetic-derived, built and labeled under our pipeline. The zero-shot models were graded on Japanese pairs written to our distribution; on the standard Japanese NLI benchmarks they were built for, the comparison would look different. We did not train on those benchmarks, so we also cannot report on them as a neutral party — treat 0.9612 as performance on verified synthetic pairs, with real-world text expected to score lower.
- The hypothesis-only probe is weakened, not dead. ~0.63 is well below the full model's 0.9567, but above the 0.33 chance level — some hypothesis-side signal remains, as it does in essentially every NLI dataset ever built, including the famous ones. The premise-shuffle collapse (~0.38) is the stronger evidence that the premise drives the decision.
- ~109 ms is the slowest in our series so far. Premise-plus-hypothesis pairs with long premises cost more than single short texts. Fine for checking generated answers; budget accordingly for bulk scoring.
- Truncation cuts the premise, never the hypothesis. Past 1,024 tokens, the end of a long source passage silently stops participating. For long-document checks, chunk the premise yourself rather than letting the tokenizer decide what matters.
This is the eleventh in our line of small, single-purpose Japanese models. NLI is the one where "the model got a high score" is least trustworthy on its own — so this release leads with the diagnostics that try to catch the model cheating, and publishes what they found.
If you check generated Japanese text against its sources today: dedicated NLI model, a prompt to a big LLM, or nothing yet — and which failure worries you most, a missed contradiction or a false alarm?
- Model on Hugging Face: https://huggingface.co/japan-opendata/ja-nli-v1
- Browser demo: https://huggingface.co/spaces/japan-opendata/ja-nli-demo
- Code (the model series): https://github.com/japan-opendata-lab/ja-small-models
- Model page with usage notes: https://jpopendata.jp/models/ja-nli-v1/
Get the next release notes and our Kaggle experiment logs by email: https://jpopendata.jp/ (newsletter signup in the footer).
