A single percentage can make two annotation files look more complete than they are. Suppose you have two JSON files that both label the same texts, and a short script reports 100% agreement. The bundled partial-agreement example constructs exactly this case: only one of twelve records is labelled by both sides, and that one record matches, so conditional agreement is 100% while both-labelled coverage is 1/12. The percentage says nothing about the state of the other eleven records.
Annotation, checkpoints, and agreement
Annotation means assigning a text one label from a small set. A checkpoint is the annotation tool's JSON snapshot: an ordered record list plus a state for each record. Given two checkpoints that cover the same records, a reviewer first wants the places where they disagree, then the places where someone has not finished.
Conditional agreement counts only records both sides labelled: same labels divided by both-labelled records. The default example lays out the boundaries of that number. Four records are labelled by both sides, two of those match, agreement is 50%, and coverage is about 33.3%. The six comparison states contain two records each: same_label, different_label, both_abstained, one_abstained, both_unlabelled, and one_unlabelled.
When neither side labelled anything, the metric has no denominator. The no-labels example has zero both-labelled records; JSON stores label_agreement as null, and the page displays N/A, not 0%.
My tool first verifies that both checkpoints carry the same record list, taxonomy, and order. A missing record, changed text, different group, or reordered taxonomy rejects the pair. The tool does not compare only the overlapping records, join after normalizing text, or silently remap labels. An unknown group stays null instead of gaining an invented group.
This screenshot is from the actual running Hugging Face page with bundled synthetic data. It shows 12 records, 4 labelled by both roles, 50% conditional agreement, and 2/4 same labels. Agreement is not accuracy, and A/B are not verified independent people. The pinned method notes and the live Space provide the same details.
same_label is not a claim about whole files
A matching label means one record has the same label state in both annotations. The partial-agreement pair uses two checkpoints with the same content digest, so identical_checkpoint is true, yet both-labelled coverage is still one record, item-01. The default pair uses different A and B files; item-01 and item-07 have matching labels and count as same_label anyway. A record-level match cannot grow into a claim that the two complete files are the same, and it cannot represent the whole list as undisputed.
Each final decision starts unresolved, including records where both labels match. The page does not merge automatically and has no bulk accept. A reviewer can choose any existing label, abstain, or reopen a result as unresolved. A and B are file roles. They may come from the same person or the same checkpoint, and identical inputs receive an explicit warning. The original two annotation sets remain inside the exported review, while manual decisions live in a separate layer.
Each real change increments the revision; saving the identical decision again does not. The limit is 1e9. Undo restores the last 20 real changes without erasing provenance or rewriting the source text. Notes are limited to 2,000 UTF-16 code units and must contain valid Unicode scalar values. Emoji and other astral characters use two code units each. An unresolved decision must have an empty note.
What saving and restoring cost
The page keeps current work in browser memory only. There is no autosave, account, or server copy. Typing in the form is not recording. For that form edit to enter a snapshot, press Record decision first. Preparing a download captures the decisions recorded at that revision, and the page cannot confirm where the browser saved the file. Reopen as unresolved and Undo last change also update decisions and the revision when a real change occurs; they act on recorded state and do not carry unrecorded form text with them.
Restoring a review reads the complete file, recomputes digests, shows a preview, and then waits for explicit confirmation. A malformed file is rejected while active work stays in place. Swapping A and B invalidates the old binding. A saved review is self-contained and can restore the pair it originally carried after preview.
The snapshot contains both complete original checkpoints, all source text, both annotation sets, the comparison matrix, notes, and every manual decision, so I have to check its contents before sharing. Browser-local processing describes the computation path; it is not a privacy or secrecy guarantee. SHA-256 checks consistency and can be recalculated after content changes, so it does not authenticate the source. Duplicate JSON keys follow JSON.parse last-key semantics rather than strict duplicate-key rejection. BOM, invalid UTF-8, and invalid Unicode scalars are rejected.
Reproducing on the complete package
Start by copying the full public package into a fresh directory. cli.mjs loads review.mjs and vendor/queue.mjs, so two demo JSON files alone will not run. Once the complete package is in place, execute:
node cli.mjs demo-a.json demo-b.json NEW-review.json The terminal prints the comparison object: total is 12, each of the six buckets is 2, both_labelled is 4, label_agreement is 0.5, both_labelled_coverage is 0.3333333333333333, and identical_checkpoint is false. Inside NEW-review.json, all 12 decisions are unresolved, including the demo items with matching labels, item-01 and item-07. Both source checkpoints are kept in full. The command creates its output exclusively, so it fails when NEW-review.json already exists instead of overwriting it.
Direct links: live Space, direct app, and pinned source commit aa1c89b.
Written by Renchao Wu. English original.
