Last week we added a new exam to Examenzaal: Wet- en regelgeving in het werkveld, one of the five exam parts for the Dutch security guard diploma (Beveiliger 2). The exam body publishes the format (40 questions, 40 minutes, pass at 27 points) and a list of requirements. It offers no practice exams (only a few model items that show the exam software), and we don't use any of its questions.

So the questions are written by one model and checked by two others. This post is about the numbers from that pipeline, because they show something I didn't expect: the two checkers almost never disagreed about whether an answer was correct. They disagreed about whether the answer could be guessed from its wording.

The pipeline, in one paragraph

Per slot (40 slots over 42 requirements; which requirement each slot tests is our own split), a drafting model writes questions from the saved sources only (mostly legal texts: Wpbr, Rpbr, Arbowet, AVG, Sv, Sr and so on). Step B is a blind check: a second model answers the question without seeing the key, then rates it on six boolean criteria. Step D is an independent verification by a third model, with the same criteria. A question goes live as ai_verified only if both pass, and the verifier is a different model from the checker.

The counts

For this exam, from the JSON files in the repo:

  • 322 drafts: 240 for the practice exams (forms), 82 for free topic quizzes and the quick test (free_quiz).
  • Step B, 322 results: 321 pass, 1 fail. The one fail was flagged on exactlyOneDefensible and unambiguous.
  • Step D, 321 results: 303 pass, 18 fail. All 18 failures are on one criterion: levelLikeRealExam. Not one verification marked the key as wrong or the explanation as incorrect.

That last line is the interesting one. The blind checker (Sonnet 5.5 in these files) marked levelLikeRealExam true for all 322. The verifier (Fable 5.1) failed 18 on it, and its notes give the same reason every time: an answer cue. In 17 cases the key was the only option phrased differently from two wrong answers built alike; in one it was the only option that shared a word with the stem. Both agreed the answers were right.

If we had run only the blind check, those 18 would have passed. If the criteria were bundled into one "is this good?" boolean, we'd never have seen that the disagreement was about guessable answers and not about correctness.

Near-twins decide the exam forms, not the bank

A bank of verified questions isn't an exam. A practice exam needs one question per slot, no two questions that give each other away, and the same exam every time you open "Oefenexamen 3".

For this exam the practice exams are a fixed assignment in content/forms/nl-beveiliger-wrw.json, written by a script:

npx tsx scripts/beveiliger-wrw-forms.mts --write-fixed

Four forms of 40 questions each, with no question in more than one form. Every slot needs at least four verified forms questions to fill them.

The rule that keeps near-twins apart comes from the checkers' own findings. Both steps leave free-text remarks, and some of them say things like "not together in one form". We turned those into 14 overlap families in content/exams/nl-beveiliger-wrw.forms.ts. A family has an id, the rule that links the questions, the members, and where the remark came from. Example: several questions rest on the rule that a person arrested in the act must be handed over to an investigating officer without delay. One question's stem states what another one asks. The family puts them in different forms.

I wrote about the assembler itself in an earlier post. The new part is the input: the families are data derived from machine-written review notes, and each member that no checker named carries a written reason.

Say what you guessed

The exam body doesn't publish which share of questions are multiple choice, multiple response, matrix or matching. We had to choose. The exam definition says so, in the text a candidate sees:

pointsNote: "Het SVPB-examen heeft 40 vragen van elk 1 punt; met 27 punten (67%) is het gehaald. Hoe SVPB een vraag met meer goede antwoorden telt die maar voor een deel goed is beantwoord, staat niet in de regeling. Wij rekenen zo'n vraag alleen goed als het antwoord helemaal klopt. Welke vraagvorm bij welk onderwerp hoort, is onze keuze.",

In English: the exam body doesn't say how it scores a partly right multiple-response question. We score it as wrong unless it's fully right. Which question type goes with which topic is our choice.

Behind a flag until it was ready

The whole exam sits behind one environment flag:

export function beveiligerLaunched(): boolean { return process.env.BEVEILIGER_LAUNCHED === "1"; }

Off, the exam is in no registry. The landing page, the catalogue card, the sitemap entry and the checkout don't know it exists, and its route answers 404. A test checks that with every other flag on. The plan was written on 6 Nov; the verification files carry timestamps from the evening of 6 Nov, and the pages were live early on 7 Nov.

What I'd take from this

  • Keep criteria separate. One boolean hides what the checkers disagree about.
  • Two different models as checkers found different things: the second failed 18 questions the first had passed.
  • Mine your reviewers' free text. The near-twin families came from remarks that are easy to ignore.
  • Write down what you estimated, in the product, not just in the code.

The finished exam is here (in Dutch), and this page describes how we check questions.

I wrote this post with help from an AI assistant. The numbers come from the files in our own repository.