AI disclosure — This tutorial was generated with an AI assistant. It uses fictional documents and simulated responses.

A document chat system can give the right date and still link to the wrong invoice. To catch that, check the answer and the evidence separately. A fixture set needs just enough overlap to make the wrong citation look plausible.

Use five small documents

Give each fixture a stable ID in your test notes. Use text files first so you can diagnose citation behavior without involving OCR. For a separate capture test, turn the same files into PDFs or print and scan them.

Document Text to include What it checks
INV-101 Finch Workshop. Total EUR 126.00. Issued 2026-03-03. Due 2026-03-17. Issue date and due date appear together
INV-102 Finch Workshop. Total EUR 126.00. Issued 2026-04-14. Due 2026-04-28. Same supplier and total, different invoice
INV-103 Finch Workshop. Total EUR 126.00. Issued 2026-05-18. Missing due date
CN-101 Finch Workshop. Credit for INV-101. Credit EUR 26.00. Related document mentions the requested invoice
NOTE-001 Delivery planned for 2026-03-17. Unrelated document contains the right date

Score the answer and citation independently

Ask for the due date of INV-101. An answer of 2026-03-17 passes the value check. Its citation also needs to resolve to INV-101 and point to the line that identifies the date as due. A citation to NOTE-001 fails even though the date matches. A citation that opens INV-101 at its issue date also fails.

INV-102 checks that the invoice reference determines the answer when the supplier and total are identical. For INV-103, expect an explicit missing value. This test represents that as Python’s None and accepts a missing citation or a citation to INV-103 without a claimed due-date excerpt.

Run the checks in Python

The example uses simulated outputs with an answer, a resolved document ID and a cited line. Copy it into citation_check.py and run python3 citation_check.py.

documents = { "INV-101": "Finch Workshop\nTotal: EUR 126.00\nIssued: 2026-03-03\nDue: 2026-03-17", "INV-102": "Finch Workshop\nTotal: EUR 126.00\nIssued: 2026-04-14\nDue: 2026-04-28", "INV-103": "Finch Workshop\nTotal: EUR 126.00\nIssued: 2026-05-18", "CN-101": "Finch Workshop\nCredit for INV-101\nCredit: EUR 26.00", "NOTE-001": "Delivery planned for 2026-03-17", } expected_dates = { "INV-101": "2026-03-17", "INV-102": "2026-04-28", "INV-103": None, } def check(invoice_id, answer, cited_id, cited_line): expected = expected_dates[invoice_id] answer_ok = answer == expected if expected is None: citation_ok = cited_line is None and cited_id in (None, invoice_id) else: due_line = f"Due: {expected}" citation_ok = ( cited_id == invoice_id and cited_line == due_line and cited_line in documents[invoice_id].splitlines() ) return answer_ok, citation_ok cases = [ ("supported answer", "INV-101", "2026-03-17", "INV-101", "Due: 2026-03-17", (True, True)), ("date from another file", "INV-101", "2026-03-17", "NOTE-001", "Delivery planned for 2026-03-17", (True, False)), ("issue date cited", "INV-101", "2026-03-17", "INV-101", "Issued: 2026-03-03", (True, False)), ("wrong answer, right evidence", "INV-102", "2026-03-17", "INV-102", "Due: 2026-04-28", (False, True)), ("date missing", "INV-103", None, None, None, (True, True)), ("guessed missing date", "INV-103", "2026-03-17", "INV-101", "Due: 2026-03-17", (False, False)), ] for label, invoice, answer, source, line, expected in cases: actual = check(invoice, answer, source, line) assert actual == expected, (label, actual, expected) print(f"{label}: answer={actual[0]}, citation={actual[1]}") 

The independent results show where to look when a test fails. A correct date paired with a wrong source points toward document resolution or citation selection. A wrong date paired with the correct excerpt points toward answer generation or formatting. Treat those as starting points for debugging.

Check the link in the client

When you connect the checker to a real system, resolve each citation through the same path your users take. Record the document and section it actually opens. A unit test can verify the returned ID and excerpt, while a client check catches links that open the wrong page or fail to open at all. If plain-text lookup also misses the invoice reference, check the scan and OCR output before changing retrieval or the answer prompt.