Short answer: hash and version sources at ingest, group near-duplicate passages at retrieval, and require a current source ID plus page citation for every appointment claim.
For a healthcare appointment assistant, deduplicate sources before the language model sees them, and keep the original document IDs beside every passage. That gives the retrieval architecture one authoritative evidence trail instead of five copies of the same clinic policy competing in the prompt. The decision is about grounding and citation, not about making the prompt smaller.
The failure is easy to miss. A scheduling PDF gets exported again after a holiday-hours edit; the index now contains both files. A search for “walk-in” returns the old answer at rank two and the current answer at rank six. The assistant can cite a real page and still tell a patient the wrong thing.
One bad appointment suggestion is enough.
How should a healthcare appointment assistant deduplicate retrieval sources?
Treat deduplication as a two-stage operation: identify document versions at ingest, then remove near-duplicate passages at retrieval. Exact hashes catch repeated uploads. A similarity check catches a PDF with a new filename, a different metadata header, or one changed footer. Keep both the canonical source ID and the chunk ID in the result; collapsing the citation to a bare text string makes later review painful.
The ingest record needs a stable identity, not a filename. Normalize Unicode, line endings, and whitespace before hashing. Store the content hash, source URI, observed timestamp, and a status such as active or superseded. In a healthcare system, an explicit supersession relationship is more useful than silently deleting an older file: an auditor can see what was replaced, while retrieval can exclude it by policy. Imagine a clinic manager uploading hours-final.pdf on Monday and hours-final-2.pdf on Tuesday after changing the Saturday close time. If the pipeline keys on filenames, both become first-class evidence; if it keys on normalized content and an approved source ID, the second file can supersede the first while preserving the lineage. The reviewer can then answer three separate questions: which bytes were indexed, who marked the file current, and which page supported the answer shown to the patient. Those are different questions, so one opaque “document confidence” field is not enough.
At query time, first retrieve a generous candidate set, then group candidates by canonical source and normalized passage fingerprint. Select the best-scoring passage from each group, with a small allowance for genuinely different sections of the same source. This prevents one long policy from crowding every other clinic's hours out of the context window.
The RAG paper describes the general retrieve-then-generate pattern, but it does not define a healthcare citation policy. That policy belongs in your application contract. For example, require a source ID and page number for every scheduling claim, and return “I don't have a current source” when the evidence is missing or contradictory.
The safe implementation boundary
The deduper should be deterministic and boring. Here is a compact Go implementation for exact fingerprints and canonical-source selection. It does not decide whether a document is medically authoritative; that is a governance decision upstream.
package main import ( "crypto/sha256" "encoding/hex" "regexp" "sort" "strings" ) type Passage struct { CanonicalID string ChunkID string Text string Score float64 Page int } var spaces = regexp.MustCompile(`\\s+`) func fingerprint(text string) string { normalized := strings.ToLower(strings.TrimSpace(text)) normalized = spaces.ReplaceAllString(normalized, " ") sum := sha256.Sum256([]byte(normalized)) return hex.EncodeToString(sum[:]) } // Deduplicate keeps the highest scoring passage for each source and fingerprint. func Deduplicate(passages []Passage, maxPerSource int) []Passage { best := make(map[string]Passage) for _, p := range passages { key := p.CanonicalID + ":" + fingerprint(p.Text) if old, ok := best[key]; !ok || p.Score > old.Score { best[key] = p } } bySource := make(map[string][]Passage) for _, p := range best { bySource[p.CanonicalID] = append(bySource[p.CanonicalID], p) } result := make([]Passage, 0, len(best)) for _, group := range bySource { sort.Slice(group, func(i, j int) bool { return group[i].Score > group[j].Score }) if len(group) > maxPerSource { group = group[:maxPerSource] } result = append(result, group...) } sort.Slice(result, func(i, j int) bool { return result[i].Score > result[j].Score }) return result } This code only removes exact normalized duplicates. For near duplicates, add a separate similarity stage with a documented threshold and test set. Do not hide that threshold in a vendor setting: it changes which page the assistant can cite. I am not sure one global threshold will fit every clinic; appointment tables and prose policies have very different token patterns, so your mileage may vary.
Verification before the first patient query
Build a fixture with three copies of one policy, one superseded version, and two unrelated clinic schedules. Assert that the current canonical ID survives, that a superseded ID is excluded, and that every returned passage still has a page number. Then test contradictions: give the retriever two active sources with different closing times and require a refusal or an escalation flag rather than a blended answer.
Watch operational signals after deployment. Track the duplicate ratio (raw candidates minus retained candidates), citation coverage, superseded-source hits, and disagreement rate between retrieved sources. Log hashes and IDs, not patient text. A spike in superseded-source hits usually means ingest metadata or filtering changed; it is a page-worthy signal even when model latency is normal.
Roll out in shadow mode. Generate answers without displaying them, compare citations with a reviewer-approved set, and keep the previous retrieval index available for rollback. If citation coverage falls or contradiction flags rise, switch the read path back to the prior index while you inspect the new normalization or threshold. The rollback unit should be an index version, not an emergency code edit.
Trade-offs and boundaries
Source-level grouping improves citation diversity but can hide a legitimate exception recorded in a second section of the same policy. A per-source cap of two or three passages is a starting point, not a law. Larger candidate sets improve recall and increase the chance that stale text reaches the reranker. More aggressive similarity thresholds reduce repetition and risk deleting a small but important wording change.
This architecture is not suitable when your records have no stable provenance or page coordinates; fix document identity first, or use a human-reviewed search workflow. It is also a poor fit for real-time availability data that changes every few seconds. Keep a transactional scheduling API as the authority for open slots, and use retrieval for explanatory policy and citation. Stick with a simpler keyword index when the corpus is tiny and manually curated; adding semantic deduplication there can create more review work than it removes.
The practical rule is short: one claim, one current source, one traceable citation. Everything else is a candidate until the evidence pipeline proves otherwise.
References
- Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”: https://arxiv.org/abs/2005.11401
- HL7 FHIR R4, Appointment resource: https://hl7.org/fhir/R4/appointment.html
- RFC 9562, Universally Unique IDentifiers (UUIDs): https://www.rfc-editor.org/rfc/rfc9562
- NIST, AI Risk Management Framework 1.0: https://www.nist.gov/itl/ai-risk-management-framework