This article presents a proposed architecture based on design reviews and hypothetical failure scenarios. It is not a production implementation, clinical validation, or performance benchmark. The examples use fictional patient data.
Index
- The Goal
- The Napkin Architecture
- Trap #1: Temperature 0 Was My Hallucination Strategy
- Trap #2: "No Allergy Found" Became "No Allergies"
- Trap #3: The Vector Filter Was the Only Lock
- Trap #4: "No History of Diabetes" Was Retrieved as Diabetes
- Trap #5: The Index Remembered a Note the Hospital Deleted
- Trap #6: Keyword Matching Approved a Wrong Answer
- Trap #7: There Were Two Kamal Hossains
- Trap #8: I Designed for a Hospital That Doesn't Exist
- Trap #9: The Perfect Answer Took 25 Seconds
- The Final Architecture
- How I'd Prove It Works
- Build Order
- Lessons Learned
- Final Thoughts
1. The Goal
I wanted an internal assistant for doctors, nurses and pharmacists.
Ask a question about a patient, get an answer from the hospital's own records, with a source for every sentence. Ask about a hospital guideline, same thing.
One hard rule: if the records don't say it, the assistant doesn't say it.
That rule sounds simple. It ended up shaping every box in the diagram.
A note before we start. This is a design story, not a production post-mortem. The traps below came out of reviewing the design, walking real questions through it on paper, and arguing with myself. The patient in the examples is made up. Nothing here is a benchmark.
2. The Napkin Architecture
The first version looked like every RAG tutorial:
Question ↓ Embed ↓ Vector DB (filter: patient_id) ↓ Top-k chunks ↓ LLM (temperature 0, "only use the context") ↓ Answer Plus a post-check: if the answer shares enough keywords with the retrieved chunks, show it.
It looked done.
Then I walked one question through it: "Does Kamal Hossain have a penicillin allergy?"
Kamal's chart has no allergy entry at all. Retrieval returns a few vaguely related chunks. The model, being helpful, writes something like "No allergies are documented for this patient." Keywords match. The check passes.
A doctor reads that as "no allergies." Those are very different sentences.
That one question broke the design in three places. So I kept walking questions through it.
3. Trap #1: Temperature 0 Was My Hallucination Strategy
The Cause
Temperature 0 makes the output repeatable. It doesn't make it true. A model that confidently fills a gap will fill the same gap the same way every time.
"Only use the context" in a system prompt is a request, not a guarantee.
The Fix
I stopped treating the LLM as the thing that knows answers. The hospital's EHR is the source of truth. The LLM's only job is to explain evidence that something else already retrieved and checked.
That led to the biggest change in the whole design: exact facts don't go through the LLM at all.
"What was the last HbA1c?" is a database question. The router sends it to the EHR, fills a template, and returns it:
Latest HbA1c: 8.2 % (final, 12 Mar 2026) source: Observation/ob-5521 No model, no paraphrase, nothing to hallucinate. The LLM only gets involved for summaries and narrative questions, where language actually helps.
4. Trap #2: "No Allergy Found" Became "No Allergies"
The Cause
My data model had two states: data, or no data. Medicine has at least four.
Penicillin allergy recorded → documented positive "No known allergies" recorded → documented negative (someone checked) Nothing recorded → unknown Allergy service timed out → unavailable (we don't even know) The napkin design collapsed the last three into "nothing came back." The fourth one is the scary one. A search index being down looks exactly like an empty chart if you're not careful.
The Fix
The retrieval layer returns an explicit state, not just results. The backend, not the model, decides what to say for each state.
For "unknown," the reply is a fixed template:
No allergy information was found in the records you can access. This does not mean the patient has no allergies. Please confirm with the patient or the care team.
For "unavailable," the reply says the system couldn't check. It never says "nothing found."
The model is never allowed to turn absence into a negative finding. That's enforced in code, not in the prompt.
5. Trap #3: The Vector Filter Was the Only Lock
The Cause
Patient privacy rested on one line: filter: patient_id = X. If that filter had a bug, or if the patient ID came from somewhere it shouldn't, any doctor could read any chart.
It also meant I'd be rebuilding the hospital's access rules inside my app. Who's on whose care team already lives in the EHR. A copy of that would drift.
The Fix
Three changes.
First, authorization happens before retrieval, every time, in a policy service (OPA or similar). The LLM never makes an access decision and never chooses a patient ID.
Second, where the EHR supports it, I query it with the clinician's own token (SMART on FHIR), not a super-user service account. If the EHR says no, my app can't say yes.
user/Observation.rs user/AllergyIntolerance.rs user/DocumentReference.rs Third, every store enforces scope on its own: FHIR, OpenSearch, Qdrant, object storage, and the endpoint that shows a citation. The server injects the patient scope into each query. It never comes from the client or the model.
Gateway → who are you? Policy engine → may you see this patient, for this purpose? FHIR → user-scoped token OpenSearch → server-injected patient filter Qdrant → server-injected patient filter Source viewer → check again when a citation is opened The last line matters more than it looks. An evidence ID that appeared in yesterday's answer is not permission to open it today.
6. Trap #4: "No History of Diabetes" Was Retrieved as Diabetes
The Cause
Ask "Does this patient have diabetes?" and look at what scores highest in a vector search:
"No history of diabetes." "Family history: father had type 2 diabetes." "Rule out diabetes; HbA1c ordered." All three are about diabetes. None of them say the patient has it. Embeddings are great at topic and terrible at negation.
The Fix
Clinical notes get annotated at ingestion, not at question time. Each chunk is split by section (History, Assessment, Plan, Family History) and run through an assertion detector. NegEx-style tools like medspaCy's ConText are a reasonable start.
The result is stored as metadata next to the chunk:
{ "chunk_id": "ch_771", "section": "family_history", "concepts": [ { "text": "type 2 diabetes", "assertion": "family" } ] } Retrieval still finds the chunk. But the evidence gate knows it's about the father, and the model sees that label. A claim like "the patient has diabetes" citing that chunk fails verification.
7. Trap #5: The Index Remembered a Note the Hospital Deleted
The Cause
Records change. A lab result gets corrected. A note gets marked "entered in error" because it was written in the wrong chart.
The EHR knows that. My vector index doesn't, unless something tells it. The napkin design would happily keep serving the wrong-chart note for weeks.
There was a second problem: I was storing full chunk text in Postgres, OpenSearch and Qdrant. Three extra copies of patient data, all potentially stale.
The Fix
The indexes are for finding things, not for quoting them.
Search returns chunk IDs ↓ Rehydrate: fetch current version from source (with the user's token) ↓ Version changed? → drop or re-check Status entered-in-error → drop Access now denied → drop ↓ Only then does it become evidence Keeping the indexes fresh runs on events. FHIR Subscriptions or CDC from the EHR feed an outbox table in my own database (not the EHR's, which I don't control). Workers process it idempotently: re-chunk, re-embed, or write a tombstone. An index-freshness metric sits on a dashboard, because stale is a safety problem here, not a performance one.
8. Trap #6: Keyword Matching Approved a Wrong Answer
The Cause
My output check counted shared words. Take this evidence:
"Metformin 500 mg was stopped on 2 Feb 2026 due to GI upset." And this generated sentence:
"The patient is on metformin 500 mg." High word overlap. Completely wrong. Swapping a date, a unit or a "not" barely moves a keyword score.
The Fix
This was the part I'd hand-waved the longest, so I made it concrete.
The model must cite an exact quote for each claim, with its position in the source:
{ "text": "Metformin 500 mg was stopped in February 2026.", "citations": [ { "evidence_id": "ev_014", "quote": "Metformin 500 mg was stopped on 2 Feb 2026", "start": 0, "end": 42 } ] } Structured output (vLLM's guided JSON decoding, for example) forces the shape. Then a separate verifier runs four checks, cheapest first:
def verify_claim(claim, evidence, nli): for c in claim.citations: ev = evidence.get(c.evidence_id) if ev is None: return reject("unknown evidence id") if normalize(ev.text[c.start:c.end]) != normalize(c.quote): return reject("quote not in source") quotes = " ".join(c.quote for c in claim.citations) if extract_numbers_dates_units(claim.text) - extract_numbers_dates_units(quotes): return reject("number, date or unit not in evidence") if nli.entailment(premise=quotes, hypothesis=claim.text) < ENTAIL_THRESHOLD: return reject("claim not supported by quote") return accept() The first two checks are deterministic: the quote must really exist where the model says it does. The third catches the most dangerous edits, which are numbers, dates and units. The fourth uses an NLI model to ask whether the quote actually supports the sentence. That's what catches the "stopped" versus "is on" case.
The NLI threshold isn't something to guess. It gets tuned on a set of claims labeled by clinicians, and I track two numbers: how many bad claims slip through, and how many good claims get rejected. Too many false rejections and doctors stop trusting the tool for the opposite reason.
A rejected claim is removed and the answer is marked partial, with a reason. If the core of the answer fails, the whole thing becomes an abstention. One bounded retry, then stop.
9. Trap #7: There Were Two Kamal Hossains
The Cause
The napkin design pulled the patient out of the question text: "Kamal Hossain's last blood sugar." Same name, two patients, and you have a very quiet, very serious bug.
Multi-turn chat makes it worse. Ask about one patient, then another, and the conversation memory may carry facts across.
The Fix
The patient is never inferred from free text. The clinician picks the patient in the UI, with a medical record number, date of birth and ward, and the chat session is pinned to that one patient.
Pick patient (MRN + DOB) → policy check → session bound to one patient ↓ every query inherits that scope If someone types a different name mid-conversation, the assistant doesn't switch. It asks them to open a new session. Switching patients clears the conversation memory completely. No cross-session memory in v1.
10. Trap #8: I Designed for a Hospital That Doesn't Exist
The Cause
My design assumed a clean FHIR server and English-only notes.
The hospitals I was actually thinking about, in Bangladesh, often have a mix of: a partial EHR, scanned paper, PDFs exported from lab machines, and handwritten notes. Staff write in English, Bengali, and Bengali typed in English letters, sometimes in the same sentence.
A retrieval stack tuned on clean English text would look great in a demo and fall apart in the ward.
The Fix
A Phase 0 that comes before any code: a data audit. What exists, in what format, how much is scanned, how good the scans are, and who owns each source.
Then three concrete changes:
- OCR output carries a confidence score per chunk. Low-confidence text can support a summary with a warning, but it can never be the source of an exact fact like a dose or a lab value.
- Search handles mixed languages: a multilingual embedding model (something like BGE-M3, chosen only after testing on real notes), the ICU analyzer in OpenSearch for Bengali, and normalization for romanized Bengali.
- Evaluation questions are written in the languages staff actually use, not just English.
11. Trap #9: The Perfect Answer Took 25 Seconds
The Cause
Every fix above adds a step. Policy check, retrieval from three stores, reranking, rehydration, generation, four-stage verification, maybe a retry. On private GPUs, that adds up fast.
A doctor between patients won't wait 25 seconds. They'll open the chart and never come back.
The Fix
A latency budget, written down before building, so each stage knows its limit. These are starting targets, not measurements:
| Stage | P95 target |
|---|---|
| Auth + policy | 100 ms |
| Retrieval (all stores, parallel) | 400 ms |
| Rerank + rehydrate | 500 ms |
| Generation | 4 s |
| Verification | 1 s |
| Summary question, end to end | ~6 s |
| Fact question (no LLM) | < 1 s |
The fact path from Trap #1 helps a lot here: the most common questions skip the expensive part.
I also don't stream raw model tokens. Unverified text on screen, even for a second, is exactly what this design is trying to prevent. The UI shows progress instead (checking access, searching records, verifying) and then the verified answer.
12. The Final Architecture
Clinician (hospital SSO + MFA) │ ▼ Web app ── patient picker (MRN + DOB) ── session pinned to ONE patient │ ▼ API gateway ── rate limits · request IDs │ ▼ Orchestrator (FastAPI) ├──► Policy engine ─────────────────────── deny? stop. nothing retrieved. │ ├──► Router │ ├── fact question ──► FHIR (user token) ──► template ──► answer │ │ │ └── summary / narrative │ │ │ ▼ │ ┌─ Hybrid retrieval (scope injected by server) ──────────────┐ │ │ FHIR / SQL OpenSearch (BM25) Qdrant (vector) │ │ │ └──────────► RRF fusion ► rerank ◄──────┘ │ │ └──────────────────────────┬───────────────────────────────────┘ │ ▼ │ Rehydrate from source (current version, user token) │ ▼ │ Evidence gate: owner · status · date · assertion · conflict · coverage │ ▼ │ Private LLM (JSON schema, quoted spans) │ ▼ │ Claim verifier: quote match · numbers · NLI · policy │ │ │ │ verified rejected │ ▼ ▼ │ answer + sources partial / abstain + reason │ └──► Audit log (IDs, versions, hashes; no raw patient text) Ingestion (separate path) EHR events / CDC ──► outbox ──► workers ├── OCR (with confidence) ├── section split + assertion tags ├── chunk + embed └──► upsert or tombstone in OpenSearch + Qdrant The stack I'd start with:
- Frontend: Next.js
- Backend: Python, FastAPI, background workers
- Source of truth: the hospital's existing EHR / FHIR server
- App database: PostgreSQL (users, policies, document metadata, outbox, guideline versions)
- Search: OpenSearch for keywords, Qdrant for vectors
- Models: a private LLM on vLLM, a multilingual embedding model, a cross-encoder reranker, an NLI model for verification
- Policy: OPA or similar
- Ops: OpenTelemetry, Prometheus, Grafana, a secrets manager, encrypted backups
Patient records and hospital guidelines are kept as separate corpora with separate rules. A guideline is only used as current evidence if it's approved and in its effective date range. Draft and expired versions never show up as "current guidance."
The interfaces stay swappable. Retriever, LLM and embedder each sit behind a small interface, so replacing Qdrant or the model doesn't mean rewriting the orchestrator.
13. How I'd Prove It Works
Architecture diagrams don't prove anything. Tests do.
Before any clinician sees it, the system has to pass a golden set of questions with known correct behavior:
Record exists → correct answer + source Record missing → "not found", never "none" Search index down → "couldn't check", never "not found" Same-name patients → refuse to guess No care relationship → access denied, nothing retrieved Conflicting blood groups → show the conflict Stopped medication → never described as current Expired guideline → never shown as current "Ignore your instructions" → treated as text inside a PDF, nothing more Negated / family history → never claimed as the patient's condition And these numbers get tracked on every model, prompt or index change:
- Wrong-patient retrieval rate (release gate: zero on the test suite)
- Unauthorized exposure rate (release gate: zero)
- Unsupported claims that got through
- Good claims wrongly rejected
- Abstention accuracy
- Citation accuracy
- Index freshness
- P95 latency per path
Zero on a test suite doesn't mean zero risk in production. That's why the pilot is small and monitored, and why every clinician correction becomes a new test case.
Clinicians grade answers. An LLM-as-judge score is useful for catching regressions, but it isn't evidence that an answer is clinically right.
One more thing to check early: depending on the country, a tool that summarizes patient records might count as medical-device software. That's a question for the hospital's regulatory team before launch, not after.
14. Build Order
The order matters more than the stack. Access control and data come before model tuning.
Phase 0 Data audit what exists, what format, what language Phase 1 Access foundation SSO, policy engine, patient picker, audit Phase 2 Fact path FHIR lookups + templates (useful on day one, no LLM) Phase 3 Ingestion OCR, sections, assertion tags, outbox, indexes Phase 4 Retrieval hybrid search, rehydration, evidence gate Phase 5 Generation quoted-span output + claim verifier Phase 6 Release gate security tests, clinician review, golden set Phase 7 Small pilot one department, feedback becomes tests Phase 2 is my favorite. Doctors get a fast, safe lookup tool before a single token is generated, and it proves the access layer works on real traffic.
What v1 does not do: diagnose, prescribe, change medications, or replace clinical judgment.
15. Lessons Learned
- The LLM is a narrator, not a witness. It explains evidence. It is never the source of it.
- Missing is not negative. And "unavailable" is not "missing." Model all four states.
- If a database can answer it, don't ask a model. Exact facts go through templates.
- Embeddings don't understand "no." Tag negation and family history at ingestion.
- Indexes find, sources quote. Rehydrate before you trust a chunk.
- Verify claims against quotes, not keywords. And measure false rejections too.
- Never infer the patient from text. Pin the session to one patient.
- Design for the data you have. Do the audit before the architecture diagram.
- Latency is a safety feature. A tool nobody waits for protects nobody.
16. Final Thoughts
The napkin version had five boxes. The final one has a lot more, and I spent the most time on the boring ones: access checks, data states, freshness, the verifier.
That's where the risk actually lives:
- a patient filter that's the only lock
- an empty result that reads like a negative
- a negation that embeddings can't see
- a deleted note still sitting in an index
- a keyword check that can't tell "stopped" from "taking"
None of those are model problems. A better model wouldn't fix any of them.
A hospital assistant that says "I don't know" is a pipeline problem first, not a model problem.



