Short answer: treat every claim document as an immutable revision, and make retrieval select a valid revision before it selects a chunk. For an insurance marketplace building an internal knowledge-base bot, this keeps an adjuster from receiving a superseded coverage clause while still allowing an audit to reconstruct what the bot saw.
The expensive part is rarely the embedding call. It is retention: storing every extracted chunk, vector, source pointer, and re-indexing attempt for documents that may be replaced minutes apart. I start by measuring that inventory, then decide which history must remain queryable and which history only needs an audit record.
Where the bill and the risk actually come from
A claims intake pipeline commonly keeps four copies of meaning: the original file, normalized text, chunk records, and index metadata. A fifth copy appears when a revised file is embedded before the old vectors are retired. At marketplace scale, that overlap dominates storage and reprocessing work more than the raw upload.
The practical accounting unit is a revision, not a file. A 12-page medical invoice and its corrected 12-page invoice should have separate content hashes, extraction timestamps, policy identifiers, and effective intervals. Their chunks may have identical text, but they do not have identical legal context. Deduplicating by text alone can silently merge two valid revisions. I once assumed a single document_id plus an updated_at field would be enough. It wasn't. A late-arriving correction made an older chunk look current because ingestion time and policy effective time were different clocks. The fix was to store both clocks and require an explicit supersedes edge. Keep the raw artifact and a compact manifest for every revision, but keep vectors only for revisions that can be returned by the product's search policy. That is the first retention decision.
When a revision is superseded, mark it non-current immediately, but do not delete it synchronously. Deletion can race with an in-flight answer, and the audit trail still needs the old hash. A tombstone plus a delayed garbage-collection job gives operators a deterministic boundary.
How should insurance claims intake handle document versioning for retrieval?
Use a two-stage query. Stage one resolves the admissible revision set; stage two ranks chunks inside that set. The filter should consider claim ID, document type, jurisdiction, policy effective date, and a revision status such as current or historical. Similarity is a ranking signal, not a time machine.
Full stop.
A useful record shape is deliberately boring:
from dataclasses import dataclass from datetime import datetime @dataclass(frozen=True) class Revision: claim_id: str revision_id: str content_sha256: str effective_from: datetime effective_to: datetime | None supersedes: str | None status: str def admissible(revision: Revision, as_of: datetime) -> bool: return ( revision.status == "current" and revision.effective_from <= as_of and (revision.effective_to is None or as_of < revision.effective_to) ) The bot should pass an as_of value derived from the claim event, not from the moment a human asks a question. If a payment dispute is reviewed six months later, the answer may need the clause effective on the loss date. If the workflow is strictly operational and always wants the latest approved guidance, say that in the policy and test it separately.
Chunk IDs should include the revision ID. A stable chunk number such as page-03:chunk-02 is useful for display, but it is not a globally stable identity after OCR or layout changes. Store character offsets or page anchors so a reviewer can open the exact source span.
Five rules that keep freshness measurable
Publish revisions, do not mutate them. Ingestion writes a new revision and a manifest event. A publisher flips the retrieval status only after extraction, chunking, and index writes have all succeeded.
Define freshness as a contract. For example, an approved endorsement must become retrievable within a stated number of minutes, while a scanned attachment can wait for a slower OCR queue. Track the age of the newest eligible revision, not only queue latency.
Version the chunker and parser. A parser change can alter boundaries even when the source file is byte-for-byte identical. Record
parser_version,chunker_version, and embedding model identifiers beside each chunk so replays are explainable.Make stale reads visible. Return revision metadata with every passage: revision ID, effective date, and source hash. The answer composer can then refuse to cite a passage outside the admissible set instead of quietly blending it with a current passage.
Test replacement races. Feed events in this order: revision B arrives, revision A finishes indexing, then B publishes. The expected result is still B for current queries. Run the inverse order too. Race tests catch more real failures than another offline similarity score.
Freshness is a user-visible property. A dashboard that says “index healthy” while a new exclusion is not searchable is reporting infrastructure, not correctness.
Retention is a product decision, not a cleanup script
There are three sensible retention tiers. Keep current chunks hot for normal retrieval. Keep superseded chunks cold but addressable for investigations. Keep a minimal manifest and source hash for revisions that policy allows you to purge. The exact durations belong to legal and records teams; engineering should expose the knobs rather than hard-code a universal number.
The catch is that historical retrieval can become a liability if the bot has no explicit mode switch. A claims handler asking “what applies now?” should not receive a clause preserved only for an old filing. Conversely, an appeals reviewer may need that old clause. Use separate query intents or an as_of requirement, and log which one was selected.
Do not retain full text merely because storage is available. Redaction rules, access controls, and deletion requests still apply to cold data. If a source must be removed, remove its searchable derivatives and leave only the minimum audit evidence your policy permits.
Operating the pipeline under change
Observability should join ingestion events to retrieval traces. I look for four timestamps: source received, extraction complete, revision published, and passage returned. Their differences reveal whether a freshness miss came from OCR, queueing, publication, or filtering. A trace that contains only a vector-search duration cannot answer that question.
For deployments, use a shadow index when changing chunking or embeddings. Replay a fixed sample of claim revisions, compare selected revision IDs and citation spans, then promote the new index by changing a read alias. Keep the old alias until active answers drain. Your mileage may vary on the replay window; it depends on claim volume and the longest in-flight workflow.
Standards help at the boundaries. HTTP caching semantics describe validators and freshness, while the Retrieval-Augmented Generation paper frames retrieval as a separate source of context for generation. Neither standard chooses your retention policy; that is an application decision.
A small runbook is enough: verify the manifest event, verify the revision filter, inspect the cited hash, and only then inspect ranking scores. This order keeps debugging grounded in what the user was allowed to see.
That ordering matters.
