Short answer: for a financial research dashboard, the retrieval architecture should be a costed, traceable pipeline with observability for index freshness, evidence quality, and latency. That decision matters more than picking a fashionable vector store. The same design works for a property-management watcher that compares listing pages, where every unnecessary re-index is a direct operating cost.
Make the signal explain the bill.
How should observability shape a retrieval architecture for a financial research dashboard?
Start with one trace per user question. Give the request a stable trace ID, then record query rewriting, candidate retrieval, reranking, context assembly, and generation as child spans. Each span should carry a small set of dimensions: corpus name, index version, filter count, candidate count, token estimate, and outcome. Keep document text and account identifiers out of attributes; trace metadata is not a free second database.
This gives an operator a useful causal chain. A slow answer can be a cold shard, an over-broad date filter, or a reranker receiving 200 candidates instead of 40. A confident answer with weak citations points to a different failure: stale or poorly chunked evidence. Those are different fixes, and a single end-to-end latency number cannot distinguish them.
For a financial dashboard, log the as-of timestamp and the source publication timestamp on every evidence item. A dashboard may combine filings, market notices, and analyst notes. The retrieval record must say which index snapshot supplied each chunk, so an auditor can reproduce the answer after a re-embedding job.
A runnable TypeScript trace for staged retrieval
The following small adapter uses the OpenTelemetry API and a vendor-neutral retriever. It deliberately measures counts and timings rather than exporting document contents. Replace retriever.search with the implementation behind your service boundary.
import { trace, SpanStatusCode } from "@opentelemetry/api"; type Hit = { id: string; score: number; publishedAt: string }; type Retriever = { search(query: string, limit: number): Promise<Hit[]> }; const tracer = trace.getTracer("research-retrieval"); export async function retrieveEvidence( question: string, retriever: Retriever, ): Promise<Hit[]> { return tracer.startActiveSpan("retrieval.pipeline", async (span) => { const started = performance.now(); try { span.setAttribute("retrieval.query_length", question.length); const hits = await tracer.startActiveSpan("retrieval.candidates", async (child) => { child.setAttribute("retrieval.limit", 40); const result = await retriever.search(question, 40); child.setAttribute("retrieval.candidate_count", result.length); child.end(); return result; }); const fresh = hits.filter((hit) => Date.parse(hit.publishedAt) > Date.now() - 1000 * 60 * 60 * 24 * 365); span.setAttribute("retrieval.fresh_count", fresh.length); span.setAttribute("retrieval.elapsed_ms", Math.round(performance.now() - started)); span.setStatus({ code: SpanStatusCode.OK }); return fresh; } catch (error) { span.recordException(error as Error); span.setStatus({ code: SpanStatusCode.ERROR }); throw error; } finally { span.end(); } }); } The freshness filter is an example policy, not a universal rule. A ten-year annual report can be exactly the right evidence for a trend question. Put such policies in the query contract, and emit the policy name as an attribute instead of silently changing it in a worker.
Where index cost appears, and how to see it
Index cost is usually a multiplication problem: documents ingested, chunks per document, embedding calls per chunk, replicas, and refresh frequency. Observability should expose each factor. Track documents_changed, chunks_written, embedding_requests, and index_bytes per job. Then derive cost from your actual provider invoices or internal rates outside the hot path. A dashboard that only reports monthly spend tells you that you lost; it does not tell you which ingestion rule caused it.
The property-management watcher makes the failure mode concrete. Suppose a page is fetched every 15 minutes and a parser emits a new chunk ID whenever whitespace shifts. The page has not meaningfully changed, but the pipeline re-embeds every chunk. Store a normalized content hash, compare it before embedding, and trace both bytes_fetched and chunks_skipped. The alert should be based on a semantic diff, while the index job should be based on a stable hash. In practice, normalization needs a deliberately boring test corpus: the same listing with a rotated tracking parameter, a reordered amenity list, a changed price, and a genuinely new unit. For each fixture, record whether the fetch, parse, hash, embedding, and alert stages ran. That lets you distinguish a parser change from a source change. It also lets finance teams map an index spike to a deploy, rather than guessing from a weekly invoice. The extra assertions take an afternoon; the saved investigation time compounds every time a source redesigns its HTML.
I once treated a rising vector count as a storage problem. It was a canonicalization problem: tracking parameters made identical listings look new. The useful signal was the ratio of unique_content_hashes to fetched_pages, not the vector database dashboard. Three words: measure the cause. It matters.
Your mileage may vary when market data arrives in bursts. A queue can smooth embedding work, but it also increases freshness lag. Record queue age and index version in the retrieval span, and make the UI show that lag beside the answer. Users can then choose a slightly older, cheaper snapshot knowingly instead of assuming “latest” means current.
Failure modes that traces should make boring
A robust test set includes empty retrieval, duplicate chunks, a source deleted between retrieval and generation, and a reranker timeout. For each case, assert a useful response contract: citations identify stable source IDs, the answer states when evidence is missing, and the trace closes every span. Do not rely on log sampling for these checks; sampled traces can hide the one query that matters during an incident.
Propagate context with the W3C Trace Context headers across the API, queue, and indexing worker. In a multi-tenant financial product, also enforce attribute allowlists at the exporter. Query text can contain material nonpublic information, so redaction belongs before telemetry leaves the process, not in a dashboard query later.
The catch is operational overhead. Full-fidelity traces for every token and document are not suitable when ingestion volume is high or retention rules are strict. Keep aggregate metrics for all traffic, sample representative traces, and temporarily raise sampling for a failing index version. Stick with plain structured logs when the system is a single process and the bottleneck is already obvious; distributed tracing earns its complexity when a request crosses queues and retrieval services.
Before shipping, walk one question from browser to answer and verify that its trace includes the index version, candidate count, freshness policy, evidence IDs, and generation outcome. Re-run the same question after an index rebuild. If those fields change without an intentional deployment marker, your observability contract is incomplete.
