Evaluating Retrieval Quality in .NET

Qdrant can rank a shutdown chunk first and still miss the passage about cancellation. Check whether retrieval found the required evidence before passing results to a model.

We'll evaluate actual Qdrant results using labeled questions and a custom IEvaluator. The evaluator calculates the metrics in C# without calling a judge model. The reporting library saves the results, and the aieval console tool produces the HTML report.

The querying article built the read path. Payload modeling and metadata filters restricted results to published runbooks, while context assembly selected the text for generation. Here, we evaluate the ranked results before assembly.

A complete, standalone version of this example is available in the sample repository.

The excerpts below explain the evaluation implementation. To run the complete experiment, use sample 07. It includes corpus preparation and validation in a standalone program that builds on the retrieval approach from the earlier articles.

Why a relevant first result is not enough

The shutdown runbook contains these passages:

For example, consider the question: “How do I stop accepting new queue items and pass cancellation to operations during shutdown?” The answer requires information from both shutdown chunks. In a measured local run:

Both result lists place a relevant chunk first. Reciprocal rank cannot distinguish these outcomes.

The two chunks also contain the same deadline paragraph. For the question “Should I drain all queued work during shutdown?”, either chunk provides the necessary condition. Labeling both as relevant makes sense, but retrieving only one doesn’t always mean the answer is incomplete. Chunk recall measures labeled passages, not individual facts. Identify the required evidence before reviewing the ranking. A plausible top result can make missing information easy to overlook.

Build a small labeled evaluation set

The original filter includes three chunks from a single published runbook, so returning all three makes Recall@3 trivial. Sample 07 adds four more published runbooks as competition:

For the complete fixture and indexing code, see sample 07 in the GitHub repository.

Including the original draft runbook and a published note, the fixture contains eleven stored points. Only seven chunks qualify for the published-runbook query. The HTTP shutdown passage is a plausible competing result: it shares shutdown terminology but does not explain a background queue.

Sample 07 uses a separate collection, blog-samples-07-eval-ollama-v1, to keep the evaluation corpus isolated from earlier examples. The pipeline continues to use the same chunking rules, payload contract, and filters. Its console application implements --index to prepare the corpus and --evaluate to validate and read it without rewriting document vectors. Both modes check the exact point count and independently read back expected identities and payloads. These checks don’t confirm vector equality or provenance.

Use .NET 10, Qdrant and Qdrant.Client 1.19.0, OllamaSharp 5.4.30, and Microsoft.Extensions.AI.Evaluation with Microsoft.Extensions.AI.Evaluation.Reporting 10.9.0. The local embeddinggemma:300m profile uses 768 dimensions and cosine distance, applying title: ... | text: ... for documents and task: search result | query: ... for queries. The standalone sample includes the provider setup and helpers. Extend its two-minute cancellation deadline if embedding calls run longer locally.

Follow the repository README to start Qdrant and configure Ollama with embeddinggemma:300m. Then run these commands from the repository root. The -- separator passes the following argument to the sample application:

dotnet run --project samples/07-retrieval-evaluation/ollama/Sample.csproj -- --index dotnet run --project samples/07-retrieval-evaluation/ollama/Sample.csproj -- --evaluate 

Run --index before the first evaluation. Repeat --evaluate against the prepared corpus when comparing retrieval settings.

The Azure OpenAI path from Indexing Your First Documents can use the same evaluator and relevance labels. It needs its own collection and document embeddings generated with the selected Azure model, along with the matching query preprocessing. The measured results in this article come from EmbeddingGemma. Evaluate the Azure configuration separately rather than assuming it produces the same rankings.

The experiment defines seven answerable cases and two unsupported questions. The Key includes the document and the current section-paragraphs-80w-overlap1-v1 chunking version, ensuring that identical chunk names from different sources do not match by mistake:

ChunkKey Key(string document, string chunk) => new(document, chunkingVersion, chunk); RetrievalCase[] cases = [ new("drain-deadline", "conditions", "Should I drain all queued work during shutdown?", [Key("worker-shutdown", "shutdown/p0"), Key("worker-shutdown", "shutdown/p1")]), new("cancel-draining", "paraphrase", "How do I pass cancellation to asynchronous work while draining the queue?", [Key("worker-shutdown", "shutdown/p1")]), new("shutdown-sequence", "multiple-passages", "How do I stop accepting new queue items and pass cancellation to operations during shutdown?", [Key("worker-shutdown", "shutdown/p0"), Key("worker-shutdown", "shutdown/p1")]), new("bounded-producer", "technical-term", "What happens to a producer when a bounded Channel with FullMode.Wait is full?", [Key("worker-shutdown", "capacity/p0")]), new("http-pooling", "technical-term", "How does IHttpClientFactory reuse HTTP handlers?", [Key("http-client-lifetime", "clients/p0")]), new("retry-budget", "conditions", "How should HTTP retries use backoff, jitter, and a total deadline?", [Key("http-request-retries", "retries/p0")]), new("restart-recovery", "paraphrase", "How can a worker recover unfinished jobs after a process restart?", [Key("durable-work-queue", "recovery/p0")]) ]; string[] unsupportedQuestions = [ "Which exact timeout does our production ingress use?", "How many days does our company retain completed job records?" ]; public sealed record RetrievalCase( string Id, string Category, string Question, ChunkKey[] Relevant); 

The baseline applies binary judgments: assign grade 2 to each listed relevant chunk and grade 0 to all other eligible chunks. For cancel-draining, only shutdown/p1 directly answers the question. The deadline overlap in shutdown/p0 doesn’t count. In future datasets, grade 1 might indicate useful supporting evidence, but that would require a separate labeling policy.

Every eligible chunk receives a judgment in this setup. Don’t treat unjudged results as irrelevant if you expand to a larger corpus. Always review these positional labels when source text or chunk boundaries change. The chunking version describes splitting rules, not the source revision.

Calculate retrieval metrics in C

Use these definitions at a fixed cutoff K:

Precision always uses K as its denominator, even if fewer results are returned. For example, if there’s one relevant result at K=3, Precision@3 is 1/3. If you measure precision over the actual number of returned results, give that metric a different name.

Each query yields a reciprocal rank. Average these values across answerable questions to compute MRR at the same cutoff. MRR does not indicate whether a second required passage was retrieved.

For nDCG, calculate gain as 2^grade - 1 and discount as log2(rank + 1), with ranks starting at one. The ideal ranking sorts all eligible judgments by grade and selects the top K. Unsupported questions do not have a relevant-chunk denominator or positive ideal gain. Evaluate their acceptance separately.

The calculation preserves Qdrant’s returned order:

public sealed record ChunkKey(string DocumentId, string Version, string ChunkId); public sealed record RankingScores( double Precision, double Recall, double ReciprocalRank, double Ndcg); public static class RetrievalMetrics { public static RankingScores Calculate( IReadOnlyList<ChunkKey> ranked, IReadOnlyDictionary<ChunkKey, int> judgments, int k) { ArgumentOutOfRangeException.ThrowIfNegativeOrZero(k); if (ranked.Distinct().Count() != ranked.Count) throw new ArgumentException("Duplicate chunk identities."); if (judgments.Values.Any(grade => grade is < 0 or > 2)) throw new ArgumentException("Grades must be 0, 1, or 2."); if (ranked.Any(key => !judgments.ContainsKey(key))) throw new ArgumentException("Returned chunk has no judgment."); int relevantCount = judgments.Count(pair => pair.Value > 0); if (relevantCount == 0) throw new ArgumentException("Score no-answer cases separately."); int[] grades = ranked.Take(k).Select(key => judgments[key]).ToArray(); int hits = grades.Count(grade => grade > 0); int firstHit = Array.FindIndex(grades, grade => grade > 0); double ideal = Dcg(judgments.Values.OrderDescending().Take(k)); return new RankingScores( (double)hits / k, (double)hits / relevantCount, firstHit < 0 ? 0 : 1d / (firstHit + 1), Dcg(grades) / ideal); } private static double Dcg(IEnumerable<int> grades) => grades.Select((grade, index) => (Math.Pow(2, grade) - 1) / Math.Log2(index + 2)).Sum(); } 

An empty ranking with valid relevant labels is a scored miss. Input errors include duplicate identities, missing judgments, invalid grades, or an empty relevant set. Keep these cases distinct so a broken adapter or dataset does not appear as a simple relevance failure.

Before connecting to Qdrant, test a hand-crafted order [C, A, B] with grades C=0, A=2, B=1. At K=3, this yields Precision=2/3, Recall=1, reciprocal rank=1/2, and nDCG around 0.659. Also check short lists, cutoff boundaries, duplicate identities, and evaluator cancellation. These arithmetic checks require no services.

A second controlled test uses the shutdown labels. Place the irrelevant HTTP passage before both queue passages: [0, 2, 2]. Precision@3 stays at 2/3, Recall@3 at 1, but reciprocal rank drops to 1/2 and nDCG to about 0.693. This reordered fixture simulates a ranking failure. It does not reflect an actual Qdrant regression.

Connect Qdrant to Microsoft’s evaluation framework

IEvaluator gives the C# calculations a common result format and reporting path. Assertions still test the arithmetic and enforce application acceptance criteria. The evaluator does not decide whether a release passes.

Return standard metrics

RankingContext contains the returned identities and their judgments. Its serialized evidence keeps the retrieved text, similarity scores, and eligibility metadata, allowing reports to explain low scores:

using System.Text.Json; using Microsoft.Extensions.AI; using Microsoft.Extensions.AI.Evaluation; public sealed record RetrievedEvidence( ChunkKey Key, float Score, string Text, string DocumentType, string PublicationState); public sealed class RankingContext : EvaluationContext { public ChunkKey[] Ranked { get; } public IReadOnlyDictionary<ChunkKey, int> Judgments { get; } public RankingContext(RetrievedEvidence[] evidence, IReadOnlyDictionary<ChunkKey, int> judgments) : base("RetrievalRanking", JsonSerializer.Serialize(new { evidence, judgments = judgments.Select(pair => new { key = pair.Key, grade = pair.Value }) })) { Ranked = evidence.Select(item => item.Key).ToArray(); Judgments = new Dictionary<ChunkKey, int>(judgments); } } public sealed class RetrievalMetricsEvaluator : IEvaluator { private readonly int _k; public IReadOnlyCollection<string> EvaluationMetricNames { get; } public RetrievalMetricsEvaluator(int k) { ArgumentOutOfRangeException.ThrowIfNegativeOrZero(k); _k = k; EvaluationMetricNames = [$"Precision@{k}", $"Recall@{k}", $"ReciprocalRank@{k}", $"nDCG@{k}"]; } public ValueTask<EvaluationResult> EvaluateAsync( IEnumerable<ChatMessage> messages, ChatResponse modelResponse, ChatConfiguration? chatConfiguration = null, IEnumerable<EvaluationContext>? additionalContext = null, CancellationToken cancellationToken = default) { cancellationToken.ThrowIfCancellationRequested(); NumericMetric[] metrics = EvaluationMetricNames .Select(name => new NumericMetric(name)).ToArray(); var result = new EvaluationResult(metrics); try { RankingContext[] contexts = additionalContext? .OfType<RankingContext>().ToArray() ?? []; if (contexts.Length != 1) throw new ArgumentException("Supply exactly one RankingContext."); RankingContext context = contexts[0]; RankingScores scores = RetrievalMetrics.Calculate( context.Ranked, context.Judgments, _k); double[] values = [scores.Precision, scores.Recall, scores.ReciprocalRank, scores.Ndcg]; for (int i = 0; i < metrics.Length; i++) { metrics[i].Value = values[i]; metrics[i].Reason = $"Scored {Math.Min(_k, context.Ranked.Length)} results at K={_k}."; metrics[i].AddOrUpdateContext(context); } } catch (ArgumentException error) { foreach (NumericMetric metric in metrics) { metric.AddDiagnostics(EvaluationDiagnostic.Error(error.Message)); metric.Interpretation = new EvaluationMetricInterpretation( EvaluationRating.Inconclusive, failed: true, reason: error.Message); } } return ValueTask.FromResult(result); } } 

The evaluator does not call a model and ignores ChatConfiguration. The runner provides an empty response to meet the interface, but this is not a generated answer. Invalid input leaves metric values unset and adds error diagnostics. A valid empty ranking results in zeroes. The runner should fail runs with input errors but include zero scores in overall averages.

The runner owns aggregation and the dataset denominator. The fixture text is authored. Before storing real document bodies, decide what evidence to retain and who can access it.

Query the fixed collection

Sample 07 includes the Qdrant client, embedding generator, query preprocessing, and payload helpers. The following excerpt shows the retrieval boundary after collection validation. The expected identities come from labels. The actual identities come from Qdrant:

var eligibility = new Filter { Must = { MatchKeyword("chunking_version", chunkingVersion), MatchKeyword("document_type", "runbook"), MatchKeyword("publication_state", "published") } }; float? threshold = null; // Baseline: no score-based acceptance cutoff. SearchParams? searchParameters = null; // No per-query overrides; use server defaults. async Task<ScoredPoint[]> SearchAsync(ReadOnlyMemory<float> vector, int k) { var hits = await client.QueryAsync(collectionName, query: vector.ToArray(), filter: eligibility, limit: (ulong)k, scoreThreshold: threshold, searchParams: searchParameters, payloadSelector: new[] { "document_id", "chunk_id", "chunking_version", "document_type", "publication_state", "text" }, vectorsSelector: false, cancellationToken: token); foreach (ScoredPoint hit in hits) { if (PayloadFields.RequiredString(hit.Payload, "publication_state") != "published" || PayloadFields.RequiredString(hit.Payload, "document_type") != "runbook" || PayloadFields.RequiredString(hit.Payload, "chunking_version") != chunkingVersion) throw new InvalidOperationException("Retrieval eligibility violation."); } return hits.ToArray(); } static RetrievedEvidence ToEvidence(ScoredPoint hit) => new( new ChunkKey( PayloadFields.RequiredString(hit.Payload, "document_id"), PayloadFields.RequiredString(hit.Payload, "chunking_version"), PayloadFields.RequiredString(hit.Payload, "chunk_id")), hit.Score, PayloadFields.RequiredString(hit.Payload, "text"), PayloadFields.RequiredString(hit.Payload, "document_type"), PayloadFields.RequiredString(hit.Payload, "publication_state")); async Task<ReadOnlyMemory<float>> EmbedQueryAsync(string question) { ReadOnlyMemory<float> vector = await generator.GenerateVectorAsync( prepareQuery(question), cancellationToken: token); if (vector.Length != vectorSize) throw new InvalidOperationException("Query dimension mismatch."); return vector; } 

The mandatory filter limits candidates within Qdrant. If you apply filtering only after retrieving an unrestricted top K, you could lose eligible evidence before evaluation. The payload guard ensures that returned results meet the contract.

The runner embeds each question once and reuses that vector for K=1, 3, and 5. Keep the corpus, labels, filters, and score cutoff unchanged. Approximate searches may return different ranking prefixes as K changes. Check result identities if your analysis requires consistency. In this run, they matched. Note that recall against relevance labels differs from ANN recall with exact nearest neighbors.

Persist each question and configuration

Sample 07 constructs prepared from the complete source fixture before indexing or evaluation. The reporting excerpt builds the eligible identity set from these sources, not from the returned hits. Otherwise, a missing expected chunk could vanish from the denominator, making a miss appear as perfect recall.<...