Evaluating LLM Reliability for Incident Diagnosis: An SRE Approach to Testing AI Systems

──────────────────────────────────────────────────────────────────────────── LLM FAILURE MODES VS DETERMINISTIC SOFTWARE FAILURE MODES DETERMINISTIC SOFTWARE: Failure mode: Returns error code; times out; returns wrong type Detection: Standard integration tests catch these Characteristic: Failures are generally consistent and reproducible LLM FAILURE MODES (not detectable by standard integration tests): → Confidently wrong: Returns a plausible but incorrect diagnosis with high confidence score. No error code. No exception. Appears to function correctly. → Inconsistently right: Returns correct diagnosis on 70% of invocations for the same incident, wrong diagnosis on 30%. Passes integration tests that check one invocation. → Distribution shift degradation: Was reliable at deployment; degrades silently as environment evolves and the gap between training data and current incident patterns widens. → Context length sensitivity: Correct with minimal context; incorrect when context exceeds a threshold (token limit effects, attention dilution). Standard integration tests use minimal context; production uses full context. → Confidence miscalibration: Reports 0.90 confidence when empirically accurate 65% of the time. Escalation policies that gate on reported confidence are then systematically wrong about autonomy thresholds. ──────────────────────────────────────────────────────────────────────────── 

──────────────────────────────────────────────────────────────────────────── TEST DATASET CONSTRUCTION PROTOCOL STEP 1: Identify candidate incidents Source: PagerDuty or equivalent incident management system Filter: P1/P2 incidents only (highest quality postmortems) Time window: Last 18 months (sufficient volume; recent enough to be representative of current system state) Target size: Minimum 50 incidents; 100+ for statistical validity STEP 2: Confirm root cause ground truth Requirement: Root cause must be independently confirmed (not just "as stated in the postmortem" — postmortems are not always accurate; some attribute to proximate cause rather than root cause) Ground truth confirmation sources (in order of reliability): a) Engineering postmortem with specific code change identified b) Engineering postmortem with specific configuration identified c) Peer-reviewed incident analysis (two engineers agree) d) Postmortem with remediation that resolved and did not recur Exclude: Incidents where root cause remains disputed or "unknown" Exclude: Incidents where symptoms match multiple simultaneous causes STEP 3: Reconstruct the incident context as it appeared at alert time This is the most labour-intensive step and the most critical. The test must evaluate the LLM on the context that was available WHEN THE INCIDENT OCCURRED — not on hindsight context. Context to reconstruct: → Prometheus metrics in the 30-minute window before first alert → Splunk log events in the 30-minute window before first alert → Recent deployments (Argo CD sync events, last 24 hours) → Recent configuration changes (git commits, last 7 days) → Active alerts at time of incident creation Context to EXCLUDE (post-hoc information not available at alert time): → Logs generated after the incident started (investigation artefacts) → Configuration changes made during incident response → Any information that assumes knowledge of the outcome STEP 4: Label the dataset For each incident, record: → incident_id, timestamp, affected_service → confirmed_root_cause (structured: category + specific cause) → root_cause_category (deployment, infrastructure, dependency, configuration, capacity, external) → severity (P1/P2), duration_minutes, budget_consumed_pct DATASET QUALITY METRIC: Ground truth confidence score: fraction of incidents where root cause is confirmed by source type (a) or (b) above. Target: > 80% of dataset confirmed at source (a) or (b) level. Datasets with high proportions of source (c) or (d) produce noisier accuracy measurements. ──────────────────────────────────────────────────────────────────────────── 

# LLM Accuracy Evaluation — Automated Test Runner # Presents reconstructed incident context to LLM and measures # diagnosis accuracy against ground truth dataset  import json from dataclasses import dataclass from typing import Optional @dataclass class IncidentTestCase: incident_id: str context: dict # Reconstructed at-alert-time context  ground_truth: dict # Confirmed root cause  @dataclass class DiagnosisResult: incident_id: str llm_diagnosis: str confidence_score: float model_used: str latency_ms: int is_correct: Optional[bool] = None # Set after evaluation  def evaluate_accuracy( test_cases: list[IncidentTestCase], llm_client, evaluator_fn ) -> dict: """ Run accuracy evaluation against historical incident test dataset. Returns precision, recall, and per-category breakdown. """ results = [] for case in test_cases: # Present context to LLM (same prompt as production)  response = llm_client.diagnose( context=case.context, prompt_template=PRODUCTION_PROMPT_TEMPLATE ) # Evaluate correctness against ground truth  # (evaluator_fn uses semantic similarity + keyword matching)  is_correct = evaluator_fn( diagnosis=response.diagnosis, ground_truth=case.ground_truth, threshold=0.80 # 80% semantic similarity = correct  ) results.append(DiagnosisResult( incident_id=case.incident_id, llm_diagnosis=response.diagnosis, confidence_score=response.confidence, model_used=response.model, latency_ms=response.latency_ms, is_correct=is_correct )) # Calculate metrics  total = len(results) correct = sum(1 for r in results if r.is_correct) diagnosed = sum(1 for r in results if r.llm_diagnosis is not None) precision = correct / diagnosed if diagnosed > 0 else 0 recall = correct / total # All incidents have ground truth  # Per root-cause-category breakdown  by_category = {} for case in test_cases: cat = case.ground_truth["root_cause_category"] category_results = [r for r in results if r.incident_id == case.incident_id] if category_results: by_category.setdefault(cat, {"correct": 0, "total": 0}) by_category[cat]["total"] += 1 if category_results[0].is_correct: by_category[cat]["correct"] += 1 return { "precision": precision, "recall": recall, "total_incidents": total, "by_category": { cat: data["correct"] / data["total"] for cat, data in by_category.items() }, "gate_1_pass": precision >= 0.70 and recall >= 0.60 } 

# Calibration Testing — Confidence Score vs Empirical Accuracy # Measures whether reported confidence correlates with actual accuracy  def evaluate_calibration(results: list[DiagnosisResult]) -> dict: """ Measure confidence calibration: does reported confidence match empirical accuracy rate? """ confidence_bands = { "90-100": {"min": 0.90, "max": 1.01}, "85-89": {"min": 0.85, "max": 0.90}, "80-84": {"min": 0.80, "max": 0.85}, "70-79": {"min": 0.70, "max": 0.80}, "60-69": {"min": 0.60, "max": 0.70}, "below-60": {"min": 0.0, "max": 0.60} } calibration = {} for band_name, bounds in confidence_bands.items(): band_results = [r for r in results if bounds["min"] <= r.confidence_score < bounds["max"]] if len(band_results) >= 5: # Minimum sample for validity  empirical_accuracy = sum(1 for r in band_results if r.is_correct) / len(band_results) mean_confidence = sum(r.confidence_score for r in band_results) / len(band_results) calibration_delta = empirical_accuracy - mean_confidence calibration[band_name] = { "sample_size": len(band_results), "empirical_accuracy": empirical_accuracy, "mean_stated_confidence": mean_confidence, "calibration_delta": calibration_delta, "status": "CALIBRATED" if abs(calibration_delta) <= 0.10 else "MISCALIBRATED" } # Gate 2 pass: all bands with sufficient samples are calibrated  gate_pass = all( band["status"] == "CALIBRATED" for band in calibration.values() if band["sample_size"] >= 10 ) return {"calibration_by_band": calibration, "gate_2_pass": gate_pass} 

# Consistency Testing — Same input, same output? # Critical for autonomous execution: inconsistent diagnoses cannot # be trusted for policy-governed automatic remediation  def evaluate_consistency( test_cases: list[IncidentTestCase], llm_client, n_reruns: int = 3 ) -> dict: """ Test consistency: present the same incident context N times and measure semantic agreement across responses. """ consistency_scores = [] for case in test_cases[:30]: # Sample: 30 incidents × 3 reruns = 90 calls  responses = [] for _ in range(n_reruns): response = llm_client.diagnose( context=case.context, prompt_template=PRODUCTION_PROMPT_TEMPLATE, temperature=0.1 # Low temperature for consistency testing  ) responses.append(response.diagnosis) # Pairwise semantic similarity across all response pairs  pairs = [(responses[i], responses[j]) for i in range(len(responses)) for j in range(i+1, len(responses))] pairwise_similarities = [ semantic_similarity(a, b) for a, b in pairs ] incident_consistency = sum(pairwise_similarities) / len(pairwise_similarities) consistency_scores.append(incident_consistency) mean_consistency = sum(consistency_scores) / len(consistency_scores) return { "mean_consistency_score": mean_consistency, "incidents_tested": len(consistency_scores), "gate_3_pass": mean_consistency >= 0.85 } 

# Context Length Stress Test # Tests whether diagnosis accuracy degrades with increasing context size # Production incidents often have larger context than lab tests  def evaluate_context_length_sensitivity( test_cases: list[IncidentTestCase], llm_client ) -> dict: """ Test diagnosis accuracy at multiple context sizes. Reveals context length threshold where accuracy degrades. """ context_sizes = ["minimal", "standard", "extended", "maximum"] results_by_size = {} for size_label in context_sizes: size_results = [] for case in test_cases[:20]: # 20 incidents per size  # Build context at specified size  context = build_context_at_size(case.context, size_label) response = llm_client.diagnose(context=context) is_correct = evaluate_diagnosis(response.diagnosis, case.ground_truth) size_results.append({ "incident_id": case.incident_id, "context_tokens": len(context) // 4, # approximate  "is_correct": is_correct, "confidence": response.confidence, "latency_ms": response.latency_ms }) accuracy = sum(1 for r in size_results if r["is_correct"]) / len(size_results) avg_latency = sum(r["latency_ms"] for r in size_results) / len(size_results) results_by_size[size_label] = { "accuracy": accuracy, "avg_latency_ms": avg_latency, "avg_tokens": sum(r["context_tokens"] for r in size_results) / len(size_results) } # Identify degradation threshold  accuracies = [results_by_size[s]["accuracy"] for s in context_sizes] degradation_threshold = None for i in range(1, len(accuracies)): if accuracies[i] < accuracies[i-1] * 0.85: # >15% accuracy drop  degradation_threshold = context_sizes[i] break return { "by_context_size": results_by_size, "degradation_threshold": degradation_threshold, "gate_4_pass": degradation_threshold != "standard" # Fail if accuracy degrades at standard context size (production representative)  } 

──────────────────────────────────────────────────────────────────────────── LLM INCIDENT DIAGNOSIS PoC EXIT GATES GATE 1 — ACCURACY (Technical Performance) Pass criteria: Precision ≥ 0.70 (correct root cause in ≥ 70% of diagnoses made) Recall ≥ 0.60 (identifies root cause in ≥ 60% of all test incidents) Deployment-caused incidents: precision ≥ 0.80 (highest-volume category) Measurement: Automated accuracy evaluation on 50+ test cases Sample required: ≥ 50 P1/P2 incidents with confirmed ground truth Failure response: Review prompt engineering; assess context completeness; evaluate model selection GATE 2 — CALIBRATION Pass criteria: Calibration delta within ±0.10 for all confidence bands ≥ 0.60 No confidence band shows systematic overconfidence > 0.15 Measurement: Calibration evaluation on same 50+ test cases Failure response: Adjust confidence thresholds in escalation policy; if systematic overconfidence: apply calibration correction factor before gating GATE 3 — CONSISTENCY Pass criteria: Mean semantic consistency ≥ 0.85 across 30 test incidents × 3 reruns No individual incident shows consistency < 0.60 (outlier cap) Measurement: Rerun test suite with same 30 incidents × 3 times Failure response: Reduce temperature; review prompt for ambiguity; evaluate model quantisation effects GATE 4 — CONTEXT LENGTH STABILITY Pass criteria: Accuracy does not degrade by > 15% at standard context size (representative of typical production incident context) Latency at maximum context: < 120 seconds (Ollama), < 60 seconds (external) Measurement: Context size stress test at 4 levels Failure response: Reduce context assembly scope; prioritise high-signal context (recent deployments, correlated alerts) over full log ingestion GATE 5 — ENGINEER TRUST SURVEY Pass criteria: SRE team mean score ≥ 3.5/5.0 across five dimensions No individual respondent < 2.0 (veto threshold) Completion rate: 100% of on-call SRE engineers surveyed Measurement: Structured survey after 30-day shadow mode Survey dimensions: Q1: Recommendations are relevant to the actual incident (1–5) Q2: I can verify a recommendation in under 5 minutes (1–5) Q3: The tool changes how I approach investigation (1–5) Q4: I would trust this recommendation enough to act on it (1–5) Q5: The explanations help me understand the reasoning (1–5) Failure response: This gate cannot be fixed with technical changes. Failure means the tool is not ready for the team, regardless of technical performance. Return to shadow mode; understand specific objections; address them before re-surveying. ──────────────────────────────────────────────────────────────────────────── GATE SEQUENCE AND DEPENDENCIES: Gates 1–4 can be evaluated in parallel (all from test dataset) Gate 5 requires Gates 1–4 to pass first (survey shadow mode starts only after technical validation is complete) All five gates must pass before any operational authority is granted ──────────────────────────────────────────────────────────────────────────── 

# Prometheus: LLM Diagnosis Reliability SLIs # Continuous measurement of all four reliability dimensions post-deployment groups: - name: llm.diagnosis.reliability rules: # Accuracy proxy: fraction of acted-on recommendations # that led to incident resolution (tracked post-incident) - record: llm:diagnosis_accuracy:rate7d expr: | sum(rate(llm_diagnosis_resolution_confirmed_total[7d])) / sum(rate(llm_diagnosis_acted_on_total[7d])) # Operator override rate (high = operators don't trust diagnoses) - record: llm:operator_override_rate:rate7d expr: | sum(rate(llm_diagnosis_operator_reversed_total[7d])) / sum(rate(llm_diagnosis_acted_on_total[7d])) # Calibration delta (production: rolling comparison) - record: llm:confidence_calibration_delta:rate30d expr: | abs( avg_over_time(llm_diagnosis_confidence_score_avg[30d]) - avg_over_time(llm_diagnosis_empirical_accuracy_avg[30d]) ) # Composite trust score (same weights as AIOps Trustworthy post) - record: llm:trust_score:composite expr: | (llm:diagnosis_accuracy:rate7d * 0.40) + ((1 - clamp_max(llm:confidence_calibration_delta:rate30d, 0.30)) * 0.20) + (llm:diagnosis_consistency:rate7d * 0.20) + ((1 - llm:operator_override_rate:rate7d) * 0.20) # Trust score degradation alert - alert: LLM_DiagnosisTrustScoreDegraded expr: llm:trust_score:composite < 0.65 for: 6h labels: severity: ticket domain: ai_reliability annotations: summary: > LLM diagnosis trust score at {{ $value | humanize }}. Accuracy or calibration degrading — review recent incident corpus for distribution shift. Consider re-evaluation.