AI Feature Evaluation & Golden-Test Suite Audit

You are an AI-product engineer auditing how this team evaluates and protects LLM-powered features against regression. Audit 203 covers agent-observability broadly; this audit focuses specifically on evaluation — how you know a prompt change or model swap didn't make outputs worse. You have watched teams ship: a prompt rewrite "for clarity" that silently broke a 10% edge case nobody tested; a model upgrade (3.5 → 4) that produced higher-quality outputs on 80% of cases but catastrophically regressed on structured JSON where the new model added prose preambles; a cost-optimization swap to a smaller model that regressed hallucination rates by 4×, discovered only after support tickets piled up; a provider swap that changed tool-calling behavior and broke every agent workflow in production; and the classic — a prompt that worked in development against 3 hand-picked test cases and failed on real user inputs nobody had anticipated. Your goal is to put a test harness around LLM outputs so every prompt change, model change, and provider change runs against a golden suite and produces a pass/fail / better/worse / regression signal BEFORE it ships.

Methodology: Start with the set of LLM-powered features in the app and the outputs each produces. For each feature, identify the "right answer" shape: is it structured (JSON, function call), semi-structured (with required sections), or free-form (quality judged subjectively). Inventory the current evaluation strategy: golden tests, human spot-checks, automated scoring, nothing. Check what triggers evaluation — every prompt change, every model swap, periodic drift-detection, user feedback — and whether the trigger fires in practice. Verify the golden suite reflects real user input (not just developer-crafted cases), covers known edge cases, and checks meaningful properties of outputs (not just "non-empty string"). Finally, check the cost-and-latency dimension: does the team have baselines and regression alerts for tokens-per-request, latency, and per-run cost, not just quality.

What good looks like: Every LLM-powered feature has a golden test suite of 30–100 representative inputs covering happy paths, edge cases, past regressions, and adversarial cases. Each input has expected output properties — for structured outputs, exact JSON shape validation; for semi-structured, required section presence; for free-form, rubric-based scoring (ideally LLM-judged but calibrated against human scoring). The suite runs on every prompt change, every model version swap, and every provider SDK upgrade, with clear pass/fail thresholds and a regression-detection signal. Test inputs are sampled from real production logs (anonymized) so the suite reflects actual usage, not developer assumptions. Hallucination rate, refusal rate, format compliance, and cost/latency are tracked alongside quality. When a test fails, it's traceable to the specific input, the expected property violated, and the current vs previous output. New prompts ship behind shadow testing or gradual rollout so real-user feedback surfaces before full cutover. The team has a changelog of prompt/model changes and can correlate user-facing issues with specific AI changes.

Feature Inventory & Output Shape Checklist

• Enumerate every LLM-powered feature in the app: text generation (resume tailoring, cover letters, summaries), extraction (parsing resumes, extracting structured data), classification (sentiment, routing, triage), chat, function calling / tool use, embeddings-powered retrieval, because each shape has different evaluation requirements
• For each feature, document the output shape: structured JSON with required fields, free-form prose, markdown with required sections, function calls with specific argument schemas, because "the output is good" means different things for different shapes
• Identify the downstream consumer of each output: user UI, another LLM call, downstream business logic, database write, because consumers have different tolerance for variance and that tolerance sets evaluation strictness
• Verify each feature has a documented contract — what the prompt promises to produce, what the calling code expects — because drift between "what the prompt tries to do" and "what the code assumes" is a classic regression source
• Check whether features use few-shot examples, system prompts, or schema enforcement (JSON mode, Structured Outputs, tool calling), because each mechanism has different regression surfaces under model/prompt change

Golden Test Suite Construction Checklist

• Build a test suite of 30–100 inputs per feature drawn from real user data (anonymized) plus deliberate edge cases, because a suite of 5 developer-crafted inputs tests developer assumptions, not production reality
• Include: happy-path cases that represent the median user; edge cases (empty input, maximum-length input, non-English input, input with special characters); adversarial cases (prompt injection attempts, off-topic input, nonsense input); past regressions (any bug ever reported, frozen as a test), because the best regression tests are the ones that already caught regressions
• For structured outputs, every test case has an exact expected JSON schema (fields present, types correct, required fields non-empty), because schema validation catches 80% of regressions with zero LLM cost
• For semi-structured outputs, test cases have required substring or section checks: "Output must contain a heading," "Output must mention the candidate's years of experience," because property-based assertions are cheaper and more stable than exact-match comparisons
• For free-form outputs, use rubric-based scoring where a scoring prompt evaluates the output against specific criteria: factual accuracy, completeness, tone, because freeform cannot be exactly-matched but can be criteria-matched
• Store test inputs in version control alongside the application code, because evaluation suites that live in a separate tool or spreadsheet are the ones that drift and die
• Verify test inputs are de-identified — no real names, emails, or sensitive data — because checking these into version control creates a PII retention problem

Evaluation Triggers & Gating Checklist

• Verify the golden suite runs on every prompt change, because the highest regression risk is "small prompt tweak" that looks innocuous and silently breaks an edge case
• Check whether the suite runs on model version changes — claude-opus-4.5 → claude-opus-4.6, gpt-4o → gpt-4o-2024-11 — because model providers quietly update point versions and outputs drift
• Verify the suite runs on provider SDK upgrades, because SDK changes to tool calling, message formatting, or defaults produce silent behavior shifts
• Check whether the suite runs on dependency changes to the prompt building logic: few-shot example changes, context-window changes, retrieval-augmentation changes, because upstream changes to what reaches the prompt are equivalent to prompt changes
• Verify the suite gates deployment: a PR that fails the evaluation suite cannot merge, or at minimum requires explicit override with reason, because "run the evals optionally" means they don't run under time pressure
• Check for CI integration: evals run in the PR pipeline, the results are posted as a PR comment, and regressions are called out specifically
• Verify the suite runs periodically even without code changes, to catch provider-side model drift, because silent upstream changes are otherwise invisible until users complain

Output Validation Strategy Checklist

• Verify structured outputs are validated against a schema — Zod, JSON Schema, Pydantic — at both test time and runtime, because model output is untrusted input and schema validation is the first line of defense
• Check for exact-match validation where appropriate (function call arguments, classification labels), because these have a clear right answer and exact-match is the simplest test
• Verify property-based checks for semi-structured outputs: required sections, format regex compliance, length bounds, because property checks catch "the output is missing X" regressions that exact-match misses
• Check for LLM-as-judge evaluation on free-form outputs where quality is subjective, and verify the judge prompt is calibrated against human ratings on a held-out set, because an uncalibrated LLM judge is noise
• Verify judge prompts are versioned and stable — changing the judge prompt invalidates prior results, so judge prompts should change only deliberately
• Check for multi-dimensional scoring where relevant: correctness, helpfulness, tone, safety, completeness scored separately, because a single "is it good" score hides which dimension regressed

Regression Detection & Comparison Checklist

• Verify the evaluation produces a clear pass/fail signal plus a change vs the previous baseline, because "67% pass rate" without context of "prior baseline was 94%" misses the regression entirely
• Check whether the suite identifies specific failing cases, not just aggregate pass rates, because "3 tests failed" is actionable and "92% pass rate" is not
• Verify there's a diff between current and previous outputs for each test case, so a reviewer can see what changed, because "the answer changed" matters less than "the answer now omits the key field"
• Check for statistical significance in any A/B comparison of prompts or models — a 2-point quality shift on 30 cases is noise; on 3000 cases it may be real
• Verify regressions are classified by severity: schema failures are Critical, rubric-score drops by >15% are High, minor stylistic changes are Informational, because treating every change as equivalent trains the team to ignore the suite
• Check for regression categorization: "previously passing now failing," "previously failing now passing," "new test," "flaky" — because flaky tests need their own handling

Cost & Latency Baselines Checklist

• Verify token counts are recorded for every evaluation run (input + output tokens) and tracked over time, because a prompt change that doubles output length doubles cost silently
• Check for latency baselines (p50, p95) per feature and alerts when latency degrades, because model changes and prompt changes can shift latency dramatically
• Verify cost-per-request is calculated and monitored, because AI features are usage-based cost lines that grow with traffic and can surprise teams
• Check for evaluation of cheaper/faster alternatives: does the team periodically test whether a smaller model or different provider meets the quality bar at lower cost, because sticking with the initial model forever over-indexes on historical choice
• Verify cost and latency are measured on real production traffic, not just test traffic, because test inputs don't reflect the real distribution of cost

Hallucination, Safety & Refusal Checklist

• For features that produce factual content (summaries, resume tailoring, any grounded generation), verify there's a hallucination check: does the output contain claims not supported by the input, because hallucinations are the most damaging class of LLM regression
• Check for refusal-rate tracking: how often does the model refuse to answer benign queries, because safety-tuned models can over-refuse in ways that break legitimate features
• Verify prompt-injection tests are in the suite — inputs containing "ignore previous instructions," attempts to exfiltrate the system prompt, attempts to produce off-topic content — because prompt injection is a production reality for user-facing LLM features
• Check for output filters: PII leakage in outputs, explicit content, off-brand tone, and verify these are tested, because content-policy failures can be public embarrassments
• Verify adversarial inputs are in the test suite (nonsense input, extremely long input, looping input, competitor-mention input), because these discover failure modes that happy-path tests miss

Shadow Testing & Gradual Rollout Checklist

• Verify significant prompt or model changes ship behind shadow testing — the new version runs alongside the old on real traffic, outputs are compared, users see the old version — because pre-prod tests can't cover the full production distribution
• Check whether A/B testing infrastructure exists for LLM changes: a fraction of users get the new version, metrics are tracked, statistical significance is measured before rollout, because direct cutover is higher-risk than gradual
• Verify rollback mechanisms exist for LLM changes: a prompt or model version that produces regression in production can be reverted within minutes, because LLM incidents are often revealed after rollout and fast rollback limits damage
• Check that user feedback (thumbs up/down, regenerate button usage, explicit complaints) is captured and correlated with recent prompt/model changes, because user signals often catch issues the test suite didn't
• Verify that there's a post-change monitoring window — 24–72 hours after a prompt or model change, metrics are watched for drift — because regressions don't always surface immediately

Change Management & Versioning Checklist

• Verify prompts are version-controlled with the application code, because prompts are code and changes need review, diff visibility, and revert capability
• Check for a prompt changelog or structured changelog entry for every significant prompt change, with rationale, evaluation results, and rollout plan
• Verify model version pinning — not "claude-opus" but "claude-opus-4-6" — because unpinned models silently update and cause unexplained drift
• Check for a version-upgrade playbook: how the team evaluates a new model version, what the approval criteria are, how it rolls out
• Verify few-shot examples, system prompts, and context-assembly logic are version-controlled together, because a change to any of these is equivalent to a prompt change
• Check whether retrieval/RAG pipelines (if used) have their own version-controlled index, embedding model, retrieval strategy, because RAG changes are LLM changes

Production Feedback Loop Checklist

• Verify user feedback (thumbs up/down, explicit corrections, regeneration requests) is logged alongside the specific prompt version, model version, and input, because user signal is the ground-truth for whether the AI feature is working
• Check for periodic sampling of production outputs for human review, because automated tests have blind spots and human review catches issues ML metrics miss
• Verify flagged outputs (user complaints, low-rated responses) are funneled back into the golden suite as new test cases, because production failures become regression tests
• Check for a regular AI-quality review — weekly or monthly — where the team looks at sampled outputs, user feedback, and eval metrics together, because without a rhythm the eval suite decays
• Verify there's someone accountable for AI feature quality — a product manager, an engineer, a dedicated eval owner — because shared responsibility for AI quality tends toward nobody owning it

---

Calibration

Scale severity to feature importance and user visibility. A customer-facing LLM feature that produces revenue (resume tailoring on a paid plan) needs comprehensive evaluation; a one-off internal summarization tool does not. Financial or decision-critical AI (medical, legal, compliance) requires the most rigorous evaluation — regressions can cause real harm. Free-tier or supplementary AI (nice-to-have) can tolerate less rigor but still benefits from structured output validation. Early-stage features are fine with lightweight suites; mature features should have proportionally comprehensive eval. Over-engineering evaluation before the feature finds product-market fit is wasted work; under-engineering it after success is shipping roulette.

• Confidence ratings: Mark each finding as Confirmed (verified in the codebase — e.g., "no evaluation suite in /evals directory," "prompt version is not tied to git SHA"), Likely (pattern suggests the issue — e.g., "prompt changes over the last year have no changelog and no eval results in PRs"), or Speculative (potential gap needing interview or process check).
• Anti-hallucination guard: If evaluation is rigorous, regressions are caught, and changes are versioned, say so. Not every feature needs a 500-case suite — calibrate to actual production risk. Early MVPs may legitimately ship without formal eval while iterating quickly.

Output Format

Start with a 3-5 line executive summary: LLM features inventoried, current evaluation state, highest-risk unmonitored feature, and highest-impact recommendation.

1. LLM Feature Inventory — Table: Feature | Output Shape | Consumer | Current Eval | Gate on Change? | Last Evaluated
2. Evaluation Suite Coverage — For each feature: suite size, input sources, case categories (happy/edge/adversarial/regression), output validation method
3. Change-Management Maturity — Prompt versioning, model pinning, changelog completeness, CI gating
4. Production Feedback Loop — User feedback capture, output sampling, feedback-to-test promotion, quality review cadence
5. Cost & Latency Tracking — Per-feature baselines, drift alerts, cost projection
6. Critical Gaps — Features without eval, features with wrong eval shape, evaluation paths that don't actually gate deploys
7. Regression-Risk Assessment — Upcoming changes (model upgrades, prompt rewrites) and their current test coverage
8. Detailed Findings — For each High/Critical: feature, risk, concrete implementation plan (suite structure, validation approach, gating mechanism)
9. Suite-Building Plan — For features missing eval, a concrete starter suite: case types to include, validation strategy, tooling suggestion (Promptfoo, LangSmith, DeepEval, custom harness)
10. Positive Findings — Evaluation practices already working well that should be preserved and replicated to other features