Skip to content

How eval testing works & keeping it current

This document explains the conceptual model behind the agent evals and the discipline for keeping the corpus honest over time. For the architecture, see eval-system.md. For the operational run procedure, see eval-running-guide.md.

The pieces

Piece Where Role
Fixtures evals/fixtures/ Input code (deliberately good or bad) the agents review.
Expectations evals/expected/*.json The contract: what a correct verdict looks like per fixture/agent.
Grader scripts/eval_grade.py Deterministic, model-free: compares recorded actuals to expectations.
Regression diff scripts/compare_eval_results.py Diffs two --actuals result files against evals/expected/*.json, gating on true/false-positive-proxy count regression. It sits at the repo root next to eval_grade.py (ADR 0032 category 2, monorepo-dev-only). The plugin does not ship it. eval_ablation.py does ship, but only for its unrelated generic --find-latest reader mode.
Variance scripts/eval_variance.py Aggregates K trials → pass@k, flap rate, quarantine.
Trend .claude/metrics/eval-variance.jsonl Append-only stability history (metrics only).
Semver contract scripts/eval_semver_classify.sh The eval corpus IS the version contract (#101).
CI gates .github/workflows/agent-eval.yml Structural check always; live regression when keyed.

How grading works (the rules)

eval_grade.py grade_agent checks each expectation field. An empty failure list means PASS:

  • expectedStatus — pass or fail. The value must match the agent's status.
  • issueCount: {min, max} — the number of reported issues must fall in range.
  • severities: {error: {min,max}, ...} — the count per severity must fall in range.
  • mustMention: [...] — every keyword must appear (case-insensitive substring) in the issue messages and summary. All-of.
  • mustNotMention: [...] — no keyword may appear. All-of.

Grading is intentionally dumb (no judgment). This lets it run as a CI gate and keeps variance reproducible.

The calibration trap (learn this — #198)

Matching is plain substring, so expectations drift out of sync with how agents phrase correct verdicts. Two failure modes exist. Both are fixture bugs, not agent bugs:

  1. mustNotMention is negation-blind. A clean-pass fixture forbidding "hardcoded" fails when the agent correctly says "no hardcoded secrets". Fix: drop mustNotMention on clean-pass fixtures. expectedStatus:pass + issueCount (+ error 0-0) already encode "found clean."
  2. mustMention too strict. Requiring the exact token "SRP" fails when the agent says "responsibilities" or "divergent change". Fix: use stems ("responsibilit") and the vocabulary agents emit. Avoid all-of lists of rare tokens.

When a correct agent verdict fails grading, suspect the fixture first. Verify the fix against the agent's real output, not a hand-typed approximation.

Keeping the corpus current

Adding a fixture

  1. Add the input file under evals/fixtures/.
  2. Add evals/expected/<stem>.json with fixture, applicableAgents, and the per-agent expectation. Prefer expectedStatus + issueCount. Add mustMention stems only when the agent must name a specific concept.
  3. Run python3 scripts/eval_grade.py --check-corpus. Every expectation must be schema-valid and pair with a fixture. CI runs this check.

Changing an expectation = a version bump (#101)

The eval corpus is the semver contract. eval_semver_classify.sh (pre-push and CI) enforces the contract:

  • GREEN-preserving change → patch.
  • Adds expectations → minor (feat:).
  • Edits existing expectations → minor/major (feat: / feat!:). An edit changes the agents' observable contract.

The classifier rejects a fix: commit that edits an expectation. Use the bump the classifier names.

Watching stability over time

  • Run per-agent variance batches periodically (see the running guide).
  • The trend in .claude/metrics/eval-variance.jsonl accumulates pass@k and flap rate.
  • Flaky pairs (0 < pass@k < 1) go on the quarantine list. They inform the #99 gate but must not hard-block it.
  • A persistently flaky fixture is either borderline (tighten it) or genuinely non-deterministic for that agent.
  • Saturated agents (identical grades for many runs) may have expectations too loose to detect regressions. Consider tightening the ranges.

Cardinal rules

  1. Faithful actuals. The grader sees what you record. Abbreviating issue messages drops mustMention keywords and fabricates flaps.
  2. Neutral dispatch. Never leak status or severity examples into the agent's prompt (see the running guide).
  3. Fixtures are the contract. Keep them honest. A stable-fail on correct output is a corpus bug to fix, not noise to ignore.
  4. Metrics only. Digests, the trend, and reports carry counts, ratios, and names. They never carry prompt or code content.