Evals

Evaluates AI agents — their transcripts, tool-call sequences, and multi-turn conversations, not just single outputs. Three grader types cover it: code-based for deterministic checks, model-based for nuanced quality rubrics, and human for the gold standard. Scores with pass@k (capability) and pass^k (consistency). Splits evals into capability suites (~70% target) and regression suites (~99% target), and plugs into Algorithm ISC rows as a verification method.

You can't tell if an agent got better or worse by eyeballing a few runs. A change that looks fine in one transcript may quietly regress on the next prompt, and a single run gives no statistical signal. Judging only the final output also misses how the agent got there — wrong tools, wrong order, lucky guess. This skill measures the whole workflow across repeated trials, so improvements and backsliding both show up as numbers you can gate on.

Agent evaluation system based on Anthropic's "Demystifying Evals for AI Agents" (Jan 2026). It evaluates agent workflows (transcripts, tool calls, multi-turn conversations), not just single outputs.

• "run evals", "test this agent", "evaluate", "check quality", "benchmark"
• "regression test", "capability test"
• "run scenario", "multi-turn eval", "simulated user test"
• "create scenario", "simulate conversation"
• Compare agent behaviors across changes
• Validate agent workflows before deployment
• Verify ALGORITHM ISC rows
• Create new evaluation tasks from failures

Three Grader Types

Evaluation Types

Key Metrics

• pass@k: Probability of at least 1 success in k trials (measures capability)
• pass^k: Probability all k trials succeed (measures consistency/reliability)

CLI Commands

# Run an eval suite
bun run ${CLAUDE_SKILL_DIR}/Tools/AlgorithmBridge.ts -s <suite>

# Log a failure for later conversion
bun run ${CLAUDE_SKILL_DIR}/Tools/FailureToTask.ts log "description" -c category -s severity

# Convert failures to test tasks
bun run ${CLAUDE_SKILL_DIR}/Tools/FailureToTask.ts convert-all

# Manage suites
bun run ${CLAUDE_SKILL_DIR}/Tools/SuiteManager.ts create <name> -t capability -d "description"
bun run ${CLAUDE_SKILL_DIR}/Tools/SuiteManager.ts list
bun run ${CLAUDE_SKILL_DIR}/Tools/SuiteManager.ts check-saturation <name>
bun run ${CLAUDE_SKILL_DIR}/Tools/SuiteManager.ts graduate <name>

ALGORITHM Integration

Evals is a verification method for THE ALGORITHM ISC rows:

# Run eval and update ISC row
bun run ${CLAUDE_SKILL_DIR}/Tools/AlgorithmBridge.ts -s regression-core -r 3 -u

ISC rows can specify eval verification:

| # | What Ideal Looks Like | Verify |
|---|----------------------|--------|
| 1 | Auth bypass fixed | eval:auth-security |
| 2 | Tests all pass | eval:regression |

Code-Based (Fast, Deterministic)

Model-Based (Nuanced)

Pre-configured grader stacks for common agent types:

See Data/DomainPatterns.yaml for full configurations.

task:
  id: "fix-auth-bypass_1"
  description: "Fix authentication bypass when password is empty"
  type: regression  # or capability
  domain: coding

  graders:
    - type: binary_tests
      required: [test_empty_pw.py]
      weight: 0.30

    - type: tool_calls
      weight: 0.20
      params:
        sequence: [read_file, edit_file, run_tests]

    - type: llm_rubric
      weight: 0.50
      params:
        rubric: prompts/security_review.md

  trials: 3
  pass_threshold: 0.75

1. Start with 20-50 real failures - Don't overthink, capture what actually broke
2. Unambiguous tasks - Two experts should reach identical verdicts
3. Balanced problem sets - Test both "should do" AND "should NOT do"
4. Grade outputs, not paths - Don't penalize valid creative solutions
5. Calibrate LLM judges - Against human expert judgment
6. Check transcripts regularly - Verify graders work correctly
7. Monitor saturation - Graduate to regression when hitting 95%+
8. Build infrastructure early - Evals shape how quickly you can adopt new models

• ALGORITHM: Evals is a verification method
• Science: Evals implements scientific method
• Browser: For visual verification graders

• Choose the right grader type: Code-based for deterministic checks (fast, cheap). Model-based for nuanced quality (flexible, expensive). Human for calibration (gold standard, slow).
• pass@k scoring requires multiple runs. A single run doesn't give statistical significance. Default to pass@3 minimum.
• Transcript capture must be enabled BEFORE the test run. Can't retroactively capture transcripts.
• Eval results go to the current work directory — not a global location. Tie evals to the work item.
• Don't evaluate skills with trivial prompts. Simple one-liners may not trigger skill usage. Test prompts must be substantive.

Example 1: Compare two prompts

User: "evaluate which prompt produces better summaries"
→ Creates eval suite with 3+ test cases
→ Runs both prompts against test cases
→ Model-based grader scores quality
→ Reports pass@k and comparative analysis

Example 2: Regression test a skill change

User: "run evals on the Research skill after the update"
→ Uses existing test fixtures for Research
→ Before/after comparison
→ Reports any quality regressions