snapeval

Harness-agnostic eval runner for agentskills.io skills.

snapeval runs every eval case with and without your skill, grades assertions, and computes a benchmark delta — so you can see exactly what value your skill adds.

snapeval — greeter
Baseline = without SKILL.md (raw AI response)
────────────────────────────────────────────────────────────
  #1 formal greeting for Eleanor
    Skill: 100% | Baseline: 33% | 5.2s
  #2 casual greeting for Marcus
    Skill: 100% ↑ was 67% | Baseline: 67% | 2.7s
  #3 pirate greeting for Zoe
    Skill: 100% | Baseline: 67% | 2.5s
────────────────────────────────────────────────────────────
Summary:
  Skill pass rate:    100.0%
  Baseline pass rate: 55.6%
  Improvement:        +44.4%

How it works

1. You write a SKILL.md and an evals.json with test cases and assertions
2. snapeval runs each eval twice — once with your skill loaded, once without (baseline)
3. Assertions are graded by an LLM judge (semantic) and/or shell scripts (deterministic)
4. A benchmark shows where your skill adds value vs. where the raw AI already handles it

Quick start

As a Copilot plugin

copilot plugin install matantsach/snapeval

Then in Copilot CLI, just say evaluate my skill — the snapeval skill handles the rest.

Standalone CLI

git clone https://github.com/matantsach/snapeval.git
cd snapeval && npm install
npx tsx bin/snapeval.ts eval <skill-dir>

Eval format

my-skill/
├── SKILL.md
└── evals/
    ├── evals.json
    └── scripts/         ← optional deterministic checks
        └── validate.sh

evals.json:

{
  "skill_name": "greeter",
  "evals": [
    {
      "id": 1,
      "label": "formal greeting for Eleanor",
      "prompt": "Can you give me a formal greeting for Eleanor?",
      "expected_output": "Returns the formal greeting addressed to Eleanor.",
      "assertions": [
        "Output contains the name Eleanor",
        "Output uses a formal tone",
        "script:validate.sh"
      ]
    }
  ]
}

Field	Required	Description
id	yes	Unique numeric identifier
prompt	yes	The user prompt sent to the harness
expected_output	yes	Human description of the expected behavior
label	no	Human-readable name shown in terminal output
slug	no	Filesystem-safe name for the eval directory
assertions	no	List of assertions to grade (LLM semantic or script: prefixed)
files	no	Input files to attach to the prompt

Assertions

Semantic — graded by an LLM. Write specific, verifiable statements:

"Output contains a YAML block with an 'id' field for each issue"
"Response declines because the pipeline already has unclaimed issues"

Script — prefix with script:. Scripts live in evals/scripts/, receive the output directory as $1, and pass on exit code 0:

"script:validate-json-structure.sh"

CLI reference

eval

Run evals, grade assertions, compute benchmark.

npx snapeval eval [skill-dir] [options]

Flag	Description	Default
--harness <name>	Harness adapter	copilot-sdk
--inference <name>	Inference adapter for grading	auto
--workspace <path>	Output directory	../{skill_name}-workspace
--runs <n>	Harness invocations per eval for statistical averaging	1
--concurrency <n>	Parallel eval cases (1-10)	1
--only <ids>	Run specific eval IDs (e.g. --only 1,3,5)	all
--threshold <rate>	Minimum pass rate 0-1 for exit code 0	none
--old-skill <path>	Compare against old skill version	none
--feedback	Write feedback.json template for human review	off

Exit codes

Code	Meaning
0	Success
1	Threshold not met (eval ran but pass rate below --threshold)
2	Config/input error (bad JSON, missing fields, invalid flags)
3	File not found (missing skill dir, evals.json, or script)
4	Runtime error (harness failure, grading failure, timeout)

Output artifacts

Each run creates an iteration directory:

workspace/
└── iteration-1/
    ├── benchmark.json       ← aggregate stats with delta
    ├── SKILL.md.snapshot    ← copy of skill used
    └── eval-{slug}/
        ├── with_skill/
        │   ├── outputs/output.txt
        │   ├── timing.json
        │   ├── grading.json
        │   └── transcript.log
        └── without_skill/
            ├── outputs/output.txt
            ├── timing.json
            └── grading.json

benchmark.json includes metadata: eval_count, eval_ids, skill_name, runs_per_eval, timestamp.

CI integration

name: Skill Evaluation
on: [pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx snapeval eval skills/my-skill --threshold 0.8 --runs 3

Exit code 1 when pass rate falls below threshold — blocks the PR.

Configuration

Create snapeval.config.json in your skill or project root:

{
  "harness": "copilot-sdk",
  "inference": "auto",
  "workspace": "../{skill_name}-workspace",
  "runs": 1,
  "concurrency": 1
}

Resolution order: defaults → project config → skill-dir config → CLI flags.

Harness adapters

Adapter	Description	Default
copilot-sdk	Programmatic via @github/copilot-sdk with native skill loading	yes
copilot-cli	Shells out to copilot CLI binary	no

The SDK harness loads skills natively via skillDirectories, captures full transcripts, and extracts real token counts from assistant.usage events.

Inference adapters

Adapter	Description
auto	Uses @github/copilot-sdk by default, falls back to GitHub Models API
copilot-sdk	@github/copilot-sdk programmatic
github-models	GitHub Models API (requires GITHUB_TOKEN)

Contributing

See CONTRIBUTING.md.

License

MIT