Cross-harness benchmarking - generate instructions for Codex/Gemini/OpenCode, import results, and compare across harnesses

ARC-AGI Cross-Harness

You are managing cross-harness benchmarking for ARC-AGI. This skill lets users generate benchmark instructions for other CLI harnesses (Codex, Gemini CLI, OpenCode), import their results, and compare them against Claude Code runs.

Sub-commands

This skill has three sub-commands, parsed from user arguments after /arc-cross-harness:

Sub-command	Syntax	Purpose
`generate`	`/arc-cross-harness generate <HARNESS> [--ref <RUN_ID>] [--seed <N>] [--games <LIST>]`	Generate instruction document + helper scripts for a target harness
`import`	`/arc-cross-harness import <HARNESS> <PATH>`	Import and normalize a result file from another harness
`compare`	`/arc-cross-harness compare [run-ids...] [--all]`	Compare runs across harnesses (delegates to `/arc-compare`)

If no sub-command is provided, display the table above as usage help and stop.

Supported harness values: codex, gemini, opencode.

Step 1: Pre-flight Checks

1a: Detect Virtual Environment

Determine the venv paths. Do NOT try to source activate scripts.

if [ -f ".arc-agi-venv/bin/python" ]; then
  VENV_PYTHON=".arc-agi-venv/bin/python"
elif [ -f ".arc-agi-venv/Scripts/python.exe" ]; then
  VENV_PYTHON=".arc-agi-venv/Scripts/python.exe"
else
  echo "ERROR: Virtual environment not found. Run /arc-setup first."
  exit 1
fi
echo "VENV_PYTHON=$VENV_PYTHON"

Use $VENV_PYTHON for ALL Python commands below. Store the resolved path.

1b: Read Configuration

$VENV_PYTHON -c "
import json, sys
try:
    with open('.arc-agi-benchmarks/config.json') as f:
        cfg = json.load(f)
    print(json.dumps(cfg, indent=2))
except FileNotFoundError:
    print('ERROR: config.json not found. Run /arc-setup first.', file=sys.stderr)
    sys.exit(1)
"

Extract from the config:

environments_dir (default: ./environment_files)
default_seed (default: 0)
default_max_steps (default: 500)
default_max_resets (default: 10)

1c: Verify arc-agi is Available

$VENV_PYTHON -c "
from arc_agi import Arcade, OperationMode
from arcengine import GameAction, GameState
print('arc-agi: OK')
"

If this fails, tell the user to run /arc-setup first and stop.

1d: Parse Sub-command and Arguments

Parse the first argument as the sub-command (generate, import, or compare). If not recognized, display usage help and stop.

Then parse remaining arguments based on the sub-command:

For generate:

Argument	Format	Required	Default	Example
Harness name	`codex`, `gemini`, `opencode`	yes	--	`codex`
`--ref <RUN_ID>`	UUID or `latest`	no	(none)	`--ref latest`
`--seed <N>`	integer	no	from config	`--seed 42`
`--games <LIST>`	comma-separated or `all`	no	`all`	`--games bt11,ls20`

For import:

Argument	Format	Required	Example
Harness name	`codex`, `gemini`, `opencode`	yes	`codex`
Result file path	file path	yes	`./codex-results.json`

For compare:

Argument	Format	Required	Example
Run IDs	space-separated UUIDs	Required unless `--all`	`<run-id-1> <run-id-2>`
`--all`	flag	no	auto-select latest run from each harness (still requires 2+ runs to exist)

Validate that harness names are one of: codex, gemini, opencode. If invalid, report:

Invalid harness <NAME>. Supported harnesses: codex, gemini, opencode

Step 2: Execute Sub-command

Branch based on the parsed sub-command. Follow the corresponding section below.

Sub-command: Generate (`/arc-cross-harness generate <HARNESS>`)

G1: Resolve Game Set and Parameters

If --ref is provided, resolve the reference run:

$VENV_PYTHON -c "
import json, os, sys

ref_arg = '<REF_ARG>'  # 'latest' or a UUID

runs_dir = '.arc-agi-benchmarks/runs'
if ref_arg == 'latest':
    completed = []
    if os.path.isdir(runs_dir):
        for d in os.listdir(runs_dir):
            meta_path = os.path.join(runs_dir, d, 'run-meta.json')
            if os.path.isfile(meta_path):
                with open(meta_path) as f:
                    meta = json.load(f)
                if meta.get('status') == 'completed':
                    completed.append(meta)
    completed.sort(key=lambda m: m.get('timestamp', ''), reverse=True)
    if not completed:
        print(json.dumps({'error': 'No completed runs found. Run /arc-benchmark first.'}))
        sys.exit(1)
    ref_meta = completed[0]
else:
    meta_path = os.path.join(runs_dir, ref_arg, 'run-meta.json')
    if not os.path.isfile(meta_path):
        print(json.dumps({'error': f'Reference run {ref_arg} not found.'}))
        sys.exit(1)
    with open(meta_path) as f:
        ref_meta = json.load(f)

print(json.dumps({
    'run_id': ref_meta.get('run_id'),
    'game_ids': ref_meta.get('game_ids', []),
    'seed': ref_meta.get('seed', 0),
    'max_steps': ref_meta.get('max_steps', 500),
    'max_resets': ref_meta.get('max_resets', 10),
    'game_set': ref_meta.get('game_set', 'all')
}))
"

IMPORTANT: Replace <REF_ARG> with the actual --ref value provided by the user.

If no --ref is provided, resolve the game list from available environments:

$VENV_PYTHON -c "
import json
with open('.arc-agi-benchmarks/config.json') as f:
    cfg = json.load(f)
env_dir = cfg.get('environments_dir', './environment_files')
seed = <SEED>
games_filter = '<GAMES_FILTER>'  # 'all' or comma-separated IDs

from arc_agi import Arcade, OperationMode
op_mode = OperationMode(cfg.get('operation_mode', 'normal'))
arc = Arcade(operation_mode=op_mode, environments_dir=env_dir)
envs = arc.get_environments()

all_ids = []
for env in envs:
    gid = getattr(env, 'game_id', getattr(env, 'id', str(env)))
    all_ids.append(gid)

if games_filter == 'all':
    game_ids = all_ids
else:
    requested = [g.strip() for g in games_filter.split(',')]
    game_ids = [g for g in requested if g in all_ids]
    missing = [g for g in requested if g not in all_ids]
    if missing:
        import sys
        print(f'WARNING: Games not found: {missing}', file=sys.stderr)

print(json.dumps({
    'game_ids': game_ids,
    'seed': seed,
    'max_steps': cfg.get('default_max_steps', 500),
    'max_resets': cfg.get('default_max_resets', 10)
}))
"

IMPORTANT: Replace <SEED> with the seed integer (from --seed or config default) and <GAMES_FILTER> with the --games value (default all).

Store the resolved game_ids, seed, max_steps, max_resets, and optionally the reference run_id.

G2: Generate Instruction Document, Game Driver, and Result Collector

Generate all three files in a single Python script:

$VENV_PYTHON -c "
import json, os, textwrap
from datetime import datetime, timezone

harness = '<HARNESS>'
game_ids = <GAME_IDS_JSON>
seed = <SEED>
max_steps = <MAX_STEPS>
max_resets = <MAX_RESETS>
ref_run_id = '<REF_RUN_ID>'  # or 'none'

with open('.arc-agi-benchmarks/config.json') as f:
    cfg = json.load(f)
env_dir = cfg.get('environments_dir', './environment_files')

out_dir = f'.arc-agi-benchmarks/cross-harness/{harness}'
os.makedirs(f'{out_dir}/runs', exist_ok=True)

# ===== 1. GAME DRIVER SCRIPT =====
game_driver = '''#!/usr/bin/env python3
\"\"\"
ARC-AGI Game Driver for cross-harness benchmarking.
Provides a stdin/stdout interface for any harness to play ARC-AGI games.

Usage:
  python game_driver.py <game_id> <seed> [--env-dir <path>]

Protocol:
  1. On start, prints an observation JSON to stdout.
  2. Reads action JSON from stdin (one line).
  3. Executes action, prints next observation JSON to stdout.
  4. Repeat until state is WIN or GAME_OVER.
  5. On GAME_OVER, send {\"command\": \"reset\"} to reset (up to max_resets).
  6. Send {\"command\": \"quit\"} to end the session.

Observation JSON:
  {
    \"state\": \"PLAYING|WIN|GAME_OVER\",
    \"frame\": [[[int, ...], ...], ...],
    \"levels_completed\": int,
    \"total_levels\": int,
    \"baseline_actions\": [int, ...],
    \"available_actions\": [{\"name\": str, \"is_complex\": bool}, ...],
    \"step\": int,
    \"resets\": int
  }

Action JSON:
  {\"command\": \"step\", \"action\": \"ACTION1\", \"data\": {\"x\": 0, \"y\": 0}, \"reasoning\": \"...\"}
  {\"command\": \"reset\"}
  {\"command\": \"quit\"}
\"\"\"
import json, sys, argparse

def main():
    parser = argparse.ArgumentParser(description='ARC-AGI Game Driver')
    parser.add_argument('game_id', help='Game ID to play')
    parser.add_argument('seed', type=int, help='Random seed')
    parser.add_argument('--env-dir', default='./environment_files', help='Path to environment files')
    parser.add_argument('--max-steps', type=int, default=''' + str(max_steps) + ''', help='Max steps per game')
    parser.add_argument('--max-resets', type=int, default=''' + str(max_resets) + ''', help='Max resets per game')
    args = parser.parse_args()

    from arc_agi import Arcade, OperationMode
    from arcengine import GameAction

    arc = Arcade(operation_mode=OperationMode.NORMAL, environments_dir=args.env_dir)

    # Query environment metadata for baseline_actions and total_levels
    env_meta_baseline_actions = []
    env_meta_total_levels = 5
    try:
        envs = arc.get_environments()
        for e in envs:
            eid = getattr(e, 'game_id', getattr(e, 'id', str(e)))
            if eid == args.game_id:
                env_meta_baseline_actions = getattr(e, 'baseline_actions', [])
                if hasattr(env_meta_baseline_actions, 'tolist'):
                    env_meta_baseline_actions = env_meta_baseline_actions.tolist()
                env_meta_total_levels = getattr(e, 'number_of_levels', getattr(e, 'total_levels', 5))
                break
    except Exception:
        pass

    if env_meta_total_levels == 5 and not env_meta_baseline_actions:
        print(f'WARNING: Could not determine total_levels from environment metadata for {args.game_id}; falling back to total_levels=5', file=sys.stderr, flush=True)

    env = arc.make(args.game_id, seed=args.seed, save_recording=False, render_mode=None)
    # NOTE: In arc-agi, env.observation_space returns a FrameDataRaw object
    # (the current observation), not a Gym-style Space descriptor.
    obs = env.observation_space

    step_count = 0
    reset_count = 0

    def obs_to_dict(obs):
        frame = obs.frame
        if hasattr(frame, 'tolist'):
            frame = frame.tolist()
        actions = []
        for a in env.action_space:
            name = a.name if hasattr(a, 'name') else str(a)
            is_complex = a.is_complex() if hasattr(a, 'is_complex') else False
            actions.append({'name': name, 'is_complex': is_complex})
        return {
            'state': obs.state.name if hasattr(obs.state, 'name') else str(obs.state),
            'frame': frame,
            'levels_completed': getattr(obs, 'levels_completed', 0),
            'total_levels': env_meta_total_levels,
            'baseline_actions': env_meta_baseline_actions,
            'available_actions': actions,
            'step': step_count,
            'resets': reset_count
        }

    # Print initial observation
    print(json.dumps(obs_to_dict(obs)), flush=True)

    for line in sys.stdin:
        line = line.strip()
        if not line:
            continue
        try:
            request = json.loads(line)
        except json.JSONDecodeError:
            print(json.dumps({'error': 'Invalid JSON'}), flush=True)
            continue

        cmd = request.get('command', 'step')

        if cmd == 'quit':
            break
        elif cmd == 'reset':
            if reset_count >= args.max_resets:
                print(json.dumps({'error': 'Max resets exceeded'}), flush=True)
                continue
            obs = env.reset()
            if obs is None:
                print(json.dumps({'error': 'env.reset() returned None'}), flush=True)
                continue
            reset_count += 1
            print(json.dumps(obs_to_dict(obs)), flush=True)
        elif cmd == 'step':
            if step_count >= args.max_steps:
                print(json.dumps({'error': 'Max steps exceeded'}), flush=True)
                continue
            action_name = request.get('action', '')
            try:
                action = GameAction[action_name]
            except KeyError:
                print(json.dumps({'error': f'Unknown action: {action_name}'}), flush=True)
                continue
            step_kwargs = {'action': action}
            if request.get('data'):
                step_kwargs['data'] = request['data']
            if request.get('reasoning'):
                step_kwargs['reasoning'] = {'thought': request['reasoning']}
            obs = env.step(**step_kwargs)
            if obs is None:
                print(json.dumps({'error': 'env.step() returned None'}), flush=True)
                continue
            step_count += 1
            print(json.dumps(obs_to_dict(obs)), flush=True)
        else:
            print(json.dumps({'error': f'Unknown command: {cmd}'}), flush=True)

if __name__ == '__main__':
    main()
'''

with open(f'{out_dir}/game_driver.py', 'w', newline='\\n') as f:
    f.write(game_driver)

# ===== 2. RESULT COLLECTOR SCRIPT =====
collect_results = '''#!/usr/bin/env python3
\"\"\"
ARC-AGI Result Collector for cross-harness benchmarking.
Reads session log files and produces the cross-harness result JSON.

Usage:
  python collect_results.py --sessions-dir <dir> --output <path> --harness <name> [--seed <N>]

Session log format (one file per game, named <game_id>.jsonl):
  Each line is a JSON object with:
    {\"type\": \"observation\", \"data\": {...}}  -- observation from game_driver
    {\"type\": \"action\", \"data\": {...}}       -- action sent to game_driver

If you are using the game_driver.py interactively, you can capture the session
by logging all stdin/stdout exchanges to a JSONL file.

Alternatively, pass --results-file <path> to directly provide a pre-built result JSON
for validation only (the script will validate and pretty-print it).
\"\"\"
import json, sys, argparse, os
from datetime import datetime, timezone

def compute_level_score(baseline_actions, actions_taken, completed):
    if not completed:
        return 0.0
    if actions_taken == 0:
        return 0.0
    return min((baseline_actions / actions_taken) ** 2, 1.0)

def compute_game_score(level_scores):
    if not level_scores:
        return 0.0
    n = len(level_scores)
    weighted = sum(level_scores[i] * (i + 1) for i in range(n))
    total_weight = sum(range(1, n + 1))
    return weighted / total_weight if total_weight > 0 else 0.0

def main():
    parser = argparse.ArgumentParser(description='ARC-AGI Result Collector')
    parser.add_argument('--sessions-dir', help='Directory containing session JSONL files')
    parser.add_argument('--results-file', help='Pre-built result JSON to validate')
    parser.add_argument('--output', default='result.json', help='Output file path')
    parser.add_argument('--harness', default=''' + repr(harness) + ''', help='Harness name')
    parser.add_argument('--seed', type=int, default=''' + str(seed) + ''', help='Seed used')
    parser.add_argument('--model', default='unknown', help='Model identifier')
    parser.add_argument('--version', default='unknown', help='Harness version')
    parser.add_argument('--notes', default='', help='Additional notes')
    args = parser.parse_args()

    if args.results_file:
        with open(args.results_file) as f:
            result = json.load(f)
        print(json.dumps(result, indent=2))
        return

    if not args.sessions_dir:
        print('ERROR: --sessions-dir or --results-file required', file=sys.stderr)
        sys.exit(1)

    games = []
    for fname in sorted(os.listdir(args.sessions_dir)):
        if not fname.endswith('.jsonl'):
            continue
        game_id = fname.replace('.jsonl', '')
        observations = []
        actions = []
        with open(os.path.join(args.sessions_dir, fname)) as f:
            for line in f:
                line = line.strip()
                if not line:
                    continue
                entry = json.loads(line)
                if entry.get('type') == 'observation':
                    observations.append(entry['data'])
                elif entry.get('type') == 'action':
                    actions.append(entry['data'])

        if not observations:
            games.append({
                'game_id': game_id,
                'state': 'NOT_PLAYED',
                'levels_completed': 0,
                'total_levels': 5,
                'total_actions': 0,
                'total_resets': 0,
                'levels': []
            })
            continue

        last_obs = observations[-1]
        first_obs = observations[0]
        state = last_obs.get('state', 'GAME_OVER')
        levels_completed = last_obs.get('levels_completed', 0)
        total_actions = sum(1 for a in actions if a.get('command') == 'step')
        total_resets = sum(1 for a in actions if a.get('command') == 'reset')

        # Read baseline_actions and total_levels from observation data (set by game_driver.py)
        env_baseline_actions = first_obs.get('baseline_actions', [])
        env_total_levels = first_obs.get('total_levels', 5)

        # Build level info by iterating action entries with command='step'.
        # We pair each step action with the observation that follows it
        # (observations[action_index + 1]) to detect level transitions.
        # This avoids over-counting from initial/reset observations.
        levels = []
        current_level = 0
        level_action_count = 0
        step_obs_idx = 0  # tracks which observation corresponds to next step result
        for act in actions:
            if act.get('command') != 'step':
                continue
            step_obs_idx += 1  # skip past the preceding observation
            level_action_count += 1
            # The observation after this step (if available) tells us the new state
            if step_obs_idx < len(observations):
                lc = observations[step_obs_idx].get('levels_completed', 0)
                if lc > current_level:
                    ba = env_baseline_actions[current_level] if current_level < len(env_baseline_actions) else 0
                    levels.append({
                        'level_index': current_level + 1,
                        'completed': True,
                        'actions_taken': level_action_count,
                        'baseline_actions': ba
                    })
                    current_level = lc
                    level_action_count = 0

        # Add final level (may be incomplete)
        if level_action_count > 0 or not levels:
            ba = env_baseline_actions[current_level] if current_level < len(env_baseline_actions) else 0
            levels.append({
                'level_index': current_level + 1,
                'completed': state == 'WIN' and current_level + 1 <= levels_completed,
                'actions_taken': level_action_count,
                'baseline_actions': ba
            })

        games.append({
            'game_id': game_id,
            'state': state,
            'levels_completed': levels_completed,
            'total_levels': env_total_levels,
            'total_actions': total_actions,
            'total_resets': total_resets,
            'levels': levels
        })

    result = {
        'schema_version': '1.0.0',
        'scoring_formula_version': '1.0.0',
        'harness': args.harness,
        'timestamp': datetime.now(timezone.utc).isoformat(),
        'seed': args.seed,
        'games': games,
        'metadata': {
            'model': args.model,
            'version': args.version,
            'notes': args.notes
        }
    }

    with open(args.output, 'w') as f:
        json.dump(result, f, indent=2)
    print(f'Result written to {args.output}')
    print(f'Games: {len(games)}, Completed: {sum(1 for g in games if g[\"state\"] == \"WIN\")}')

if __name__ == '__main__':
    main()
'''

with open(f'{out_dir}/collect_results.py', 'w', newline='\\n') as f:
    f.write(collect_results)

# ===== 3. INSTRUCTION DOCUMENT =====

game_ids_str = ', '.join(game_ids)
game_ids_json = json.dumps(game_ids)
ref_line = f'Reference run: {ref_run_id}' if ref_run_id != 'none' else 'Reference run: none'
now = datetime.now(timezone.utc).isoformat()

# Harness-specific sections
if harness == 'codex':
    harness_guidance = '''## Harness-Specific Guidance: Codex CLI

### Sandbox Awareness

Codex operates in a sandboxed environment. You must install `arc-agi` inside the sandbox:

```bash
pip install arc-agi

Verify the installation:

python -c \"from arc_agi import Arcade, OperationMode; print('OK')\"

Execution Model

Codex uses file I/O rather than interactive stdin/stdout. The recommended approach:

Write a Python script for each game (or a single script that iterates all games)
The script should:
- Initialize the game via the arc-agi API
- Analyze the frame data
- Choose actions based on pattern recognition
- Loop until WIN or GAME_OVER
- Write results to a JSON file
Execute the script via Codex sandbox

Running Codex

Use the codex exec command for unattended execution:

codex exec -m <model> --full-auto "<prompt>"

Additional flags:

--sandbox workspace-write -- control sandbox write permissions
--json -- enable JSON event streaming for structured output parsing

Prompt Template

Use this system prompt when invoking Codex:

You are playing ARC-AGI games. For each game, you will analyze grid patterns and choose actions to transform the grid. Use the game_driver.py script to interact with games. Read observations, reason about patterns, and submit actions. Collect results using collect_results.py.

Limitations

Codex sandbox may have network restrictions -- ensure arc-agi is pre-installed
File system access is sandboxed -- keep all files in the working directory ''' elif harness == 'gemini': harness_guidance = '''## Harness-Specific Guidance: Gemini CLI

Tool Use

Gemini CLI uses function calling / tool_use for shell commands. Ensure tool_use is enabled so Gemini can execute Python commands.

Running Gemini CLI

For headless (non-interactive) mode, use the -p flag:

gemini -p "<prompt>"

Use --output-format json for structured output parsing:

gemini -p "<prompt>" --output-format json

Important: Gemini CLI has a 5-minute timeout on shell tool invocations. For games that may take longer, consider breaking them into smaller batches or using a wrapper script that handles individual game sessions.

Execution Model

Gemini drives a Python session, reading observations and choosing actions:

Start by running the game_driver.py script
Gemini reads the observation JSON output
Gemini reasons about the grid pattern
Gemini sends an action JSON to stdin
Repeat until WIN or GAME_OVER

Prompt Structure

Structure the Gemini conversation to maintain context across turns within a game:

You are playing ARC-AGI games. Use the provided game_driver.py to interact with each game. Run it as: python game_driver.py <game_id> <seed> Read the JSON observations, analyze the grid patterns, and send action JSON via stdin. After all games, run collect_results.py to produce the final result file.

Multi-Turn Context

Gemini should maintain context across turns within a game. Each observation builds on the previous one. Share your reasoning about the pattern as you discover it across levels. ''' elif harness == 'opencode': harness_guidance = '''## Harness-Specific Guidance: OpenCode

Configuration

Create or update opencode.json (no leading dot) in the project root:

{
  \"tools\": {
    \"shell\": true
  }
}

Running OpenCode

For scripted execution, use the opencode run command:

opencode run --model <model> "<prompt>"

Backend Flexibility

OpenCode supports multiple LLM backends. Configure your preferred backend in opencode.json. The game interaction is the same regardless of backend.

Execution Model

Similar to Claude Code -- OpenCode uses shell tool execution to run Python scripts:

Execute python game_driver.py <game_id> <seed> via shell tool
Read the observation JSON
Reason about the grid pattern
Send action JSON to stdin (or use the batch approach below)
Repeat until WIN or GAME_OVER

Session Management

Maintain game state across OpenCode tool calls. The game_driver.py uses stdin/stdout, so you can either:

Run it interactively in a single shell session
Use the batch approach: write a wrapper script that plays the full game autonomously ''' else: harness_guidance = ''

instructions = f'''# ARC-AGI Cross-Harness Benchmark Instructions

Target harness: {harness} Generated: {now} {ref_line} Game set: {len(game_ids)} games Seed: {seed} Max steps per game: {max_steps} Max resets per game: {max_resets}

Prerequisites

Python >= 3.12
pip or uv package manager

Environment Setup

1. Install arc-agi

pip install arc-agi

Or with uv:

uv add arc-agi

2. Verify Installation

python -c \"from arc_agi import Arcade, OperationMode; print('OK')\"

3. Environment Files

Ensure ARC-AGI environment files are available. They should be in {env_dir} relative to the project root, or set a custom path when running the game driver:

python game_driver.py <game_id> {seed} --env-dir {env_dir}

Game Set

Play these {len(game_ids)} games with seed={seed}:

{game_ids_str}

Game IDs as JSON array (for scripting):

{game_ids_json}

Game Interaction Protocol

How ARC-AGI Games Work

Each game has multiple levels (typically 5). Each level presents a grid (2D array of integers). You must figure out the transformation rule and apply it by choosing actions. The same rule applies across all levels -- use early levels to learn the pattern.

Grid Interpretation

frame is a list of 2D integer arrays (grid layers)
Each integer represents a color (0 = background, 1-9 = colors)
Grid sizes start small (e.g., 3x3) and grow with each level
Look for: symmetry, borders, objects, repeating patterns, color relationships

Using the Game Driver

The game_driver.py script (included alongside these instructions) provides a simple stdin/stdout protocol:

python game_driver.py <game_id> {seed} --env-dir {env_dir}

Observation output (printed to stdout as JSON):

{{
  "state": "PLAYING",
  "frame": [[[0, 1, 0], [1, 0, 1], [0, 1, 0]]],
  "levels_completed": 0,
  "total_levels": 5,
  "baseline_actions": [4, 6, 8, 10, 12],
  "available_actions": [
    {{"name": "ACTION1", "is_complex": false}},
    {{"name": "ACTION2", "is_complex": true}}
  ],
  "step": 0,
  "resets": 0
}}

Action input (send as JSON to stdin):

For simple actions (is_complex=false):

{{"command": "step", "action": "ACTION1", "reasoning": "Applying rotation pattern"}}

For complex actions (is_complex=true, requiring x,y coordinates):

{{"command": "step", "action": "ACTION2", "data": {{"x": 2, "y": 1}}, "reasoning": "Filling cell at column 2, row 1"}}

To reset after GAME_OVER:

{{"command": "reset"}}

To end the session:

{{"command": "quit"}}

Game States

State	Meaning	Action
PLAYING	Game continues	Analyze grid, choose action
WIN	All levels completed	Move to next game
GAME_OVER	Failed current level	Reset (if under max_resets={max_resets}) or move on

Action Space

Important: The set of available actions varies between games and may change between levels within the same game. Always consult the available_actions array in each observation before choosing an action. Do not assume actions from one game are valid in another.

Strategy Tips

Level 1: Experiment to discover what each action does
Level 2+: Apply the rule learned from earlier levels
After reset: Try a different approach based on what you learned
Efficiency matters: Fewer actions = higher score

{harness_guidance}

Scoring Reference

Scores are calculated using these formulas:

Per-level score (0.0 to 1.0):

level_score = min((baseline_actions / actions_taken) ** 2, 1.0) if completed else 0.0

Per-game score (weighted average, later levels weighted more):

game_score = sum(level_score[i] * (i+1) for i in 0..N-1) / sum(1..N)

Overall score:

overall_score = average(all game scores)

Result Collection

After playing all games, produce a result JSON file using collect_results.py or manually.

Automated Collection

If you logged sessions as JSONL files (one per game, in a sessions/ directory):

python collect_results.py \\
  --sessions-dir ./sessions \\
  --output result.json \\
  --harness {harness} \\
  --seed {seed} \\
  --model \"<your-model-name>\" \\
  --version \"<harness-version>\" \\
  --notes \"<any context>\"

Manual Result File

Create a JSON file matching this schema:

{{
  "schema_version": "1.0.0",
  "scoring_formula_version": "1.0.0",
  "harness": "{harness}",
  "timestamp": "<ISO 8601 timestamp>",
  "seed": {seed},
  "games": [
    {{
      "game_id": "<game_id>",
      "state": "WIN or GAME_OVER or NOT_PLAYED",
      "levels_completed": 5,
      "total_levels": 5,
      "total_actions": 45,
      "total_resets": 0,
      "levels": [
        {{
          "level_index": 1,
          "completed": true,
          "actions_taken": 4,
          "baseline_actions": 4
        }}
      ]
    }}
  ],
  "metadata": {{
    "model": "<model used>",
    "version": "<harness version>",
    "notes": "<any context>"
  }}
}}

Result Template

Here is a pre-filled template with all {len(game_ids)} game IDs:

import json
from datetime import datetime, timezone

games = []
for gid in {game_ids_json}:
    games.append({{
        \"game_id\": gid,
        \"state\": \"NOT_PLAYED\",
        \"levels_completed\": 0,
        \"total_levels\": 5,
        \"total_actions\": 0,
        \"total_resets\": 0,
        \"levels\": []
    }})

result = {{
    \"schema_version\": \"1.0.0\",
    \"scoring_formula_version\": \"1.0.0\",
    \"harness\": \"{harness}\",
    \"timestamp\": datetime.now(timezone.utc).isoformat(),
    \"seed\": {seed},
    \"games\": games,
    \"metadata\": {{
        \"model\": \"<your-model>\",
        \"version\": \"<version>\",
        \"notes\": \"\"
    }}
}}

with open(\"result.json\", \"w\") as f:
    json.dump(result, f, indent=2)
print(\"Template written to result.json -- fill in game results.\")

Importing Results

Once you have the result JSON file, import it back into the ARC-AGI benchmarker:

/arc-cross-harness import {harness} <path-to-result.json>

This will validate, normalize scores, and store the results for comparison. '''

with open(f'{out_dir}/instructions.md', 'w', newline='\n') as f: f.write(instructions)

print(json.dumps({ 'status': 'generated', 'instructions': f'{out_dir}/instructions.md', 'game_driver': f'{out_dir}/game_driver.py', 'collect_results': f'{out_dir}/collect_results.py', 'game_count': len(game_ids), 'seed': seed, 'ref_run_id': ref_run_id })) "


**IMPORTANT**: Replace the following angle-bracket placeholders with actual literal values:
- `<HARNESS>`: the harness name (e.g., `codex`)
- `<GAME_IDS_JSON>`: Python list literal of game IDs (e.g., `["bt11", "ls20"]`)
- `<SEED>`: integer seed value
- `<MAX_STEPS>`: integer max steps
- `<MAX_RESETS>`: integer max resets
- `<REF_RUN_ID>`: the reference run ID string, or `none` if no `--ref` was provided

### G3: Report to User

After generation, report:

Instructions generated: .arc-agi-benchmarks/cross-harness/<HARNESS>/instructions.md Game driver: .arc-agi-benchmarks/cross-harness/<HARNESS>/game_driver.py Result collector: .arc-agi-benchmarks/cross-harness/<HARNESS>/collect_results.py Game set: <N> games, seed=<SEED> Reference run: <RUN_ID or "none">

Next steps:

Open the instructions file and follow the setup steps in the target harness
Run the benchmark on the target harness
Save the result JSON file
Import with: /arc-cross-harness import <HARNESS> <result-file-path>


---

## Sub-command: Import (`/arc-cross-harness import <HARNESS> <PATH>`)

### I1: Validate the Result File

```bash
$VENV_PYTHON -c "
import json, sys, os

harness = '<HARNESS>'
result_path = '<RESULT_PATH>'

# Validate harness
if harness not in ('codex', 'gemini', 'opencode'):
    print(f'ERROR: Invalid harness \"{harness}\". Supported: codex, gemini, opencode', file=sys.stderr)
    sys.exit(1)

# Validate file exists
if not os.path.isfile(result_path):
    print(f'ERROR: Result file not found: {result_path}', file=sys.stderr)
    sys.exit(1)

# Read and validate schema
with open(result_path) as f:
    try:
        result = json.load(f)
    except json.JSONDecodeError as e:
        print(f'ERROR: Invalid JSON in result file: {e}', file=sys.stderr)
        sys.exit(1)

errors = []
warnings = []

# Required top-level fields
for field in ['schema_version', 'harness', 'timestamp', 'seed', 'games', 'metadata']:
    if field not in result:
        errors.append(f'Missing required field: {field}')

if 'scoring_formula_version' not in result:
    result['scoring_formula_version'] = '1.0.0'

# Warn on harness mismatch
if result.get('harness') and result['harness'] != harness:
    warnings.append(f'Harness field in result (\"{result[\"harness\"]}\") does not match argument (\"{harness}\")')

# Warn on schema version
if result.get('schema_version') and result['schema_version'] != '1.0.0':
    warnings.append(f'Schema version {result[\"schema_version\"]} may not be fully supported (expected 1.0.0)')

# Validate games array
games = result.get('games', [])
if not isinstance(games, list):
    errors.append('\"games\" must be an array')
else:
    for i, game in enumerate(games):
        prefix = f'games[{i}]'
        for field in ['game_id', 'state', 'levels_completed', 'total_levels', 'total_actions', 'total_resets', 'levels']:
            if field not in game:
                errors.append(f'Missing required field: {prefix}.{field}')

        levels = game.get('levels', [])
        if isinstance(levels, list):
            for j, level in enumerate(levels):
                lprefix = f'{prefix}.levels[{j}]'
                for field in ['level_index', 'completed', 'actions_taken', 'baseline_actions']:
                    if field not in level:
                        errors.append(f'Missing required field: {lprefix}.{field}')

if errors:
    print('Schema validation errors:', file=sys.stderr)
    for e in errors:
        print(f'  - {e}', file=sys.stderr)
    sys.exit(1)

if warnings:
    for w in warnings:
        print(f'WARNING: {w}', file=sys.stderr)

print(json.dumps({'status': 'valid', 'games': len(games), 'warnings': warnings}))
"

IMPORTANT: Replace <HARNESS> with the harness name and <RESULT_PATH> with the actual file path to the result JSON.

I2: Normalize Scores and Create Run Directory

$VENV_PYTHON -c "
import json, sys, os, uuid
from datetime import datetime, timezone

harness = '<HARNESS>'
result_path = '<RESULT_PATH>'

with open(result_path) as f:
    result = json.load(f)

if 'scoring_formula_version' not in result:
    result['scoring_formula_version'] = '1.0.0'

# Duplicate import check: reject if same timestamp+harness already imported
import_ts = result.get('timestamp', '')
existing_runs_dir = f'.arc-agi-benchmarks/cross-harness/{harness}/runs'
if os.path.isdir(existing_runs_dir):
    for existing_dir in os.listdir(existing_runs_dir):
        existing_meta_path = os.path.join(existing_runs_dir, existing_dir, 'run-meta.json')
        if os.path.isfile(existing_meta_path):
            with open(existing_meta_path) as ef:
                existing_meta = json.load(ef)
            if existing_meta.get('timestamp') == import_ts and existing_meta.get('harness') == harness:
                print(f'ERROR: Duplicate import detected. A run with timestamp {import_ts} for harness {harness} already exists (run_id: {existing_dir}).', file=sys.stderr)
                sys.exit(1)

run_id = str(uuid.uuid4())
run_dir = f'.arc-agi-benchmarks/cross-harness/{harness}/runs/{run_id}'
os.makedirs(run_dir, exist_ok=True)

games = result.get('games', [])
metadata = result.get('metadata', {})

# ===== Recalculate all scores =====

def compute_level_score(baseline_actions, actions_taken, completed):
    if not completed:
        return 0.0
    if actions_taken <= 0:
        return 0.0
    return min((baseline_actions / actions_taken) ** 2, 1.0)

def compute_game_score(level_scores):
    if not level_scores:
        return 0.0
    n = len(level_scores)
    weighted = sum(level_scores[i] * (i + 1) for i in range(n))
    total_weight = sum(range(1, n + 1))
    return weighted / total_weight if total_weight > 0 else 0.0

total_environments = len(games)
total_environments_completed = 0
total_levels_completed_sum = 0
total_levels_sum = 0
total_actions_sum = 0
game_scores = []

scorecard_games = []

for game in games:
    game_id = game['game_id']
    state = game.get('state', 'NOT_PLAYED')
    levels_completed = game.get('levels_completed', 0)
    total_levels = game.get('total_levels', 5)
    total_actions = game.get('total_actions', 0)
    total_resets = game.get('total_resets', 0)
    levels = game.get('levels', [])

    if state == 'WIN':
        total_environments_completed += 1
    total_levels_completed_sum += levels_completed
    total_levels_sum += total_levels
    total_actions_sum += total_actions

    # Compute per-level scores
    level_scores_list = []
    level_actions_list = []
    level_baselines_list = []
    for lvl in levels:
        ls = compute_level_score(
            lvl.get('baseline_actions', 0),
            lvl.get('actions_taken', 0),
            lvl.get('completed', False)
        )
        level_scores_list.append(ls)
        level_actions_list.append(lvl.get('actions_taken', 0))
        level_baselines_list.append(lvl.get('baseline_actions', 0))

    # Pad to total_levels if needed
    while len(level_scores_list) < total_levels:
        level_scores_list.append(0.0)
        level_actions_list.append(0)
        level_baselines_list.append(0)

    game_score = compute_game_score(level_scores_list)
    game_scores.append(game_score)

    run_guid = str(uuid.uuid4())
    scorecard_games.append({
        'id': game_id,
        'score': game_score,
        'runs': [{
            'id': game_id,
            'guid': run_guid,
            'score': game_score,
            'levels_completed': levels_completed,
            'actions': total_actions,
            'resets': total_resets,
            'state': state,
            'completed': state == 'WIN',
            'level_scores': level_scores_list,
            'level_actions': level_actions_list,
            'level_baseline_actions': level_baselines_list,
            'number_of_levels': total_levels
        }]
    })

overall_score = sum(game_scores) / len(game_scores) if game_scores else 0.0

# ===== Write run-meta.json =====

run_meta = {
    'run_id': run_id,
    'harness': harness,
    'timestamp': result.get('timestamp', datetime.now(timezone.utc).isoformat()),
    'duration_seconds': 0,
    'game_set': 'imported',
    'game_ids': [g['game_id'] for g in games],
    'seed': result.get('seed', 0),
    'max_steps': metadata.get('max_steps', result.get('max_steps', 0)),
    'max_resets': metadata.get('max_resets', result.get('max_resets', 0)),
    'config_hash': '',
    'harness_config': {
        'model': metadata.get('model', 'unknown'),
        'plugins': [],
        'skills': [],
        'mcp_servers': []
    },
    'status': 'completed',
    'arc_agi_version': 'unknown',
    'plugin_version': '1.0.0'
}

with open(os.path.join(run_dir, 'run-meta.json'), 'w') as f:
    json.dump(run_meta, f, indent=2)

# ===== Write scorecard.json =====

scorecard = {
    'card_id': run_id,
    'source_url': None,
    'tags': [],
    'opaque': None,
    'competition_mode': False,
    'score': overall_score,
    'total_environments_completed': total_environments_completed,
    'total_environments': total_environments,
    'total_levels_completed': total_levels_completed_sum,
    'total_levels': total_levels_sum,
    'total_actions': total_actions_sum,
    'games': scorecard_games,
    'tags_scores': []
}

with open(os.path.join(run_dir, 'scorecard.json'), 'w') as f:
    json.dump(scorecard, f, indent=2)

# ===== Write environment-scores.json =====

env_scores = {
    'run_id': run_id,
    'overall_score': overall_score * 100,
    'environments': []
}

for i, game in enumerate(games):
    game_id = game['game_id']
    sc_game = scorecard_games[i]
    best_run = sc_game['runs'][0]

    env_data = {
        'game_id': game_id,
        'title': game_id,
        'tags': [],
        'score': sc_game['score'] * 100,
        'levels_completed': game.get('levels_completed', 0),
        'total_levels': game.get('total_levels', 5),
        'total_actions': game.get('total_actions', 0),
        'total_resets': game.get('total_resets', 0),
        'state': game.get('state', 'NOT_PLAYED'),
        'completed': game.get('state') == 'WIN',
        'levels': []
    }

    for j, lvl in enumerate(game.get('levels', [])):
        env_data['levels'].append({
            'level_index': lvl.get('level_index', j + 1),
            'score': best_run['level_scores'][j] * 100 if j < len(best_run['level_scores']) else 0,
            'actions_taken': lvl.get('actions_taken', 0),
            'baseline_actions': lvl.get('baseline_actions', 0),
            'completed': lvl.get('completed', False)
        })

    env_scores['environments'].append(env_data)

with open(os.path.join(run_dir, 'environment-scores.json'), 'w') as f:
    json.dump(env_scores, f, indent=2)

# ===== Report =====

print(json.dumps({
    'run_id': run_id,
    'run_dir': run_dir,
    'overall_score': overall_score,
    'total_environments': total_environments,
    'total_environments_completed': total_environments_completed,
    'total_levels_completed': total_levels_completed_sum,
    'total_levels': total_levels_sum,
    'total_actions': total_actions_sum
}))
"

IMPORTANT: Replace <HARNESS> with the harness name and <RESULT_PATH> with the actual file path.

I3: Report Import Results

After the import script runs, display the results:

Imported results from <HARNESS>:
  Run ID:          <RUN_ID>
  Games:           <N>
  Overall Score:   <SCORE> / 100
  Levels Complete: <N> / <TOTAL>
  Stored at:       .arc-agi-benchmarks/cross-harness/<HARNESS>/runs/<RUN_ID>/

To compare with a Claude Code run:
  /arc-compare <claude-run-id> <RUN_ID>

Or use:
  /arc-cross-harness compare <claude-run-id> <RUN_ID>

I4: Suggest Auto-Compare (Optional)

Check if a Claude Code run exists and suggest comparison (do NOT auto-invoke):

$VENV_PYTHON -c "
import json, os

runs_dir = '.arc-agi-benchmarks/runs'
if os.path.isdir(runs_dir):
    completed = []
    for d in os.listdir(runs_dir):
        meta_path = os.path.join(runs_dir, d, 'run-meta.json')
        if os.path.isfile(meta_path):
            with open(meta_path) as f:
                meta = json.load(f)
            if meta.get('status') == 'completed':
                completed.append(meta)
    completed.sort(key=lambda m: m.get('timestamp', ''), reverse=True)
    if completed:
        latest = completed[0]
        print(json.dumps({'latest_claude_run': latest['run_id']}))
    else:
        print(json.dumps({'latest_claude_run': None}))
else:
    print(json.dumps({'latest_claude_run': None}))
"

If a Claude Code run was found, tell the user:

A Claude Code run is available (latest: <LATEST_RUN_ID>).
Would you like to compare? Run: /arc-compare <LATEST_RUN_ID> <IMPORTED_RUN_ID>

Do NOT automatically invoke the comparison.

Sub-command: Compare (`/arc-cross-harness compare [run-ids...] [--all]`)

C1: Resolve Run IDs

If --all flag is provided, find the latest run from each source:

$VENV_PYTHON -c "
import json, os

run_ids = []

# Latest Claude Code run
runs_dir = '.arc-agi-benchmarks/runs'
if os.path.isdir(runs_dir):
    completed = []
    for d in os.listdir(runs_dir):
        meta_path = os.path.join(runs_dir, d, 'run-meta.json')
        if os.path.isfile(meta_path):
            with open(meta_path) as f:
                meta = json.load(f)
            if meta.get('status') == 'completed':
                completed.append(meta)
    completed.sort(key=lambda m: m.get('timestamp', ''), reverse=True)
    if completed:
        run_ids.append({'run_id': completed[0]['run_id'], 'harness': 'claude-code'})

# Latest from each cross-harness
cross_dir = '.arc-agi-benchmarks/cross-harness'
if os.path.isdir(cross_dir):
    for harness_name in sorted(os.listdir(cross_dir)):
        harness_runs = os.path.join(cross_dir, harness_name, 'runs')
        if not os.path.isdir(harness_runs):
            continue
        completed = []
        for d in os.listdir(harness_runs):
            meta_path = os.path.join(harness_runs, d, 'run-meta.json')
            if os.path.isfile(meta_path):
                with open(meta_path) as f:
                    meta = json.load(f)
                if meta.get('status') == 'completed':
                    completed.append(meta)
        completed.sort(key=lambda m: m.get('timestamp', ''), reverse=True)
        if completed:
            run_ids.append({'run_id': completed[0]['run_id'], 'harness': harness_name})

if len(run_ids) < 2:
    print(json.dumps({'error': 'Fewer than 2 runs available. Generate instructions and import results first.', 'found': run_ids}))
else:
    print(json.dumps({'run_ids': run_ids}))
"

If --all was not provided, use the run IDs directly as provided by the user.

C2: Delegate to Compare-Runs

Once you have at least 2 run IDs, invoke the compare-runs skill:

Now invoke /arc-compare with the resolved run IDs.

Pass the run IDs as space-separated arguments, e.g.:

/arc-compare <RUN_ID_1> <RUN_ID_2>

The compare-runs skill already supports cross-harness directory lookup (it checks .arc-agi-benchmarks/cross-harness/*/runs/<run-id>/ when a run is not found in the standard runs directory).

If fewer than 2 runs are available, tell the user:

Fewer than 2 runs available for comparison.

To get started:
1. Run a Claude Code benchmark: /arc-benchmark
2. Generate cross-harness instructions: /arc-cross-harness generate <harness>
3. Run the benchmark on the target harness
4. Import results: /arc-cross-harness import <harness> <result-file>
5. Compare: /arc-cross-harness compare --all

Error Handling

Invalid Harness Name

If the user provides an unrecognized harness name:

Invalid harness <NAME>. Supported harnesses: codex, gemini, opencode

Reference Run Not Found

If --ref <run-id> points to a non-existent run:

Run <RUN_ID> not found. Available runs:

Then list available completed runs from .arc-agi-benchmarks/runs/.

Result File Not Found

If the import path does not exist:

Result file not found: <PATH> Ensure the file exists and the path is correct.

Schema Validation Failure

If required fields are missing from the result JSON, report each error:

Schema validation errors:

Missing required field: games[0].levels_completed

Missing required field: games[1].levels[0].baseline_actions

No Environments Available

If arc.get_environments() returns empty:

No ARC-AGI environments found. Run /arc-setup first to configure environment files.

Notes

Always use $VENV_PYTHON (the shell variable set in Step 1a), never system Python.
All scores are recalculated by the plugin using the standard scoring formula. Never trust externally-reported scores.
Scoring scale: scorecard.json stores game scores on a 0-1 float scale (e.g., 0.85). environment-scores.json stores per-level scores multiplied by 100 for display (e.g., 85). When showing scores to users, multiply by 100 to get the percentage form.
The cross-harness skill does NOT install or invoke other CLI tools -- it only generates instructions and processes result files.
Generated instruction documents are self-contained: a user with no prior context can follow them.
Angle-bracket placeholders (<HARNESS>, <PATH>, etc.) must be substituted with actual literal values before running commands.

Cross Harness

ARC-AGI Cross-Harness

Sub-commands

Step 1: Pre-flight Checks

1a: Detect Virtual Environment

1b: Read Configuration

1c: Verify arc-agi is Available

1d: Parse Sub-command and Arguments

Step 2: Execute Sub-command

Sub-command: Generate (/arc-cross-harness generate <HARNESS>)

G1: Resolve Game Set and Parameters

G2: Generate Instruction Document, Game Driver, and Result Collector

Execution Model

Running Codex

Prompt Template

Limitations

Tool Use

Running Gemini CLI

Execution Model

Prompt Structure

Multi-Turn Context

Configuration

Running OpenCode

Backend Flexibility

Execution Model

Session Management

Prerequisites

Environment Setup

1. Install arc-agi

2. Verify Installation

3. Environment Files

Game Set

Game Interaction Protocol

How ARC-AGI Games Work

Grid Interpretation

Using the Game Driver

Game States

Action Space

Strategy Tips

Scoring Reference

Result Collection

Automated Collection

Manual Result File

Result Template

Importing Results

I2: Normalize Scores and Create Run Directory

I3: Report Import Results

I4: Suggest Auto-Compare (Optional)

Sub-command: Compare (/arc-cross-harness compare [run-ids...] [--all])

C1: Resolve Run IDs

C2: Delegate to Compare-Runs

Error Handling

Invalid Harness Name

Reference Run Not Found

Result File Not Found

Schema Validation Failure

No Environments Available

Notes

Sub-command: Generate (`/arc-cross-harness generate <HARNESS>`)

Sub-command: Compare (`/arc-cross-harness compare [run-ids...] [--all]`)