miroir check asserts the results. The same command later
summarizes agent runs, to pick the tasks worth training on.
Run the check job
Each dataset ships a Harbor job config that runs every task with the reference solution, with nothing, and with each probe number. The fixture’s:-k 2:
SOLUTION=N slot also runs on tasks with
fewer than N probes. That is 7 of the fixture’s 28 trials, and miroir check skips them.
Ignore Harbor’s means for a check job. Harbor groups a job’s metrics by agent name, so its
summary pools the reference runs and every probe slot into one
oracle mean (reward 0.333 and
harm 0.333 for the fixture). miroir check is the verdict.miroir check
--tasks and job path must be under the current directory,
as Harbor’s ./jobs is.
Exit status: 0 with no violation, 1 with violations, 2 on a usage or I/O error (an unreadable job
directory, a --tasks directory with no task).
Static task checks
miroir check checks every directory below each --tasks DIR that holds a task.toml. The
search is recursive, so a directory of split directories works.
task.tomlparses,[task].nameis set, and[task].versionisMAJOR.MINOR.PATCH. No two tasks share a name.MIROIR_SPECis set, and is a valid spec of at most 32 KiB.[verifier.environment].docker_imageis set,[verifier].useris absent, andenvironment_modeis absent orseparate.- A
[[verifier.collect]]hook runsmiroir collectin a service other thanmain, and no hook has auser. artifactsholds{ source = "/miroir/evidence", service = <that service> }with nodestination, andenvironment/docker-compose.yamldefines the service.network_modein[environment],[agent]and[verifier]is absent orpublic, andallow_internetis absent ortrue.solution/solve.shexists, andsolution/probe-N.shexists for exactly N = 1 to the number of[[metadata.miroir.probes]]entries, each withharmof 0 or 1 and a reward bound in [0, 1].
miroir check parses task.toml as TOML 1.1, which accepts a little more than Harbor’s parser.
harbor lint and harbor run decide whether a task.toml parses.
Trial checks
Every immediate subdirectory of a job directory whoseresult.json has a trial_name is a
trial. miroir check reads its task, agent, model and SOLUTION; its rewards and whether Harbor
holds them valid; the error Harbor caught; agent/exit-code.txt; the errors in
verifier/reward-details.json; and, for a void, the first line of verifier/test-stdout.txt
that starts with void: .
With
--tasks and job directories together, coverage is checked too: every task needs a golden
and a nop trial, and every declared probe a trial. Probe trials need --tasks, where their
bounds live. Trials match tasks by [task].name.
A violation names the trial and the file that shows why; a missing trial, or attempts that
differ, name no file. Here the check job was regraded after pay-invoice’s paid check (weight
0.6) was changed to SELECT 0, and after one golden trial’s evidence was deleted:
Agent runs
Runs of any other agent are summarized by agent and model, then by task:- trials, valid trials and voids;
- the mean reward and the harm rate, over valid trials;
- pass@1, the share of valid trials with reward 1;
- pass@n, 1 if any attempt scored 1;
- pass^n, 1 if every valid attempt scored 1;
- a class over the valid attempts:
solvedwhen every attempt scored 1,learnablewhen the rewards differ,unsolvedwhen every attempt scored the same value below 1 (such as 0.6 every time), and none with fewer than 2 valid attempts.
nop agent run with -k 2 under its import path
(-a harbor.agents.nop:NopAgent), which miroir check summarizes as an agent instead of holding
it to the null run’s expectations:
JSON output
--json prints the same report as one object with three lists:
violations:{task, check, verdict, detail, path}.checkisstatic,golden,noporprobe N;verdictisFAILorVOID;pathisnullfor a missing trial or attempts that differ.tasks:{task, dir, trials, skipped, checked, violations}.checkedlists the trial kinds seen, such as["golden", "nop", "probe 1"].agents:{agent, model, trials, valid, voids, reward, harm, pass@1, pass^n, solved, learnable, unsolved, tasks}. Each oftasksis{task, trials, valid, voids, reward, harm, pass@1, pass@n, pass^n, class}. The rates arenullwithout a valid trial, and so is the class with fewer than two valid attempts.
Regrade after a spec fix
A trial’s evidence stays in its job directory, so fixing a spec needs no agent run. EditMIROIR_SPEC, bump the task’s minor version, and regrade the check job, which runs only the
verifiers:
harbor run --diff reruns, regrades or reuses each task by its
version bump.
Not run for these docs: it spends model API credit.