A trial
A trial is one agent’s attempt at one task. Harbor starts three fresh containers for it, and removes them afterwards:main, where the agent runs;- the world, where the product runs with its data;
- the verifier, where the grader runs.
World image
The world image is your product’s own image, plus Miroir and a world: the data the product starts every trial with. The world is seeded once, atdocker build, and every trial boots a fresh copy,
with its dates moved to the trial’s day. Harbor runs the image as a service next to the agent’s
container. See Wrap a product.
Evidence
The evidence is what one trial changed and sent, in one SQLite file: the rows that changed in the product’s database, the emails and HTTP requests the product sent, and faults such as a crashed process. See Evidence.Spec
The spec says how to grade the evidence. It lives in the task’stask.toml, as
[verifier.env] MIROIR_SPEC, and has three parts:
- checks, SQL queries that measure how much of the task got done;
- gates, SQL conditions that must hold, such as “no email was sent”;
- allow, the changes the task is allowed to make.
Reward and harm
Every graded trial gets two numbers.reward, from 0 to 1, is the weighted mean of the check
scores.
harm is 1 when the trial did damage: a change no allow entry admits, a failed gate, or a
fault. Harm sets the reward to 0.
Probe
A probe is a deliberately wrong solution, such as one that also charges another customer, with the reward bounds and the harm it must get. Probes test the grader. A grader is valid when the reference solution scores reward 1 and harm 0, doing nothing scores 0 and 0, and every probe stays within its reward bounds and gets the harm it declares. See Validate graders.Void
A void is a trial Miroir failed to grade, such as one with no evidence or a broken spec. It gets no reward at all, and Harbor leaves it out of its means. Only Miroir’s own failures void a trial: anything the agent can influence is scored. See Outcomes.Commands
miroir is one static Linux binary. bake, run, collect and grade run in containers;
check and draft run on your machine, from the Miroir image.