task.toml, under
[verifier.env] MIROIR_SPEC, and has three parts:
- checks measure progress. Each scores from 0 to 1, and their weighted mean is the reward.
- gates are conditions that must hold. A gate that fails is harm.
- allow lists the changes the task may make. Any other change is harm.
Format
MIROIR_SPEC holds at most 32 KiB. Write it in task.toml as a TOML multi-line literal string
('''), in which nothing is escaped. SQL never needs ''': miroir draft writes char(39)
where a literal would hold three quotes in a row. Never write the whole value as ${…}, which
Harbor would template.
A spec is refused, and every trial of the task voids, for:
- an unknown key;
- a missing
checksorallow, or an emptychecks; - a bad or duplicate name;
- an empty
sqlorwhere; - a weight that is not a finite number above 0;
- an
opother thaninsert,updateordelete; columnson an insert or a delete, or an emptycolumns;- more than 32 KiB.
checks[1].weight: must be > 0. miroir check --tasks
reports them before any run.
How a check or a gate scores
Each one runs asSELECT * FROM (<sql>) LIMIT 2, so it must be one SELECT; a trailing ; is
fine.
A statement that does not compile, or that returns more than one column, voids the trial. Both
depend only on the spec and the world’s tables, never on what the agent did.
Scope
A row ofchanges is in scope when some allow entry has its table and op, and:
- for an update, the entry lists no
columns, or lists every changed column; - the entry has no
where, or the change is among the rows itswhereadmits:- insert:
SELECT _change FROM after."T" WHERE _op = 'insert' AND (<where>); - delete:
SELECT _change FROM before."T" WHERE _op = 'delete' AND (<where>); - update: both images, intersected, so a row can neither enter nor leave the allowed region unnoticed.
- insert:
truncate and schema changes are never in scope. A where that fails at run time admits
nothing, and its error is recorded on the scope criterion.
Reward and harm
Outcomes
A trial voids only for Miroir’s own failures. Anything the agent can influence is scored. Void. The grader printsvoid: <reason>, writes no reward file and exits 75. Harbor records
an invalid trial and leaves it out of its means.
MIROIR_SPECis missing, invalid or over 32 KiB.- A check, gate or
wheredoes not compile, or a check returns more than one column. - An
allowentry names a table or column that is not captured. - The evidence is missing, unreadable, or in another format version.
- Collect failed: the world never became ready, or the change log could not be folded. Collect exits 1 and writes no evidence.
- The world container was killed (out of memory, disk full), so collect could not run.
- A change no
allowentry admits, and anytruncateorschemachange. - A gate that scores below 1 or fails at run time.
- A fault:
crash,sink,quiet_timeout,database,log_budgetorevidence_budget.
- A check that fails at run time (past the step budget, more than one row, a text, blob, NaN or
infinite value) scores 0, and the error is recorded in
reward-details.json. - A
wherethat fails at run time: itsallowentry admits nothing, and the error is recorded. - No answer file, one that is not a regular file, or an empty one: no answer (NULL).
miroir check fails validation on any void and on any recorded check error, so spec bugs surface
before agent runs.
Authoring rules
- A check is a predicate over what the episode changed, sent and answered. Rows absent from
changesare as baked, so untouched state is a constant the author or generator knows. - For any row predicate P on a captured table,
end_count(P) = baked_count(P) − count(before ⊨ P) + count(after ⊨ P). - What
allowforbids needs no gate. Gates are for effects (“no email”) and for invariants scope cannot express. - Refer to new rows by their content, never by ids the product will assign.
- Compare row images with each other, never with seed timestamps or
now. Never compare a date column with a timestamp column: the boot’s date shift moves them by different amounts for part of each day. Never userandom()or'now'. - A check that doing nothing satisfies belongs in
gates: the null run must score 0, andmiroir checkfails a task whose null run does not.
How grading runs
/tests/test.sh in the miroir image is exec miroir grade. It reads MIROIR_SPEC, the
evidence at /miroir/evidence/evidence.db, and the answer at /logs/artifacts/answer.txt. Harbor
restores both files at the paths they were collected from.
- Parse the spec, open a copy of the evidence, and check its format.
- Check that every
allowtable and column is captured. A typo voids, naming the nearest name. - Build the
before,afterandanswertables. - Compile every check, gate and
where. One that does not compile voids the trial, naming it:check 'paid' does not compile: no such column: payment_intnt. - Compute scope, then the gates, then the checks, each under the step budget.
- Write
/logs/verifier/reward-details.json, then/logs/verifier/reward.json, each through a temporary file and a rename. Print the summary and exit 0.
reward.json always holds both keys, such as {"harm": 1, "reward": 0.0}.
reward-details.json uses the shape harbor view renders: a reward and a harm entry, each
with criteria {name, value, raw, weight}, plus reasoning and error when there are any. raw
is the value, or "NULL", "no row" or "error" when there is no number. Harm’s criteria are
scope, one gate:<name> per gate, and faults, each with weight 0.
The summary on stdout, which Harbor keeps as verifier/test-stdout.txt, here for pay-invoice’s
probe 1:
allow line that would admit it, whose
where is the row’s primary key. truncate and schema changes say “never allowed”.