open-standards/joulespec/conformance/v1/evaluation

The evaluation corpus

Cases a too-generous implementation passes incorrectly.

The wire vectors in ../vectors/ certify that an implementation round-trips a receipt's bytes. They say nothing about whether it reaches the right verdict, because they carry one scenario. This corpus is the other half.

Each case directory holds five files. Read the first four, evaluate, and compare against the fifth:

File	What it is
spec.json	the signed specification
state.json	the artifact state the witnesses were observed at
witnesses.json	the signed witnesses
work.json	the declared work, which is what the attestation axis is computed from
expect.json	the verdict §3 requires, with the trap it sets

expect.json carries its own intent and trap in prose, so a failing case says what it was for rather than only which field differed.

Why the cases are shaped the way they are

Every case states a direction. Most are over-grading: an implementation that reports conformance never established, which is the failure this standard exists to stop. At least one is under-grading, because an implementation that graded nothing at all would pass every over-grading case — a corpus with only one direction rewards a rule that refuses everything. A test asserts that both directions are present, that every over-grading case's correct verdict falls short of the bar or is blocked by a flag, and that every under-grading case's correct verdict reaches it.

The traps are the ones that are reachable by accident rather than by carelessness:

• a mean instead of a minimum, which lets well-evidenced obligations conceal one nobody looked at;
• skipping an unevidenced obligation rather than letting its absence empty the grade — and reporting E0 there, which would claim a citation nobody made;
• treating an unwitnessed non-goal as an owed obligation with no evidence, which is the under-grading case;
• grading a witnessed non-goal's evidence, or discarding it quietly, instead of raising the flag that blocks conformance;
• awarding E3 on a killed counter-artifact alone, where the check's anchor is unstated;
• awarding E4 because a prover succeeded, where the formalization is the prover's own word or its vacuity was never checked — two independent ways a proof establishes nothing about a clause;
• counting a witness bound to a superseded state, which an empty footprint makes stale on any change at all;
• silently discarding a witness whose obligation address is a clause's previous wording, which is exactly how drift stops being visible;
• reading reviewed off the evidence label when the reviewer is the clause's own author.

Where the expected verdicts come from

They are written by hand from §3 of the specification. They are not read back out of this implementation: a corpus whose expectations are whatever the code produced would certify agreement with itself and nothing else.

That distinction earned its keep while this corpus was being written. Seven of the ten expectations were wrong on first authoring — all the same mistake, and the author's rather than the evaluator's. They assumed the attestation axis was computed from who signed a witness, when §2.8 computes it from who authored the work; with no work declared, attestation is absent. The fix was not to copy the observed value into the expectation, which would have hidden the misunderstanding, but to declare work in every case so that the axis is actually exercised.

Regenerating

JOULESPEC_GEN_EVALUATION=1 cargo test -p joulespec-core --test evaluation

That republishes the input files and the expectations as the test states them. Without the variable the test reads the published expectation back and asserts it still agrees with the test's own, so an expectation edited in one place and not the other fails rather than drifting.