# The evaluation corpus

Cases a too-generous implementation passes incorrectly.

The wire vectors in [`../vectors/`](../vectors) certify that an
implementation round-trips a receipt's **bytes**. They say nothing about
whether it reaches the right **verdict**, because they carry one scenario.
This corpus is the other half.

Each case directory holds five files. Read the first four, evaluate, and
compare against the fifth:

| File | What it is |
|---|---|
| `spec.json` | the signed specification |
| `state.json` | the artifact state the witnesses were observed at |
| `witnesses.json` | the signed witnesses |
| `work.json` | the declared work, which is what the attestation axis is computed from |
| `expect.json` | the verdict §3 requires, with the trap it sets |

`expect.json` carries its own `intent` and `trap` in prose, so a failing case
says what it was for rather than only which field differed.

## Why the cases are shaped the way they are

Every case states a **direction**. Most are `over-grading`: an
implementation that reports conformance never established, which is the
failure this standard exists to stop. At least one is `under-grading`,
because an implementation that graded nothing at all would pass every
over-grading case — a corpus with only one direction rewards a rule that
refuses everything. A test asserts that both directions are present, that
every over-grading case's correct verdict falls short of the bar or is
blocked by a flag, and that every under-grading case's correct verdict
reaches it.

The traps are the ones that are reachable by accident rather than by
carelessness:

- a **mean** instead of a minimum, which lets well-evidenced obligations
  conceal one nobody looked at;
- **skipping** an unevidenced obligation rather than letting its absence
  empty the grade — and reporting `E0` there, which would claim a citation
  nobody made;
- treating an **unwitnessed non-goal** as an owed obligation with no
  evidence, which is the under-grading case;
- grading a **witnessed non-goal**'s evidence, or discarding it quietly,
  instead of raising the flag that blocks conformance;
- awarding `E3` on a killed counter-artifact alone, where the check's
  **anchor is unstated**;
- awarding `E4` because a prover succeeded, where the **formalization is the
  prover's own word** or its **vacuity was never checked** — two independent
  ways a proof establishes nothing about a clause;
- counting a witness bound to a **superseded state**, which an empty
  footprint makes stale on any change at all;
- silently discarding a witness whose obligation address is a clause's
  **previous wording**, which is exactly how drift stops being visible;
- reading `reviewed` off the evidence label when the **reviewer is the
  clause's own author**.

## Where the expected verdicts come from

They are written by hand from §3 of the specification. They are **not**
read back out of this implementation: a corpus whose expectations are
whatever the code produced would certify agreement with itself and nothing
else.

That distinction earned its keep while this corpus was being written. Seven
of the ten expectations were wrong on first authoring — all the same mistake,
and the author's rather than the evaluator's. They assumed the attestation
axis was computed from who signed a *witness*, when §2.8 computes it from who
authored the *work*; with no work declared, attestation is absent. The fix was
not to copy the observed value into the expectation, which would have hidden
the misunderstanding, but to declare work in every case so that the axis is
actually exercised.

## Regenerating

```
JOULESPEC_GEN_EVALUATION=1 cargo test -p joulespec-core --test evaluation
```

That republishes the input files and the expectations **as the test states
them**. Without the variable the test reads the published expectation back and
asserts it still agrees with the test's own, so an expectation edited in one
place and not the other fails rather than drifting.