diagnose: a verification verdict must say which instrument produced it #91

Merged

dcharlot merged 1 commit from diagnose/grounding-backing into main 2026-09-10 21:15:24 -04:00

Conversation 0 Commits 1 Files changed 7 +497 -7

dcharlot commented 2026-09-10 12:18:25 -04:00

Owner

Copy link

The grounding labels under a VerificationReport come from a learned model — a DeBERTa-v3 NLI head, per §6.4. That is the right instrument for the job: whether prose entails a claim is not decidable, and a calibrated judgement is the honest tool. The problem was never the model. It was that nothing downstream had to notice there was one.

What was already right

Worth stating, because the finding that prompted this overstated it: the provenance was already carried. EntailmentResult records model_id, calibrated label_probabilities and joules_spent; Provenance.entailments preserves all of it into the sealed artifact; FixtureEntailer already names itself. And grounding_violations() is a pure function over supplied results — it does not call a model. This is not the "claim as its own evidence" shape of the proof T3 stand-in.

What was actually missing

The other half — the family's discipline call. jcp-action-receipt has check_discipline, proof-cascade has require_substantive_proof, sandbox-attest has require_mechanism_evidence. The entailment path had no require_* at all. Carried provenance that nothing can be made to check is a record, not a control.

So: Backing on the report, and require_substantive_grounding(). Unstated, Unconsulted and Fixture are refused; SingleModel, Concurring and Contested pass. require_grounding_from additionally pins a model.

Three decisions carry the weight

The default is the floor. Entailer::backing() defaults to Unstated, which is refused — an implementation that forgets under-claims instead of over-claiming, the same discipline as Vmm::attained_tier returning S0.

A declaration cannot outrun its evidence. The check cross-references the declared backing against the model_ids actually in entailments_consulted. A report claiming two concurring models while carrying one model's results is refused: asserting corroboration a reader cannot obtain from the artifact is worse than asserting nothing.

Cache rehydration declares the floor, not a reconstructed backing. The cached entailments carry model_ids, so a backing could be derived — and that is exactly the laundering path: a FixtureEntailer run records model_id "fixture-entailer", and reconstructing SingleModel from it would promote a stand-in into a substantive backing on the way out of the cache. Evidence that a model ran is not a claim about what its labels are worth. The consequence is stated rather than hidden — rehydrated reports are refused, and the fix is persisting backing in Provenance, a cache-format change this PR deliberately does not make.

Compatibility

grounding_backing is Option + skip_serializing_if = "Option::is_none", and a test asserts it never reaches the wire when unset — so reports and vectors written before it existed serialize byte-identically and keep verifying. require_* is opt-in, so no existing consumer changes behaviour until it asks.

The sharpest test

a_clean_verdict_from_a_fixture_is_still_not_grounded: an existing fixture-backed report with no violations, verdict ProceedToComposition and invariant I2 verified — every previously-checkable signal saying "verified", and all of them true — is now refused as grounded. The verdict is about the labels; the backing is about whether the labels are worth anything.

Verification

475 test binaries green across the workspace, 9 new. All 7 dependents of jouleclaw-schema are in-workspace — checked rather than assumed, since this repo's separate workspaces mean --workspace does not always cover a shared-schema change.

🤖 Generated with Claude Code

The grounding labels under a `VerificationReport` come from a learned model — a DeBERTa-v3 NLI head, per §6.4. **That is the right instrument for the job**: whether prose entails a claim is not decidable, and a calibrated judgement is the honest tool. The problem was never the model. It was that nothing downstream had to notice there was one. ## What was already right Worth stating, because the finding that prompted this overstated it: the provenance was already **carried**. `EntailmentResult` records `model_id`, calibrated `label_probabilities` and `joules_spent`; `Provenance.entailments` preserves all of it into the sealed artifact; `FixtureEntailer` already names itself. And `grounding_violations()` is a pure function over supplied results — it does not call a model. This is *not* the "claim as its own evidence" shape of the proof T3 stand-in. ## What was actually missing The other half — the family's discipline call. `jcp-action-receipt` has `check_discipline`, `proof-cascade` has `require_substantive_proof`, `sandbox-attest` has `require_mechanism_evidence`. The entailment path had **no `require_*` at all**. Carried provenance that nothing can be made to check is a record, not a control. So: `Backing` on the report, and `require_substantive_grounding()`. `Unstated`, `Unconsulted` and `Fixture` are refused; `SingleModel`, `Concurring` and `Contested` pass. `require_grounding_from` additionally pins a model. ## Three decisions carry the weight **The default is the floor.** `Entailer::backing()` defaults to `Unstated`, which is refused — an implementation that forgets under-claims instead of over-claiming, the same discipline as `Vmm::attained_tier` returning S0. **A declaration cannot outrun its evidence.** The check cross-references the declared backing against the `model_id`s actually in `entailments_consulted`. A report claiming two concurring models while carrying one model's results is refused: asserting corroboration a reader cannot obtain from the artifact is worse than asserting nothing. **Cache rehydration declares the floor, not a reconstructed backing.** The cached entailments carry `model_id`s, so a backing *could* be derived — and that is exactly the laundering path: a `FixtureEntailer` run records model_id `"fixture-entailer"`, and reconstructing `SingleModel` from it would promote a stand-in into a substantive backing on the way out of the cache. Evidence that a model ran is not a claim about what its labels are worth. The consequence is stated rather than hidden — rehydrated reports are refused, and the fix is persisting backing in `Provenance`, a cache-format change this PR deliberately does not make. ## Compatibility `grounding_backing` is `Option` + `skip_serializing_if = "Option::is_none"`, and a test asserts it never reaches the wire when unset — so reports and vectors written before it existed serialize byte-identically and keep verifying. `require_*` is opt-in, so no existing consumer changes behaviour until it asks. ## The sharpest test `a_clean_verdict_from_a_fixture_is_still_not_grounded`: an existing fixture-backed report with no violations, verdict `ProceedToComposition` and invariant I2 verified — every previously-checkable signal saying "verified", and all of them true — is now refused as grounded. The verdict is about the labels; the backing is about whether the labels are worth anything. ## Verification 475 test binaries green across the workspace, 9 new. All 7 dependents of `jouleclaw-schema` are in-workspace — checked rather than assumed, since this repo's separate workspaces mean `--workspace` does not always cover a shared-schema change. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 1 commit 2026-09-10 12:18:26 -04:00

diagnose: a verification verdict must say which instrument produced it ... 84e1ef158c

The grounding labels under a `VerificationReport` come from a learned model — a
DeBERTa-v3 NLI head, per §6.4. That is the right instrument for the job: whether
prose entails a claim is not decidable, and a calibrated judgement is the honest
tool. The problem was never the model. It was that nothing downstream had to
notice there was one.

The provenance was already carried: `EntailmentResult` records `model_id`,
calibrated `label_probabilities` and `joules_spent`, `Provenance.entailments`
preserves the lot into the sealed artifact, and `FixtureEntailer` already names
itself. What was missing is the other half — the family's discipline call.
`jcp-action-receipt` has `check_discipline`, `proof-cascade` has
`require_substantive_proof`, `sandbox-attest` has `require_mechanism_evidence`;
the entailment path had no `require_*` at all. Carried provenance that nothing
can be made to check is a record, not a control.

So: `Backing` on the report, and `require_substantive_grounding()` to enforce it.
`Unstated`, `Unconsulted` and `Fixture` are refused; `SingleModel`, `Concurring`
and `Contested` pass. `require_grounding_from` additionally pins a model, for a
caller that has decided which instrument it trusts rather than only that one ran.

Three decisions carry the weight:

**The default is the floor.** `Entailer::backing()` defaults to `Unstated`, which
is refused — so an implementation that forgets under-claims instead of
over-claiming, the same discipline as `Vmm::attained_tier` returning S0.

**A declaration cannot outrun its evidence.** The check cross-references the
declared backing against the `model_id`s actually present in
`entailments_consulted`. A report claiming two concurring models while carrying
one model's results is refused: asserting corroboration a reader cannot obtain
from the artifact is worse than asserting nothing.

**Cache rehydration declares the floor, not a reconstructed backing.** The cached
entailments carry `model_id`s, so a backing *could* be derived from them — and
that is precisely the laundering path: a run backed by `FixtureEntailer` records
model_id "fixture-entailer", and reconstructing `SingleModel` from it would
promote a stand-in into a substantive backing on the way out of the cache.
Evidence that a model ran is not a claim about what its labels are worth. The
consequence is stated rather than hidden — a rehydrated report is refused, and
the fix is persisting the backing in `Provenance`, a cache-format change this
commit deliberately does not make.

`grounding_backing` is `Option`, `skip_serializing_if = "Option::is_none"`, and a
test asserts it never reaches the wire when unset, so reports and vectors written
before it existed serialize byte-identically and keep verifying.

The sharpest test is `a_clean_verdict_from_a_fixture_is_still_not_grounded`: an
existing fixture-backed report with no violations, verdict
`ProceedToComposition` and invariant I2 verified — every previously-checkable
signal saying "verified", and all of them true — is now refused as grounded. The
verdict is about the labels; the backing is about whether the labels are worth
anything.

Verified: 475 test binaries green across the workspace, 9 new. All 7 dependents
of `jouleclaw-schema` are in-workspace, checked rather than assumed, so
`--workspace` genuinely covers the blast radius of the schema change.

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot merged commit daf9766d1c into main 2026-09-10 21:15:24 -04:00

dcharlot referenced this pull request from a commit 2026-09-10 21:15:26 -04:00

diagnose: a verification verdict must say which instrument produced it (#91)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!91

No description provided.