jouleclaw: a claim is judged on what it cites, so a trusted source cannot be credited with what a weak one says #355

Merged

dcharlot merged 1 commit from fix/claims-grounded-by-their-own-citations into main 2026-10-20 08:28:14 -04:00

Conversation 0 Commits 1 Files changed 6 +515 -14

dcharlot commented 2026-10-20 04:27:45 -04:00

Owner

Copy link

What was open

jouleclaw-diagnose decided grounding and authority without reading what an answer cites.

• Grounding passed if any retrieved item entailed the claim.
• Authority passed if any retrieved item for the sub-query met the tier.
• cited_item_ids appeared nowhere in the crate.

So an answer could cite a primary source that does not support a claim, while the only support was a forum post it did not cite, and it passed both checks: grounded by the post, authoritative by the source. That is citation laundering. One weak source makes the claim, and a trusted one is credited with it.

The report also listed I1 (every claim has provenance) and I4 (cited sources meet the tier) as verified on every run, though nothing checked either.

#344 fixed which cited item compose credits, choosing the most authoritative one over the first. It did not check that this item supports the claim, so the displayed source could still be the laundered one.

What this changes

• citation_violations (diagnose). Each claim is judged on its own segment's citations. "Supports" means what grounding means: every model that judged the pair entailed it.
  • A supported claim that cites nothing is uncited (Major).
  • A claim supported only by items it does not cite is misattributed (Major).
  • It is laundering (Critical, not retrievable, so refused) when it also cites an item more authoritative than any that supports it.
  • Authority is judged on the most authoritative cited item that supports the claim. If even that falls short, an authority violation scoped to the claim is raised (Critical, retrievable).
• Recovery drops laundering and hedges the rest.
• Inputs. VerifyInputs::with_citations supplies the citations, and the edge pipeline passes the draft's at both verify calls.
• I1 and I4 are listed only when citations were judged. They used to be listed on every run while nothing read a citation.
• Compose attributes a segment to the most authoritative cited item that supports its claim, wherever the verifier judged that pair.
  • If no cited item supports the claim, the segment is attributed to none, with a caveat saying so.
  • If nothing was judged, the most authoritative cited item stands, as before.

Behaviour to know

An answer whose claim is supported only by a retrieved item it does not cite used to proceed. It now degrades (misattributed, hedged) or is refused (laundering). This is intended.

Verification (build host)

• Tests. jouleclaw-diagnose: 54 lib tests and 3 integration tests pass. jouleclaw-compose: 29. jouleclaw-edge-cli: 24, with its 4 end-to-end tests ignored as before (they need model weights). 8 of these tests are new, including the attack end to end through verify, with and without citations.
• Mutations. All 11 mutations of the new logic are killed. Each disables one piece:
  • citations ignored
  • laundering never flagged
  • any support counted as cited
  • authority judged on the best cited item rather than the supporting one
  • support by any model rather than every model (this survived the first round; the test was strengthened)
  • an uncited claim passed
  • I1 and I4 always claimed
  • laundering not refused
  • the recovery mapping removed
  • compose ignoring support
  • compose's missing caveat
• Clippy reports two pre-existing findings in untouched code: a test helper in valid_answer.rs, and jouleclaw-edge-cli/src/understanding.rs. Neither crate is in CI's clippy gate.
• Repo checks pass: tests-in-ci, spec-refs, orphan-sources, vanity-metrics and comparative-prose.

🤖 Generated with Claude Code

## What was open jouleclaw-diagnose decided grounding and authority without reading what an answer cites. - **Grounding** passed if *any* retrieved item entailed the claim. - **Authority** passed if *any* retrieved item for the sub-query met the tier. - **`cited_item_ids`** appeared nowhere in the crate. So an answer could cite a primary source that does not support a claim, while the only support was a forum post it did not cite, and it passed both checks: grounded by the post, authoritative by the source. That is citation laundering. One weak source makes the claim, and a trusted one is credited with it. The report also listed **I1** (every claim has provenance) and **I4** (cited sources meet the tier) as verified on every run, though nothing checked either. #344 fixed which cited item compose credits, choosing the most authoritative one over the first. It did not check that this item supports the claim, so the displayed source could still be the laundered one. ## What this changes - **`citation_violations` (diagnose).** Each claim is judged on its own segment's citations. "Supports" means what grounding means: every model that judged the pair entailed it. - A supported claim that cites nothing is `uncited` (Major). - A claim supported only by items it does not cite is `misattributed` (Major). - It is `laundering` (Critical, not retrievable, so **refused**) when it also cites an item more authoritative than any that supports it. - Authority is judged on the most authoritative cited item that supports the claim. If even that falls short, an `authority` violation scoped to the claim is raised (Critical, retrievable). - **Recovery** drops laundering and hedges the rest. - **Inputs.** `VerifyInputs::with_citations` supplies the citations, and the edge pipeline passes the draft's at both `verify` calls. - **I1 and I4** are listed only when citations were judged. They used to be listed on every run while nothing read a citation. - **Compose** attributes a segment to the most authoritative cited item that supports its claim, wherever the verifier judged that pair. - If no cited item supports the claim, the segment is attributed to none, with a caveat saying so. - If nothing was judged, the most authoritative cited item stands, as before. ## Behaviour to know An answer whose claim is supported only by a retrieved item it does not cite used to proceed. It now degrades (`misattributed`, hedged) or is refused (`laundering`). This is intended. ## Verification (build host) - **Tests.** jouleclaw-diagnose: 54 lib tests and 3 integration tests pass. jouleclaw-compose: 29. jouleclaw-edge-cli: 24, with its 4 end-to-end tests ignored as before (they need model weights). 8 of these tests are new, including the attack end to end through `verify`, with and without citations. - **Mutations.** All 11 mutations of the new logic are killed. Each disables one piece: - citations ignored - laundering never flagged - any support counted as cited - authority judged on the best cited item rather than the supporting one - support by any model rather than every model (this survived the first round; the test was strengthened) - an uncited claim passed - I1 and I4 always claimed - laundering not refused - the recovery mapping removed - compose ignoring support - compose's missing caveat - **Clippy** reports two pre-existing findings in untouched code: a test helper in `valid_answer.rs`, and `jouleclaw-edge-cli/src/understanding.rs`. Neither crate is in CI's clippy gate. - **Repo checks** pass: tests-in-ci, spec-refs, orphan-sources, vanity-metrics and comparative-prose. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 1 commit 2026-10-20 04:27:45 -04:00

jouleclaw: a claim is judged on what it cites, so a trusted source cannot be credited with what a weak one says ... e541ab6d79

jouleclaw-diagnose checked grounding and authority without reading what the
answer cites:
- grounding passed if ANY retrieved item entailed the claim;
- authority passed if ANY retrieved item for the sub-query met the tier;
- `cited_item_ids` appeared nowhere in the crate.

So an answer could cite a primary source that does not support a claim,
supported only by an uncited forum post, and pass both: grounded by the post,
authoritative by the source. That is citation laundering. One weak source
makes the claim, and a trusted one is credited with it. The report also
listed I1 (every claim has provenance) and I4 (cited sources meet the tier)
as verified on every run, though nothing checked either.

`citation_violations` now judges each claim on its own segment's citations,
using the same notion of support as grounding (every model that judged the
pair entailed it):
- a supported claim that cites nothing is `uncited` (Major);
- one supported only by items it does not cite is `misattributed` (Major);
- it is `laundering` (Critical, not retrievable, so the answer is refused)
  when it also cites an item more authoritative than any that supports it;
- authority is judged on the most authoritative cited item that supports the
  claim: an `authority` violation scoped to the claim (Critical,
  retrievable) when even that one falls short.

Recovery drops laundering and hedges the rest. `VerifyInputs::with_citations`
supplies the citations, and the edge pipeline passes the draft's at both
verify calls. I1 and I4 are listed only when the citations were judged.

jouleclaw-compose attributed a segment to its most authoritative cited item
(#344) whether or not that item supported the claim, so the displayed source
could still be the laundered one. Where the verifier judged the claim
against its citations, the attribution now goes to the most authoritative
cited item that supports it. Where none does, the segment is attributed to
none, with a caveat saying so. Where nothing was judged, the most
authoritative cited item stands as before.

Tests: 8 new across diagnose and compose, including the attack end to end
through `verify`. All 11 mutations of the new logic are killed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit 3f4de5864d into main 2026-10-20 08:28:14 -04:00

dcharlot referenced this pull request from a commit 2026-10-20 08:28:16 -04:00

Merge pull request 'jouleclaw: a claim is judged on what it cites, so a trusted source cannot be credited with what a weak one says' (#355) from fix/claims-grounded-by-their-own-citations into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!355

No description provided.