open-standards/jouleclaw/jouleclaw-rs/crates/jouleclaw-deberta/fixtures/tokenizer_pairs.json
Transaction Science d8ba701449 jouleclaw Phase 6: Edge-First v6 ports + 4 net-new standard-defining crates
12 crates landed (8 ports + 4 fresh designs). All 866 tests pass
across Phase 1-6; 0 failures; 52 ignored (39 inherited from
efficient-genai as hardware-gated, 12 in deberta as model-file-gated,
1 in sana_wm pre-existing efficient-genai input-size bug).

Phase 6a — Edge-First v6 surface (ported from pattern-lang)

  jouleclaw-schema       ~2137 LOC. QueryPlan, RetrievedItem,
                         KnowledgeAxes, EpistemicMode, AtomicClaim,
                         EntailmentResult, VerificationReport, Answer,
                         RAP, SystemCapabilities, AnytimeResult,
                         Invariants — the type vocabulary the rest of
                         the v6 surface is written against.

  jouleclaw-plan         ~1484 LOC. Query planner — picks the cascade
                         walk for a given input.

  jouleclaw-execute      ~2780 LOC. Plan executor with retry, anytime,
                         and budget primitives.

  jouleclaw-compose      ~1467 LOC. Verified-answer composer.

  jouleclaw-deberta      ~4801 LOC. DeBERTa-v3 forward pass for NLI
                         (used by jouleclaw-diagnose). Apple Accelerate
                         cblas_sgemm FFI gained `unsafe extern "C"` for
                         Rust 2024. Tests that require a real model on
                         disk now skip gracefully (`eprintln!("[skip]")`
                         + early return) instead of panicking — they
                         remain in-tree as integration tests and run
                         automatically when the model is present at
                         `models/deberta-v3-large-mnli/`.

  jouleclaw-diagnose     ~2353 LOC. Verification stage — runs NLI
                         against retrieved claims to certify the
                         answer.

  jouleclaw-edge-cli     ~2876 LOC. CLI binary that wires
                         schema/plan/execute/diagnose/compose/deberta
                         into a single edge agent surface.

  jouleclaw-edge-client  ~224 LOC. Thin stdin/stdout RPC client.

  Deferred: joule-edge (server) — its workspace depends on joule-l1 +
  lean-bridge + synth, which are OpenIE IP that do not port into the
  public open standard. Will land when the thin LawfulTier trait
  surface in jouleclaw-cascade ships in a follow-on commit.

Phase 6b-e — net-new (the standard-defining moat)

  jouleclaw-prov   Receipt-per-resolution emitter. CascadeTier (L0–L4
                   with stable wire tags + named tags),
                   ClaimProvenance (PROV-O compatible Entity shape),
                   ToolTouch (per-tool joule accounting),
                   ReceiptBuilder with worst-counter-floor provenance
                   semantics. Smart-Byte-envelope shaped so receipts
                   seal cleanly into the sibling standard's signed
                   transport. 6 tests passing.

  jouleclaw-fresh  L3.5 cascade stage — fresh retrieval with
                   provenance. SearchProvider trait (Brave / Tavily /
                   Exa / Serper plug in here), Transport trait (keeps
                   reqwest out of the standard), TrustTable
                   (bootstrap from Wikipedia perennial-sources;
                   normalises `www.` prefix), FreshFetch orchestrator,
                   `provenance_envelope()` that wraps a RetrievedClaim
                   into a jouleclaw-prov ClaimProvenance ready to
                   embed in a cascade receipt. 6 tests passing.

  jouleclaw-mcp    MCP tool surface with joule metering. MeteredTool
                   trait + `dispatch_metered()` brackets every tool
                   call with energy counter reads and emits a
                   ToolTouch into the running receipt. WireEncoding
                   (JsonRpc / Cbor) + `negotiate()` defines the
                   `joule-mcp` capability profile —
                   `x-jouleclaw/joule-mcp@1` is the canonical
                   negotiation tag; both endpoints advertise it to
                   opt into CBOR, otherwise stays JSON-RPC for
                   compat. 5 tests passing.

  jouleclaw-cli    `jclaw` binary — the Pi-class minimal harness.
                   v0.1 ships `version`, `tier-walk`, `receipt`,
                   `meter` subcommands plus a heuristic router that
                   picks a tier from input shape. Receipt builder is
                   live; energy counter binding is a deployment
                   choice (feature-gated per platform). Hot-reload +
                   tree-forked sessions + full MCP client land
                   later — v0.1 proves the type vocabulary and the
                   end-to-end receipt emission. 4 tests passing.

Phase 6 cumulative

  17 → 29 crates in the workspace.
  ~155k → ~170k LOC.
  440 → 866 tests passing.
  Net-new code surface: ~1500 LOC of standard-defining moat
  (jouleclaw-prov + fresh + mcp + cli).

Next: SPEC.md + CHARTER + conformance/v1 wire vectors + the
memory entry. Then promote to a public release tag.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-26 18:44:46 -04:00

579 lines
No EOL
8.9 KiB
JSON

{
"tokenizer_source": "models/deberta-v3-large-mnli",
"tokenizer_class": "DebertaV2Tokenizer",
"vocab_size": 128000,
"model_max_length": 512,
"special_tokens": {
"cls": "[CLS]",
"sep": "[SEP]",
"pad": "[PAD]",
"unk": "[UNK]",
"cls_id": 1,
"sep_id": 2,
"pad_id": 0,
"unk_id": 3
},
"pairs": [
{
"name": "simple_entail",
"premise": "Paris is the capital of France.",
"hypothesis": "France's capital is Paris.",
"token_ids": [
1,
3045,
269,
262,
1909,
265,
2378,
260,
2,
2378,
280,
268,
1909,
269,
3045,
260,
2
],
"attention_mask": [
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1
],
"token_strings": [
"[CLS]",
"▁Paris",
"▁is",
"▁the",
"▁capital",
"▁of",
"▁France",
".",
"[SEP]",
"▁France",
"'",
"s",
"▁capital",
"▁is",
"▁Paris",
".",
"[SEP]"
]
},
{
"name": "simple_contradict",
"premise": "The cat sat on the mat.",
"hypothesis": "There was no cat anywhere.",
"token_ids": [
1,
279,
3185,
3208,
277,
262,
8358,
260,
2,
443,
284,
363,
3185,
2619,
260,
2
],
"attention_mask": [
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1
],
"token_strings": [
"[CLS]",
"▁The",
"▁cat",
"▁sat",
"▁on",
"▁the",
"▁mat",
".",
"[SEP]",
"▁There",
"▁was",
"▁no",
"▁cat",
"▁anywhere",
".",
"[SEP]"
]
},
{
"name": "neutral",
"premise": "Marie Curie won the Nobel Prize in Physics in 1903.",
"hypothesis": "She also worked on radioactive materials.",
"token_ids": [
1,
8013,
60560,
722,
262,
14359,
7292,
267,
11208,
267,
25008,
260,
2,
503,
327,
1103,
277,
21194,
1375,
260,
2
],
"attention_mask": [
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1
],
"token_strings": [
"[CLS]",
"▁Marie",
"▁Curie",
"▁won",
"▁the",
"▁Nobel",
"▁Prize",
"▁in",
"▁Physics",
"▁in",
"▁1903",
".",
"[SEP]",
"▁She",
"▁also",
"▁worked",
"▁on",
"▁radioactive",
"▁materials",
".",
"[SEP]"
]
},
{
"name": "contraction_punct",
"premise": "It's the world's biggest waterfall, isn't it?",
"hypothesis": "Niagara Falls is the world's largest waterfall.",
"token_ids": [
1,
325,
280,
268,
262,
447,
280,
268,
1869,
15931,
261,
928,
280,
297,
278,
302,
2,
18722,
7167,
269,
262,
447,
280,
268,
1705,
15931,
260,
2
],
"attention_mask": [
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1
],
"token_strings": [
"[CLS]",
"▁It",
"'",
"s",
"▁the",
"▁world",
"'",
"s",
"▁biggest",
"▁waterfall",
",",
"▁isn",
"'",
"t",
"▁it",
"?",
"[SEP]",
"▁Niagara",
"▁Falls",
"▁is",
"▁the",
"▁world",
"'",
"s",
"▁largest",
"▁waterfall",
".",
"[SEP]"
]
},
{
"name": "all_upper",
"premise": "THE QUICK BROWN FOX JUMPS OVER THE LAZY DOG.",
"hypothesis": "A fox jumped over a dog.",
"token_ids": [
1,
1598,
60491,
49036,
17825,
84612,
430,
21720,
1598,
5240,
54368,
56180,
260,
2,
336,
16123,
6827,
360,
266,
1560,
260,
2
],
"attention_mask": [
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1
],
"token_strings": [
"[CLS]",
"▁THE",
"▁QUICK",
"▁BROWN",
"▁FOX",
"▁JUMP",
"S",
"▁OVER",
"▁THE",
"▁LA",
"ZY",
"▁DOG",
".",
"[SEP]",
"▁A",
"▁fox",
"▁jumped",
"▁over",
"▁a",
"▁dog",
".",
"[SEP]"
]
},
{
"name": "long_premise_short_hyp",
"premise": "Deep Space 1 was launched on October 24, 1998, as the first mission of NASA's New Millennium Program. It tested twelve advanced technologies, including the Remote Agent autonomous control system, the first time an AI system had primary command of a spacecraft.",
"hypothesis": "Deep Space 1 was launched in 1998.",
"token_ids": [
1,
7222,
4204,
376,
284,
2682,
277,
1370,
969,
261,
4286,
261,
283,
262,
362,
2151,
265,
7658,
280,
268,
485,
19898,
2659,
260,
325,
3457,
6883,
2171,
2737,
261,
500,
262,
12252,
8618,
12014,
719,
492,
261,
262,
362,
326,
299,
5536,
492,
330,
1862,
3282,
265,
266,
16394,
260,
2,
7222,
4204,
376,
284,
2682,
267,
4286,
260,
2
],
"attention_mask": [
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1
],
"token_strings": [
"[CLS]",
"▁Deep",
"▁Space",
"▁1",
"▁was",
"▁launched",
"▁on",
"▁October",
"▁24",
",",
"▁1998",
",",
"▁as",
"▁the",
"▁first",
"▁mission",
"▁of",
"▁NASA",
"'",
"s",
"▁New",
"▁Millennium",
"▁Program",
".",
"▁It",
"▁tested",
"▁twelve",
"▁advanced",
"▁technologies",
",",
"▁including",
"▁the",
"▁Remote",
"▁Agent",
"▁autonomous",
"▁control",
"▁system",
",",
"▁the",
"▁first",
"▁time",
"▁an",
"▁AI",
"▁system",
"▁had",
"▁primary",
"▁command",
"▁of",
"▁a",
"▁spacecraft",
".",
"[SEP]",
"▁Deep",
"▁Space",
"▁1",
"▁was",
"▁launched",
"▁in",
"▁1998",
".",
"[SEP]"
]
}
]
}