Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 15 Packages Wiki Activity Actions

wai-quantum: a study of the co-design, every measure the engine's #509

Closed
dcharlot wants to merge 1 commit from wai/quantum-study into main
pull from: wai/quantum-study
merge into: Transaction-Science:main
Conversation 0 Commits 1 Files changed 21 +1482 -28
dcharlot commented 2026-10-27 06:55:41 -04:00
Owner
Copy link

The measurement harness the co-design report puts under every feature. Each feature sits behind a flag and is compared against a control, in a study whose protocol is sealed before any data. Every measure is the engine's, computed again from what the participant made; none is a model's judgement.

Engine (engine::study)

  • Protocol: the study, its hypotheses, the primary measure, the conditions (each a setting of the Studio's features: study::FLAGS), the control, and the analysis (resamples, seed, level). It is checked, then sealed with the researcher's key before any data.
  • Assignment: a hash of the protocol's digest and the participant's key. The condition is fixed before they begin, and anyone can check it.
  • Records: the participant's record, sealed with their key. A record is refused if it was changed, made for another study, sealed by another key, or in a condition its key is not assigned.
  • The eight measures:
    • quality of the edits proved equal to what they replaced, and its trend;
    • diversity (Vendi) and novelty;
    • selection skill, against chance;
    • calibration: the mean distance from the run, and the slope;
    • unassisted skill: four probe tasks, each scored by a workbench specification;
    • ownership: five items, 1 to 7, beside the authorship share;
    • convergence on a model's proposals.
  • Cohort report: per condition, the means; each condition's difference from the control on the primary measure, with a seeded bootstrap interval; Cronbach's alpha of the ownership items; the Vendi score of the final designs; one record per participant.

CLI and MCP

  • wai-quantum study catalog | new | assign | measure | cohort, with --out to seal the report.
  • MCP: study_measure and study_cohort. Probes are not offered to agents, because a probe is done without the assistant.

Studio: the Study pane

  • Features: each can be switched off. A condition holds only the features it names, and leaving the study restores the reader's own settings.
  • Load a protocol: it is shown before "Take part".
  • A record is kept only once the reader asks for one:
    • choices in Alternatives and the Workbench, with the engine's best;
    • "How much is this yours?";
    • probe tasks, done with the assistant paused.
  • "My record", and "Export my record", sealed with the browser's key.

Checked on the build host, at low priority:

  • engine engine::: 119 pass, 4 of them new;
  • CLI: 19 pass, clippy clean with default and no default features;
  • wai-web native engine tests: 14 pass;
  • desktop: 5 pass, clippy clean;
  • Studio e2e: 145 checks pass, 13 of them new. A sealed protocol is loaded and joined, the condition is checked against the key, and the exported record is checked.

The CLI sealed a real protocol and listed its catalog. The public-names check is clean.

Two e2e fixes the new pane made necessary:

  • The editor stretches to the side column's height, so the test now brings the editor's first lines into view before reading its gutter.
  • The taste checks now count from the choices the profile already holds.

🤖 Generated with Claude Code

The measurement harness the co-design report puts under every feature. Each feature sits behind a flag and is compared against a control, in a study whose protocol is sealed before any data. Every measure is the engine's, computed again from what the participant made; none is a model's judgement. **Engine (`engine::study`)** - **Protocol:** the study, its hypotheses, the primary measure, the conditions (each a setting of the Studio's features: `study::FLAGS`), the control, and the analysis (resamples, seed, level). It is checked, then sealed with the researcher's key before any data. - **Assignment:** a hash of the protocol's digest and the participant's key. The condition is fixed before they begin, and anyone can check it. - **Records:** the participant's record, sealed with their key. A record is refused if it was changed, made for another study, sealed by another key, or in a condition its key is not assigned. - **The eight measures:** - quality of the edits proved equal to what they replaced, and its trend; - diversity (Vendi) and novelty; - selection skill, against chance; - calibration: the mean distance from the run, and the slope; - unassisted skill: four probe tasks, each scored by a workbench specification; - ownership: five items, 1 to 7, beside the authorship share; - convergence on a model's proposals. - **Cohort report:** per condition, the means; each condition's difference from the control on the primary measure, with a seeded bootstrap interval; Cronbach's alpha of the ownership items; the Vendi score of the final designs; one record per participant. **CLI and MCP** - `wai-quantum study catalog | new | assign | measure | cohort`, with `--out` to seal the report. - MCP: `study_measure` and `study_cohort`. Probes are not offered to agents, because a probe is done without the assistant. **Studio: the Study pane** - Features: each can be switched off. A condition holds only the features it names, and leaving the study restores the reader's own settings. - Load a protocol: it is shown before "Take part". - A record is kept only once the reader asks for one: - choices in Alternatives and the Workbench, with the engine's best; - "How much is this yours?"; - probe tasks, done with the assistant paused. - "My record", and "Export my record", sealed with the browser's key. **Checked on the build host**, at low priority: - engine `engine::`: 119 pass, 4 of them new; - CLI: 19 pass, clippy clean with default and no default features; - wai-web native engine tests: 14 pass; - desktop: 5 pass, clippy clean; - Studio e2e: 145 checks pass, 13 of them new. A sealed protocol is loaded and joined, the condition is checked against the key, and the exported record is checked. The CLI sealed a real protocol and listed its catalog. The public-names check is clean. Two e2e fixes the new pane made necessary: - The editor stretches to the side column's height, so the test now brings the editor's first lines into view before reading its gutter. - The taste checks now count from the choices the profile already holds. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
wai-quantum: a study of the co-design, every measure the engine's
Some checks failed
wai-quantum npu-cores member / member · npu-cores absent (linux-arm64) (pull_request) Successful in 32s
Details
wai-quantum tensor-engine member / member · tensor engine absent (linux-arm64) (pull_request) Successful in 23s
Details
WAI conformance / each feature of the independent verifiers' steps compiles alone (pull_request) Successful in 11m54s
Details
wai-sink-android native sink library / the JNI layer and the platform-free half on the host JVM · linux-arm64 (pull_request) Successful in 2m47s
Details
WAI energy receipts / energy receipts · linux-arm64 (pull_request) Successful in 10m24s
Details
wai-quantum WebGPU member / member · discrete GPU, Vulkan (jetson-hub) (pull_request) Successful in 25m29s
Details
wai-audit collector and player integration / collector, player integration and the browser end to end · macos (pull_request) Successful in 1m40s
Details
WAI conformance / cross-arch byte-equality · macos (pull_request) Successful in 27s
Details
WAI conformance / codec wrappers · fuzz smoke, text decode limits, C entry points, provenance, the semantic path (pull_request) Successful in 1m25s
Details
WAI conformance / cargo-class corpora · linux-arm64 (pull_request) Successful in 21m23s
Details
WAI energy receipts / energy receipts · macos (pull_request) Successful in 8m6s
Details
standards workspaces build + test / macOS energy backends (pull_request) Successful in 12m38s
Details
wai-quantum command line / command line · macos (pull_request) Successful in 46s
Details
wai-quantum dataflow member / member · dataflow device absent (macos) (pull_request) Successful in 15s
Details
wai-quantum FPGA member / member · FPGA datapath, simulated (macos) (pull_request) Successful in 1m18s
Details
wai-quantum Metal member / member · Metal (macos) (pull_request) Successful in 36s
Details
WAI conformance / cargo-class corpora · macos (pull_request) Successful in 21m48s
Details
wai-quantum OpenCL member / member · OpenCL GPU (macos) (pull_request) Successful in 23s
Details
WAI examples / WAI examples · the archive, the readiness check, the browser and Apple · macos (pull_request) Successful in 12m58s
Details
wai-quantum service / service · macos (pull_request) Successful in 47s
Details
wai-quantum matrix-extension member / member · matrix extension absent (macos) (pull_request) Successful in 11s
Details
wai-quantum portable tensor program / tensor program · Metal and the arm64 CPU (macos) (pull_request) Successful in 17s
Details
wai-quantum CPU vector units / member · vector unit absent (macos) (pull_request) Successful in 15s
Details
wai-quantum neural-processor member / member · neural processor (macos) (pull_request) Successful in 11m6s
Details
wai-quantum studio and desktop / desktop app · macos (pull_request) Successful in 2m47s
Details
wai-quantum WebGPU member / member · Metal (macos) (pull_request) Successful in 2m49s
Details
wai-quantum WebGPU member / member · browser WebGPU (macos) (pull_request) Successful in 1m50s
Details
WAI conformance / independent verifiers · claims, JWP, key discovery and integer payloads (Python) (pull_request) Failing after 52m29s
Details
wai-sink-android native sink library / every ABI, the AAR, and the instrumented tests in a headless emulator · macos (pull_request) Successful in 9m12s
Details
wai-sink-swift native sink package / libwai slices, desktop tests, simulator tests · macos (pull_request) Successful in 22m13s
Details
8a48b36ee8 
The measurement harness the co-design report puts under every feature:
each feature behind a flag, compared against a control, in a study whose
protocol is sealed before any data.

- engine::study:
  - Protocol: the study, hypotheses, primary measure, conditions (each a
    setting of FLAGS), control and analysis (resamples, seed, level),
    checked and sealed with the researcher's key before any data;
  - assign: the protocol's digest and the participant's key hashed
    together, so the condition is fixed and anyone can check it;
  - seal_record / open_record: the participant's record sealed with their
    key, refused if changed, for another study, sealed by another key, or
    in a condition its key is not assigned;
  - measure: the report's eight, each computed again from the record:
    quality of the edits proved equal (and its trend), diversity (Vendi),
    novelty, selection skill against chance, calibration (mean distance,
    slope), unassisted probe scores (four probes, each a workbench
    specification), ownership (five items, 1 to 7) beside the authorship
    share, and convergence;
  - cohort: per condition the means, each condition's difference from the
    control on the primary measure with a seeded bootstrap interval,
    Cronbach's alpha, the Vendi score of the final designs, one record per
    participant; call and call_sealing for hosts.
- The command line: study catalog | new | assign | measure | cohort
  (--out seals the report).
- MCP: study_measure and study_cohort. The probes are not offered to
  agents: a probe is done without the assistant.
- The Studio: the Study pane (Features, each switchable off and held where
  a study's condition names it; Load a protocol, shown before Take part; a
  record kept once asked: choices in Alternatives and the Workbench with
  the engine's best, How much is this yours?, probe tasks with the
  assistant paused; My record; Export my record, sealed with the browser's
  key). A condition holds only the features it names; leaving restores the
  reader's own.
- Bindings: QuantumEngine::study in the page (it seals with the page's
  key), study in the desktop app.
- CI: wai-web's native engine tests count 14.
- The e2e brings the editor's first lines into view before reading its
  gutter, and counts the taste profile from the choices it already holds.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
dcharlot closed this pull request 2026-10-27 09:58:18 -04:00
Some checks failed
wai-quantum on every processor / emulated · aarch64-unknown-linux-gnu (pull_request) Failing after 10s
Details
wai-quantum on every processor / emulated · armv7-unknown-linux-gnueabihf (pull_request) Failing after 3s
Details
wai-quantum on every processor / emulated · loongarch64-unknown-linux-gnu (pull_request) Failing after 3s
Details
wai-quantum on every processor / emulated · powerpc64-unknown-linux-gnu (pull_request) Failing after 3s
Details
wai-quantum on every processor / emulated · powerpc64le-unknown-linux-gnu (pull_request) Failing after 3s
Details
wai-quantum on every processor / emulated · riscv64gc-unknown-linux-gnu (pull_request) Failing after 3s
Details
wai-quantum on every processor / emulated · s390x-unknown-linux-gnu (pull_request) Failing after 3s
Details
wai-quantum on every processor / emulated · sparc64-unknown-linux-gnu (pull_request) Failing after 3s
Details
wai-quantum on every processor / emulated · i686-unknown-linux-gnu (pull_request) Failing after 3s
Details
wai-web reach builds / wai-web reach · does this change touch it (pull_request) Successful in 5s
Details
wai-web reach builds / wai-web reach builds · the package of record, every output as recorded (pull_request) Successful in 2m25s
Details
wai-web reach builds / wai-web reach builds · this tree's wai-web, every rung on its engine line (pull_request) Successful in 4m20s
Details
wai-quantum WebGPU member / member · DX12 and Vulkan (cbio-windows) (pull_request) Successful in 12m10s
Details
standards workspaces build + test / which workspaces this change touches (pull_request) Successful in 8s
Details
wai-audit collector and player integration / does this change touch it (pull_request) Successful in 7s
Details
WAI conformance / does this change touch it (pull_request) Successful in 7s
Details
WAI energy receipts / does this change touch it (pull_request) Successful in 8s
Details
WAI examples / WAI examples · does this change touch them (pull_request) Successful in 7s
Details
wai-quantum command line / does this change touch it (pull_request) Successful in 9s
Details
wai-quantum CUDA member / does this change touch it (pull_request) Successful in 6s
Details
WAI conformance / cross-arch byte-equality · linux (pull_request) Successful in 36s
Details
wai-quantum dataflow member / does this change touch it (pull_request) Successful in 11s
Details
wai-quantum signal processor / does this change touch it (pull_request) Successful in 7s
Details
wai-audit collector and player integration / collector · linux (pull_request) Successful in 1m11s
Details
wai-quantum FPGA member / does this change touch it (pull_request) Successful in 9s
Details
wai-quantum CUDA member / member · CUDA (jetson-hub) (pull_request) Successful in 28s
Details
wai-quantum HIP member / does this change touch it (pull_request) Successful in 8s
Details
wai-quantum Metal member / does this change touch it (pull_request) Successful in 8s
Details
wai-quantum npu-cores member / does this change touch it (pull_request) Successful in 7s
Details
wai-quantum neural-processor member / does this change touch it (pull_request) Successful in 7s
Details
wai-quantum OpenCL member / does this change touch it (pull_request) Successful in 6s
Details
wai-quantum-qpu / does this change touch it (pull_request) Successful in 7s
Details
WAI conformance / faces from keypoints under wasm32-wasip2 (wasmtime) (pull_request) Successful in 1m53s
Details
wai-quantum service / does this change touch it (pull_request) Successful in 6s
Details
wai-quantum matrix-extension member / does this change touch it (pull_request) Successful in 7s
Details
wai-quantum studio and desktop / does this change touch it (pull_request) Successful in 7s
Details
wai-quantum tensor-engine member / does this change touch it (pull_request) Successful in 6s
Details
repository checks / lockfiles, advisories, tests in CI, workflow hygiene, spec and section references, public text (pull_request) Successful in 2m47s
Details
wai-quantum portable tensor program / does this change touch it (pull_request) Successful in 6s
Details
wai-quantum OpenCL member / member · OpenCL GPU and CPU (jetson-hub) (pull_request) Successful in 26s
Details
wai-quantum CPU vector units / does this change touch it (pull_request) Successful in 7s
Details
wai-quantum WebGPU member / does this change touch it (pull_request) Successful in 7s
Details
wai-sink-android native sink library / does this change touch it (pull_request) Successful in 6s
Details
WAI conformance / quantum core and corpus under wasm32-wasip2 (wasmtime) (pull_request) Successful in 2m20s
Details
wai-quantum CUDA member / member · CUDA absent (linux) (pull_request) Successful in 28s
Details
wai-sink-swift native sink package / does this change touch it (pull_request) Successful in 7s
Details
WAI mbt2018_mean integer decode (regenerated vectors) / does this change touch it (pull_request) Successful in 7s
Details
wai-web wasm32 / does this change touch it (pull_request) Successful in 7s
Details
wai-quantum portable tensor program / tensor program · GPUs (jetson-hub) (pull_request) Successful in 21s
Details
wai-quantum dataflow member / member · dataflow device, simulated (pci 1e52:b140) (pull_request) Failing after 18s
Details
wai-quantum command line / command line · linux (pull_request) Successful in 1m38s
Details
wai-quantum FPGA member / member · FPGA simulation absent (linux) (pull_request) Successful in 18s
Details
WAI conformance / fast integer kernels, every x86_64 kernel set (AVX-512 silicon) (pull_request) Successful in 3m7s
Details
wai-quantum signal processor / processor and member · signal processor, emulated (pull_request) Failing after 33s
Details
wai-quantum Metal member / member · Metal absent (linux) (pull_request) Successful in 25s
Details
wai-quantum HIP member / member · HIP (framework-128gb) (pull_request) Successful in 24s
Details
wai-quantum neural-processor member / member · neural processor absent (linux) (pull_request) Successful in 19s
Details
wai-quantum OpenCL member / member · OpenCL absent (linux) (pull_request) Successful in 23s
Details
wai-quantum-qpu / clients against their contracts (linux) (pull_request) Successful in 22s
Details
wai-quantum matrix-extension member / member · matrix extension, emulated (aarch64 Linux) (pull_request) Failing after 4s
Details
wai-quantum service / service · linux (pull_request) Successful in 47s
Details
wai-quantum npu-cores member / member · npu-cores, neural processor (framework-128gb silicon) (pull_request) Successful in 1m18s
Details
wai-quantum dataflow member / member · dataflow device, simulated (pci 1e52:401e) (pull_request) Successful in 2m43s
Details
wai-quantum tensor-engine member / member · tensor engine, the kernel toolchain's simulator (linux) (pull_request) Failing after 26s
Details
wai-quantum OpenCL member / member · OpenCL GPU (framework-128gb) (pull_request) Successful in 16s
Details
wai-quantum CPU vector units / member · vector unit, rvv 1024-bit (riscv64, emulated) (pull_request) Failing after 3s
Details
wai-quantum CPU vector units / member · vector unit, sve 128-bit (aarch64, emulated) (pull_request) Failing after 4s
Details
wai-quantum CPU vector units / member · vector unit, rvv 128-bit (riscv64, emulated) (pull_request) Failing after 3s
Details
wai-quantum portable tensor program / tensor program · integrated GPU (framework-128gb) (pull_request) Successful in 9s
Details
wai-quantum CPU vector units / member · vector unit, sve 2048-bit (aarch64, emulated) (pull_request) Failing after 4s
Details
wai-quantum CPU vector units / member · vector unit, avx2 (x86_64 silicon) (pull_request) Failing after 3s
Details
wai-quantum CPU vector units / member · vector unit, sve 512-bit (aarch64, emulated) (pull_request) Failing after 3s
Details
wai-quantum CPU vector units / member · vector units, avx2 and avx512 (framework-128gb silicon) (pull_request) Successful in 12s
Details
wai-quantum portable tensor program / tensor program · CPU, mutants, GPU builds (pull_request) Successful in 40s
Details
wai-quantum studio and desktop / studio · browser (pull_request) Successful in 3m2s
Details
WAI examples / WAI examples · portable · linux (pull_request) Successful in 7m22s
Details
wai-quantum WebGPU member / member · software Vulkan (linux) (pull_request) Successful in 3m0s
Details
wai-quantum WebGPU member / shader checks · browser build · wasip2 inertness · MSRV (pull_request) Successful in 3m16s
Details
WAI energy receipts / energy receipts · linux (pull_request) Successful in 11m8s
Details
WAI conformance / v2 vocabulary · manifests validate as stated (pull_request) Successful in 23s
Details
WAI mbt2018_mean integer decode (regenerated vectors) / regenerate mbt2018_mean integer vectors → Rust decode, required (pull_request) Successful in 5m47s
Details
WAI conformance / delivery telemetry · CMCD and CMSD against an independent library (pull_request) Successful in 20s
Details
wai-web wasm32 / wai-web wasm32 build · corpus, menu and writers (pull_request) Successful in 5m55s
Details
WAI conformance / cross-arch byte-equality · linux-arm64 (pull_request) Successful in 37s
Details
wai-quantum WebGPU member / member · integrated GPU, Vulkan (framework-128gb) (pull_request) Successful in 8m31s
Details
standards workspaces build + test / joule-code-rs (joule-code/joule-code-rs, joule-code-rs, linux-arm64, true) (pull_request) Failing after 5m36s
Details
WAI conformance / C2PA manifest emission · signed, read back, validated (pull_request) Successful in 1m57s
Details
WAI conformance / cargo-class corpora · linux (pull_request) Successful in 17m6s
Details
WAI examples / WAI examples · android-app · linux-arm64 (pull_request) Successful in 4m43s
Details
wai-quantum HIP member / member · HIP absent (linux-arm64) (pull_request) Successful in 30s
Details
wai-quantum npu-cores member / member · npu-cores absent (linux-arm64) (pull_request) Successful in 32s
Details
wai-quantum tensor-engine member / member · tensor engine absent (linux-arm64) (pull_request) Successful in 23s
Details
WAI conformance / each feature of the independent verifiers' steps compiles alone (pull_request) Successful in 11m54s
Details
wai-sink-android native sink library / the JNI layer and the platform-free half on the host JVM · linux-arm64 (pull_request) Successful in 2m47s
Details
WAI energy receipts / energy receipts · linux-arm64 (pull_request) Successful in 10m24s
Details
wai-quantum WebGPU member / member · discrete GPU, Vulkan (jetson-hub) (pull_request) Successful in 25m29s
Details
wai-audit collector and player integration / collector, player integration and the browser end to end · macos (pull_request) Successful in 1m40s
Details
WAI conformance / cross-arch byte-equality · macos (pull_request) Successful in 27s
Details
WAI conformance / codec wrappers · fuzz smoke, text decode limits, C entry points, provenance, the semantic path (pull_request) Successful in 1m25s
Details
WAI conformance / cargo-class corpora · linux-arm64 (pull_request) Successful in 21m23s
Details
WAI energy receipts / energy receipts · macos (pull_request) Successful in 8m6s
Details
standards workspaces build + test / macOS energy backends (pull_request) Successful in 12m38s
Details
wai-quantum command line / command line · macos (pull_request) Successful in 46s
Details
wai-quantum dataflow member / member · dataflow device absent (macos) (pull_request) Successful in 15s
Details
wai-quantum FPGA member / member · FPGA datapath, simulated (macos) (pull_request) Successful in 1m18s
Details
wai-quantum Metal member / member · Metal (macos) (pull_request) Successful in 36s
Details
WAI conformance / cargo-class corpora · macos (pull_request) Successful in 21m48s
Details
wai-quantum OpenCL member / member · OpenCL GPU (macos) (pull_request) Successful in 23s
Details
WAI examples / WAI examples · the archive, the readiness check, the browser and Apple · macos (pull_request) Successful in 12m58s
Details
wai-quantum service / service · macos (pull_request) Successful in 47s
Details
wai-quantum matrix-extension member / member · matrix extension absent (macos) (pull_request) Successful in 11s
Details
wai-quantum portable tensor program / tensor program · Metal and the arm64 CPU (macos) (pull_request) Successful in 17s
Details
wai-quantum CPU vector units / member · vector unit absent (macos) (pull_request) Successful in 15s
Details
wai-quantum neural-processor member / member · neural processor (macos) (pull_request) Successful in 11m6s
Details
wai-quantum studio and desktop / desktop app · macos (pull_request) Successful in 2m47s
Details
wai-quantum WebGPU member / member · Metal (macos) (pull_request) Successful in 2m49s
Details
wai-quantum WebGPU member / member · browser WebGPU (macos) (pull_request) Successful in 1m50s
Details
WAI conformance / independent verifiers · claims, JWP, key discovery and integer payloads (Python) (pull_request) Failing after 52m29s
Details
wai-sink-android native sink library / every ABI, the AAR, and the instrumented tests in a headless emulator · macos (pull_request) Successful in 9m12s
Details
wai-sink-swift native sink package / libwai slices, desktop tests, simulator tests · macos (pull_request) Successful in 22m13s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Transaction-Science/open-standards!509
No description provided.