wai-quantum: a study of the co-design, every measure the engine's #512
21 changed files with 1482 additions and 28 deletions
5
.github/workflows/wai-web-wasm.yml
vendored
5
.github/workflows/wai-web-wasm.yml
vendored
|
|
@ -183,14 +183,15 @@ jobs:
|
|||
# workbench run and sealed in the page, a program's history recorded and
|
||||
# blamed there, the reader's rules checking a proposal, what is new in a
|
||||
# program and in a workbench entry, the reader's taste learned there, the
|
||||
# rewrites offered and explained there, and convergence measured there.
|
||||
# rewrites offered and explained there, convergence measured there, and a
|
||||
# study run there.
|
||||
- name: The job engine in a page (native unit tests)
|
||||
shell: bash
|
||||
working-directory: wai/wai-web
|
||||
run: |
|
||||
set -euo pipefail
|
||||
cargo test --locked --lib --features quantum_engine quantum_engine:: 2>&1 | tee "$RUNNER_TEMP/wai-web-engine.log"
|
||||
grep -q "test result: ok. 13 passed; 0 failed" "$RUNNER_TEMP/wai-web-engine.log"
|
||||
grep -q "test result: ok. 14 passed; 0 failed" "$RUNNER_TEMP/wai-web-engine.log"
|
||||
# The crate's native unit tests, run on the host: the pure-Rust halves of the
|
||||
# bindings, which wasm_tests does not run. They decode the committed integer
|
||||
# payloads to their sidecars' SHA-256 (int_payload::tests) and check that the
|
||||
|
|
|
|||
|
|
@ -2521,6 +2521,38 @@ wire format may change and such changes are recorded here.
|
|||
- **`tools/ops-gpu-bench`** times each capability's decode with its graphs on the CPU and on each adapter, and reads the energy counters the host has.
|
||||
- **CI:** `wai-ops-gpu.yml` runs the Metal and browser legs on the `macos` runner, and requires on a runner with no GPU that the device tests skip without `WAI_OPS_GPU_REQUIRE` and fail with it.
|
||||
|
||||
### wai-quantum: a study of the co-design, every measure the engine's
|
||||
|
||||
The measurement harness the co-design report puts under every feature: each feature behind a flag, compared against a control, in a study whose protocol is sealed before any data.
|
||||
|
||||
- **The protocol** (`study::Protocol`, `seal_protocol`):
|
||||
- the study, its hypotheses, the primary measure, the conditions (each a setting of the Studio's features: `study::FLAGS`), the control, and the analysis (resamples, seed, level);
|
||||
- checked, then sealed with the researcher's key before any data, its digest naming the study.
|
||||
- **Assignment** (`study::assign`): the digest and the participant's key hashed together, so the condition is fixed before they begin and anyone can check it.
|
||||
- **A record** (`seal_record`): what one participant made and did, sealed with their key: the program's history, their predictions, their choices among candidates (with the engine's best of them), their probe tasks done without the assistant, and their answers on ownership.
|
||||
- **The measures** (`study::measure`), the report's eight, each computed again by the engine from the record:
|
||||
- quality: the person's edits the engine proves equal to what they replaced, and the share that made it cheaper, with its trend;
|
||||
- diversity (Vendi) and novelty of their designs;
|
||||
- selection skill against chance;
|
||||
- calibration: the mean distance of a prediction from the run, and the slope;
|
||||
- unassisted skill: four probe tasks scored by a workbench specification each, a score of 1 matching the reference;
|
||||
- ownership: five items from 1 to 7, beside the authorship share by the history;
|
||||
- convergence on a model's proposals.
|
||||
- **The cohort report** (`study::cohort`):
|
||||
- every record checked: the protocol's digest, its seal, the key that sealed it, the condition its key is assigned, and one record per participant; a record that fails is refused with the reason;
|
||||
- per condition, the mean of each measure;
|
||||
- each condition's difference from the control on the primary measure, with a seeded bootstrap interval;
|
||||
- Cronbach's alpha of the ownership items, and the Vendi score of each condition's final designs.
|
||||
- **The command line:** `wai-quantum study catalog | new | assign | measure | cohort` (`--out` seals the report).
|
||||
- **For agents:** `study_measure` and `study_cohort`. The probes are not offered to agents: a probe is done without the assistant.
|
||||
- **The Studio:**
|
||||
- the Study pane: Features, each one switchable off, and held by a study's condition;
|
||||
- "Load a protocol", which shows what the study is about before "Take part";
|
||||
- a record kept only once asked: the choices in Alternatives and the Workbench, "How much is this yours?", and probe tasks with the assistant paused;
|
||||
- "My record", and "Export my record", sealed with the browser's key.
|
||||
- **For hosts:** `study::call` and `call_sealing`; `QuantumEngine::study` in the page, which seals with the page's key, and `study` in the desktop app.
|
||||
- **CI:** wai-web's native engine tests count 14.
|
||||
|
||||
### wai-quantum 0.6.16 and wai-quantum-cli 0.1.4: the person's rules, what is new, taste, rewrites and convergence
|
||||
|
||||
This release carries the five entries below it to the package registry, and 0.6.15's with them: 0.6.15 and 0.1.3 went out as binaries and to the package taps only. `wai-quantum-cli` 0.1.4 requires 0.6.16.
|
||||
|
|
|
|||
|
|
@ -61,6 +61,20 @@ Under Versions, "Your own way" says whether your versions are converging on the
|
|||
|
||||
"End" lets the model propose again. The mode is kept in this browser.
|
||||
|
||||
## Study
|
||||
|
||||
The Study pane lets you take part in a study of how people design with the engine, or keep a record for yourself:
|
||||
|
||||
- **Features:** each of the Studio's features can be turned off. While you take part in a study, its condition sets them and holds them, so that each can be compared against a control.
|
||||
- **Load a protocol:** a researcher gives you one, sealed before any data. You see what it studies before you choose. "Take part" assigns your condition by this browser's key, so it is fixed before you begin.
|
||||
- **Kept, in this browser only, once you ask:**
|
||||
- which of Improve's candidates you applied, with the engine's best of them;
|
||||
- which kept generator you opened in the Workbench;
|
||||
- your answers on ownership ("How much is this yours?", five items from 1 to 7);
|
||||
- the probe tasks you do with the assistant paused, scored by the engine.
|
||||
- **My record:** the engine's measures of what you made and did, each computed again from it.
|
||||
- **Export my record:** your record sealed with this browser's key, saved as a file to give the researcher. Nothing is sent. "Leave the study" gives you the features back.
|
||||
|
||||
## Your taste
|
||||
|
||||
"Your taste", in the Workbench, learns what you prefer from your own choices:
|
||||
|
|
|
|||
|
|
@ -89,6 +89,8 @@ export class TabEngine {
|
|||
rewrite(name, args) { return this.call('rewrite', { name, args }); }
|
||||
// The convergence monitor: `name` is history or cohort.
|
||||
converge(name, args) { return this.call('converge', { name, args }); }
|
||||
// A study: `name` is a call of engine::study; seal_record and seal_protocol seal with this browser's key.
|
||||
study(name, args) { return this.call('study', { name, args }); }
|
||||
async signer() { return (await this.call('signer')).did; }
|
||||
cancel() {
|
||||
this.worker?.terminate();
|
||||
|
|
@ -171,6 +173,7 @@ export class DesktopEngine {
|
|||
taste(name, args) { return this.call('taste', { name, args: JSON.stringify(args) }); }
|
||||
rewrite(name, args) { return this.call('rewrite', { name, args: JSON.stringify(args) }); }
|
||||
converge(name, args) { return this.call('converge', { name, args: JSON.stringify(args) }); }
|
||||
study(name, args) { return this.call('study', { name, args: JSON.stringify(args) }); }
|
||||
async signer() { return (await this.call('signer', {})).did; }
|
||||
saveJob(name, job) { return this.call('save_job', { name, job: JSON.stringify(job, null, 2) }); }
|
||||
saveProgram(name, text) { return this.call('save_program', { name, text }); }
|
||||
|
|
|
|||
67
wai/quantum-studio/src/flags.js
Normal file
67
wai/quantum-studio/src/flags.js
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
// The Studio's features, each on or off (engine::study::FLAGS). All are on
|
||||
// by default and the reader may turn any off. While the reader takes part in
|
||||
// a study, the features its condition names are set and held there until they
|
||||
// leave it, so each can be compared against a control; a feature it does not
|
||||
// name stays the reader's.
|
||||
import { el } from './dom.js';
|
||||
|
||||
const SAVED = 'wai.quantum-studio.flags';
|
||||
export const FLAGS = [
|
||||
['critique', 'the engine’s critique on the live diagram'],
|
||||
['assistant_asks', 'the assistant asks before it proposes, and speaks only when asked'],
|
||||
['alternatives', 'Alternatives: improve and explore, on the engine’s axes'],
|
||||
['predict_first', 'predict before a run'],
|
||||
['workbench', 'the Workbench'],
|
||||
['rules', 'rules for proposals'],
|
||||
['shortcuts', 'the classical-shortcut lens'],
|
||||
['novelty', 'what is new: labels and Found'],
|
||||
['taste', 'your taste'],
|
||||
['rewrites', 'the rewrite lens'],
|
||||
['convergence', 'your own way'],
|
||||
];
|
||||
|
||||
// `root` is the Studio's element: each feature turned off sets `qs-off-NAME` on it.
|
||||
export function featureFlags(root) {
|
||||
const state = { set: {}, locked: null };
|
||||
try { Object.assign(state, JSON.parse(localStorage.getItem(SAVED) || '{}')); } catch (e) {}
|
||||
const keep = () => { try { localStorage.setItem(SAVED, JSON.stringify(state)); } catch (e) {} };
|
||||
const heard = new Set();
|
||||
// A feature is on unless the condition holding it, or the reader, turned it off.
|
||||
const on = (f) => (state.locked && Object.hasOwn(state.locked, f) ? state.locked[f] : state.set[f]) !== false;
|
||||
function apply() {
|
||||
for (const [f] of FLAGS) root.classList.toggle(`qs-off-${f}`, !on(f));
|
||||
for (const h of heard) h();
|
||||
}
|
||||
return {
|
||||
on,
|
||||
locked: () => !!state.locked,
|
||||
// Whether the study's condition names this feature, and so holds it.
|
||||
held: (f) => !!state.locked && Object.hasOwn(state.locked, f),
|
||||
set(f, v) {
|
||||
if (state.locked && Object.hasOwn(state.locked, f)) return;
|
||||
state.set[f] = !!v;
|
||||
keep(); apply();
|
||||
},
|
||||
lock(flags) { state.locked = { ...flags }; keep(); apply(); },
|
||||
unlock() { state.locked = null; keep(); apply(); },
|
||||
onChange: (h) => heard.add(h),
|
||||
apply,
|
||||
};
|
||||
}
|
||||
|
||||
// The list of features, to turn each on or off; held while a study sets them.
|
||||
export function flagsView(flags) {
|
||||
const list = el('ul', { class: 'qs-flags', id: 'qs-flags' });
|
||||
const note = el('p', { class: 'qs-note', id: 'qs-flags-note' });
|
||||
function show() {
|
||||
note.textContent = flags.locked() ? 'The study you take part in sets the ones it names, so that each can be compared against a control; leave the study to change those.' : 'Turn any feature off; the Studio keeps your choice in this browser.';
|
||||
list.replaceChildren(...FLAGS.map(([f, what]) => {
|
||||
const box = el('input', { type: 'checkbox', id: `qs-flag-${f}`, checked: flags.on(f), disabled: flags.held(f) });
|
||||
box.addEventListener('change', () => flags.set(f, box.checked));
|
||||
return el('li', {}, el('label', { class: 'qs-check', for: `qs-flag-${f}` }, box, el('span', { text: what })));
|
||||
}));
|
||||
}
|
||||
flags.onChange(show);
|
||||
show();
|
||||
return el('details', { class: 'qs-study-flags', id: 'qs-features' }, el('summary', { text: 'Features' }), note, list);
|
||||
}
|
||||
|
|
@ -13,6 +13,7 @@ import { el, sameness } from './dom.js';
|
|||
import { chips } from './critique-view.js';
|
||||
import { noveltyLabel } from './novelty.js';
|
||||
import { declared } from './generators.js';
|
||||
import { engineBest } from './study.js';
|
||||
|
||||
const NS = 'http://www.w3.org/2000/svg';
|
||||
const svg = (tag, attrs = {}, ...kids) => {
|
||||
|
|
@ -83,7 +84,9 @@ export function atlas(draft, cands, onPick) {
|
|||
// the same circuit as the draft, taken whole or refused whole, is a choice
|
||||
// between the two, and Improve's candidates are ordered by the profile.
|
||||
// `paused()` is true while the reader explores on their own: the model is not asked.
|
||||
export function galleryPane({ engine, draft, model, apply, check = null, found = null, taste = null, paused = () => false }) {
|
||||
// `features.on(name)` says whether a feature is on (flags.js); `onChoice` hears
|
||||
// which of Improve's candidates the reader applied, and the engine's best of them.
|
||||
export function galleryPane({ engine, draft, model, apply, check = null, found = null, taste = null, paused = () => false, features = { on: () => true }, onChoice = null }) {
|
||||
const intent = el('input', { class: 'qs-input', id: 'qs-explore-aim', placeholder: 'Toward… fewer qubits, another entangler, more symmetry', autocomplete: 'off', 'aria-label': 'What Explore should aim for' });
|
||||
const improveBtn = el('button', { class: 'qs-btn qs-small', id: 'qs-improve', type: 'button', text: 'Improve', title: 'The same circuit, cheaper: each checked equivalent to the draft' });
|
||||
const exploreBtn = el('button', { class: 'qs-btn qs-small', id: 'qs-explore', type: 'button', text: 'Explore', title: 'Different circuits toward the aim, from the model, each measured by the engine' });
|
||||
|
|
@ -133,15 +136,20 @@ export function galleryPane({ engine, draft, model, apply, check = null, found =
|
|||
// draft, when the engine proved them the same circuit. One taken in part,
|
||||
// or with changes the rules blocked, says nothing either way.
|
||||
async function learn(r, c, src) {
|
||||
if (!taste || !r?.offered || r.blocked || declared(src).length || c.equivalence?.verdict !== 'equivalent') return;
|
||||
if (!taste || !features.on('taste') || !r?.offered || r.blocked || declared(src).length || c.equivalence?.verdict !== 'equivalent') return;
|
||||
const took = r.accepted === r.offered ? true : r.accepted === 0 ? false : null;
|
||||
if (took == null) return;
|
||||
await taste.choose(took ? c.qasm : src, took ? src : c.qasm, { task: 'equivalent' }).catch(() => {});
|
||||
}
|
||||
|
||||
// Which of Improve's candidates the reader applied, against the engine's best.
|
||||
function chose(mode, cands, i) {
|
||||
if (mode === 'improve' && cands.length >= 2) onChoice?.({ kind: 'alternatives', k: cands.length, picked: i, best: engineBest(cands.map((x) => x.measures)) });
|
||||
}
|
||||
|
||||
// Improve's candidates in the order of the reader's taste, once it has a choice.
|
||||
async function byTaste(src, cands) {
|
||||
if (!taste || cands.length < 2 || declared(src).length || !taste.profile().choices.length) return { cands, said: '' };
|
||||
if (!taste || !features.on('taste') || cands.length < 2 || declared(src).length || !taste.profile().choices.length) return { cands, said: '' };
|
||||
const r = await taste.rank(cands.map((c, i) => ({ name: String(i), qasm: c.qasm })), {}, 0).catch(() => null);
|
||||
if (!r) return { cands, said: '' };
|
||||
return { cands: r.ranked.map((x) => cands[Number(x.name)]), said: `Ordered by your taste (${taste.name()}, ${r.choices} choice${r.choices === 1 ? '' : 's'}).` };
|
||||
|
|
@ -151,7 +159,7 @@ export function galleryPane({ engine, draft, model, apply, check = null, found =
|
|||
const cards = cands.map((c, i) => {
|
||||
const s = sameness(c.equivalence);
|
||||
if (mode === 'explore' && s.cls === 'bad') s.cls = ''; // a different circuit is what Explore asks for
|
||||
const nv = noveltyLabel(c.novelty);
|
||||
const nv = features.on('novelty') ? noveltyLabel(c.novelty) : null;
|
||||
const by = c.source === 'engine' ? { kind: 'engine', name: 'Alternatives' } : { kind: 'assistant', name: model()?.model?.name || undefined };
|
||||
if (nv?.flag === 'route') found?.flag({ kind: 'route', title: `${c.idea || c.label}: ${nv.text}`, why: nv.why, where: mode === 'improve' ? 'Improve' : 'Explore', source: c.qasm, by });
|
||||
const card = el('li', { class: `qs-cand${nv?.flag ? ' find' : ''}`, 'data-i': i, id: `qs-cand-${i}`, tabindex: '-1' },
|
||||
|
|
@ -165,10 +173,10 @@ export function galleryPane({ engine, draft, model, apply, check = null, found =
|
|||
el('span', { class: 'qs-note', text: c.distance.changed ? `${c.distance.changed} lines changed` : 'the same lines' }),
|
||||
nv ? el('span', { class: `qs-badge qs-novelty ${nv.cls}`, title: nv.why, text: nv.text }) : null),
|
||||
c.changes?.length ? el('p', { class: 'qs-note', text: c.changes.join('; ') }) : null,
|
||||
c.chain ? chainView(c.chain) : null,
|
||||
c.chain && features.on('rewrites') ? chainView(c.chain) : null,
|
||||
el('details', { class: 'qs-cand-src' }, el('summary', { text: 'Program' }), el('pre', { class: 'qs-ai-code', text: c.qasm })),
|
||||
el('div', { class: 'qs-actions' },
|
||||
el('button', { class: 'qs-btn qs-small qs-apply', type: 'button', text: 'Apply', onclick: async () => learn(await apply(c.qasm, `${mode === 'improve' ? 'An improvement' : 'An exploration'} from ${c.source === 'engine' ? 'the engine' : 'the model'}: ${c.idea || c.label}. ${s.text[0].toUpperCase() + s.text.slice(1)}.`, by), c, src) })));
|
||||
el('button', { class: 'qs-btn qs-small qs-apply', type: 'button', text: 'Apply', onclick: async () => { chose(mode, cands, i); learn(await apply(c.qasm, `${mode === 'improve' ? 'An improvement' : 'An exploration'} from ${c.source === 'engine' ? 'the engine' : 'the model'}: ${c.idea || c.label}. ${s.text[0].toUpperCase() + s.text.slice(1)}.`, by), c, src); } })));
|
||||
return card;
|
||||
});
|
||||
const pick = (i, go) => {
|
||||
|
|
|
|||
|
|
@ -35,6 +35,8 @@ import { workbenchPane } from './workbench.js';
|
|||
import { foundPane } from './novelty.js';
|
||||
import { tasteStore } from './taste.js';
|
||||
import { organicMode, convergePanel, ORGANIC } from './converge.js';
|
||||
import { featureFlags, flagsView } from './flags.js';
|
||||
import { studyLog, studyPane } from './study.js';
|
||||
import { versionsPane, HANDS, whoName } from './versions.js';
|
||||
import { rulesPane, heldLines, LOCK, UNLOCK } from './rules.js';
|
||||
import { declared, inputsBar, withInputs } from './generators.js';
|
||||
|
|
@ -131,7 +133,12 @@ function mount(root) {
|
|||
},
|
||||
changed: () => paintHeld(),
|
||||
});
|
||||
const brief = async () => (await local.locks('brief', { base: text(), locks: rules.locks() })).rules;
|
||||
// The Studio's features (flags.js), held by a study the reader takes part in.
|
||||
const flags = featureFlags(root);
|
||||
const studyRec = studyLog();
|
||||
if (studyRec.enrolled()) flags.lock(studyRec.state().flags || {});
|
||||
const ruleSet = () => (flags.on('rules') ? rules.locks() : {});
|
||||
const brief = async () => (await local.locks('brief', { base: text(), locks: ruleSet() })).rules;
|
||||
|
||||
// ---- the program ----
|
||||
const lintBadge = el('span', { class: 'qs-badge', id: 'qs-lint', text: '…' });
|
||||
|
|
@ -211,10 +218,12 @@ function mount(root) {
|
|||
draft: () => text(),
|
||||
model: () => (aiConnected ? assistantFor() : null),
|
||||
apply: (qasm, note, by) => offer(qasm, by, note),
|
||||
check: (base, qasm) => local.locks('check', { base, proposed: qasm, locks: rules.locks(), inputs: genInputs() }),
|
||||
check: (base, qasm) => local.locks('check', { base, proposed: qasm, locks: ruleSet(), inputs: genInputs() }),
|
||||
found,
|
||||
taste,
|
||||
paused: () => organic.on(),
|
||||
features: flags,
|
||||
onChoice: (c) => studyRec.choice(c),
|
||||
});
|
||||
|
||||
// ---- the workbench ----
|
||||
|
|
@ -227,6 +236,8 @@ function mount(root) {
|
|||
found,
|
||||
taste,
|
||||
paused: () => organic.on(),
|
||||
features: flags,
|
||||
onChoice: (c) => studyRec.choice(c),
|
||||
});
|
||||
|
||||
// ---- the result ----
|
||||
|
|
@ -302,7 +313,7 @@ function mount(root) {
|
|||
text: () => text(),
|
||||
setText: (s) => setText(s),
|
||||
inputs: genInputs,
|
||||
rules: () => rules.locks(),
|
||||
rules: () => ruleSet(),
|
||||
taken: () => { if (rules.region()) rules.setRegion(null); },
|
||||
lock: (on) => { view.dispatch({ effects: editable.reconfigure(EditorState.readOnly.of(on)) }); editorHost.classList.toggle('qs-locked', on); },
|
||||
reviewHost: reviewBox,
|
||||
|
|
@ -316,7 +327,20 @@ function mount(root) {
|
|||
converge = convergePanel({ engine: native, history: () => versions.history(), inputs: genInputs, organic });
|
||||
versions.node.append(converge.node);
|
||||
|
||||
root.replaceChildren(header, el('div', { class: 'qs-grid' }, program, el('div', { class: 'qs-side' }, circuitPane, gallery.node, workbench.node, found.node, resultPane, versions.node, aiPane, historyPane)), footer);
|
||||
// ---- a study the reader takes part in, and their own record ----
|
||||
const studyUi = studyPane({ engine: native, log: studyRec, flags, history: () => versions.history(), inputs: genInputs, signer: () => local.signer(), save: (name, body, type) => saveFile(name, body, type), organic, featuresView: flagsView(flags) });
|
||||
// The features a study sets reach the controls they govern.
|
||||
const settle = () => {
|
||||
let own = false;
|
||||
try { own = localStorage.getItem(PREDICT_FIRST) === '1'; } catch (e) {}
|
||||
predictFirst.checked = flags.held('predict_first') ? flags.on('predict_first') : flags.on('predict_first') && own;
|
||||
predictFirst.disabled = flags.held('predict_first');
|
||||
aiStance.disabled = flags.held('assistant_asks') || !flags.on('assistant_asks');
|
||||
};
|
||||
flags.onChange(settle);
|
||||
flags.apply();
|
||||
|
||||
root.replaceChildren(header, el('div', { class: 'qs-grid' }, program, el('div', { class: 'qs-side' }, circuitPane, gallery.node, workbench.node, found.node, resultPane, versions.node, aiPane, historyPane, studyUi.node)), footer);
|
||||
|
||||
// ---- the editor ----
|
||||
const style = HighlightStyle.define([
|
||||
|
|
@ -423,7 +447,8 @@ function mount(root) {
|
|||
paintHeld();
|
||||
// The smells, on the diagram: each gate in one is flagged, its tooltip saying why.
|
||||
const smells = c.ok ? c.smells : [];
|
||||
mark(diagram, 'qc-flag', smells.flatMap((s) => s.ops), smells.flatMap((s) => s.qubits && !s.ops.length ? s.qubits : []));
|
||||
if (flags.on('critique')) mark(diagram, 'qc-flag', smells.flatMap((s) => s.ops), smells.flatMap((s) => s.qubits && !s.ops.length ? s.qubits : []));
|
||||
else mark(diagram, 'qc-flag', []);
|
||||
for (const s of smells) for (const op of s.ops) annotate(diagram, op, s.message);
|
||||
} catch (e) {
|
||||
critique.update({ ok: false, error: e.message });
|
||||
|
|
@ -736,7 +761,7 @@ function mount(root) {
|
|||
const assistantFor = () => {
|
||||
const m = model();
|
||||
if (!assistant || assistant.model.endpoint !== m.endpoint || assistant.model.name !== m.name) {
|
||||
assistant = new Assistant({ engine: local, model: m, context: () => ({ qasm: text(), job: request(), last: current?.job }), stance: () => (organic.on() ? 'questions' : aiStance.value), rules: brief });
|
||||
assistant = new Assistant({ engine: local, model: m, context: () => ({ qasm: text(), job: request(), last: current?.job }), stance: () => (organic.on() ? 'questions' : !flags.on('assistant_asks') ? 'suggest' : flags.held('assistant_asks') ? 'questions' : aiStance.value), rules: brief });
|
||||
}
|
||||
return assistant;
|
||||
};
|
||||
|
|
|
|||
|
|
@ -377,3 +377,32 @@
|
|||
.qs-converge .qs-note { margin: 0; }
|
||||
.qs-converge.converging { background: color-mix(in srgb, var(--accent) 7%, transparent); }
|
||||
.qs-converge .qs-badge[hidden] { display: none; }
|
||||
|
||||
/* Features turned off, by the reader or by the condition of a study they take part in. */
|
||||
.qs-off-critique #qs-critique,
|
||||
.qs-off-alternatives .qs-alt,
|
||||
.qs-off-workbench .qs-wb,
|
||||
.qs-off-rules #qs-rules,
|
||||
.qs-off-shortcuts #qs-shortcuts-panel,
|
||||
.qs-off-novelty .qs-found, .qs-off-novelty .qs-novelty, .qs-off-novelty #qs-wb-novelty,
|
||||
.qs-off-taste #qs-taste,
|
||||
.qs-off-rewrites #qs-rewrites-panel, .qs-off-rewrites .qs-chain,
|
||||
.qs-off-convergence #qs-converge-box,
|
||||
.qs-off-predict_first label[for="qs-predict-first"] { display: none !important; }
|
||||
|
||||
/* The study, and your own record. */
|
||||
.qs-study-body { display: flex; flex-direction: column; gap: 8px; padding: 8px 12px 12px; }
|
||||
.qs-study-body .qs-note, .qs-study-body .qs-check { margin: 0; }
|
||||
.qs-study-body [hidden] { display: none !important; }
|
||||
.qs-study-offer:empty, .qs-study-out:empty { display: none; }
|
||||
.qs-study-offer { display: flex; flex-direction: column; gap: 6px; padding: 8px 10px; border: 1px solid color-mix(in srgb, var(--accent) 50%, var(--border)); border-radius: 8px; }
|
||||
.qs-study-name { font-size: 0.95rem; font-weight: 650; }
|
||||
.qs-study-hyp { margin: 0; padding-left: 20px; font-size: 12px; color: var(--text-secondary); }
|
||||
.qs-study-out { display: flex; flex-direction: column; gap: 6px; }
|
||||
.qs-own { margin: 0; padding-left: 20px; display: flex; flex-direction: column; gap: 4px; font-size: 12px; }
|
||||
.qs-own li { display: flex; align-items: center; justify-content: space-between; gap: 8px; }
|
||||
.qs-study-task { margin: 0; font-size: 12px; color: var(--text-primary); }
|
||||
.qs-probe-answer { width: 100%; font: 12px/1.5 var(--font-mono, ui-monospace, monospace); resize: vertical; }
|
||||
.qs-study-measures th { font-weight: 400; color: var(--text-secondary); text-align: left; }
|
||||
.qs-study-flags summary { cursor: pointer; font: 12px var(--font-mono, ui-monospace, monospace); color: var(--text-secondary); }
|
||||
.qs-flags { list-style: none; margin: 6px 0 0; padding: 0; display: flex; flex-direction: column; gap: 3px; font-size: 12px; }
|
||||
|
|
|
|||
220
wai/quantum-studio/src/study.js
Normal file
220
wai/quantum-studio/src/study.js
Normal file
|
|
@ -0,0 +1,220 @@
|
|||
// Taking part in a study, and your own record (engine::study).
|
||||
//
|
||||
// A researcher writes a protocol and seals it before any data. The reader
|
||||
// loads it here, sees what it studies, and chooses to take part. Their
|
||||
// condition follows from this browser's key, so it is fixed before they begin
|
||||
// and anyone can check it, and it sets the Studio's features. While they take
|
||||
// part, or when they choose to keep a record for themselves, the Studio keeps,
|
||||
// in this browser only, which candidate they took among those shown (and the
|
||||
// engine's best of them), the probe tasks they do with the assistant paused,
|
||||
// and their answers on ownership. Every measure is the engine's, computed
|
||||
// again from what they made. "Export my record" seals it with this browser's
|
||||
// key into a file they give the researcher; nothing is sent anywhere.
|
||||
import { el } from './dom.js';
|
||||
import * as Predict from './predict.js';
|
||||
|
||||
const SAVED = 'wai.quantum-studio.study';
|
||||
const KEEP = 500;
|
||||
|
||||
export function studyLog() {
|
||||
const state = { keeping: false, sealed: null, digest: null, study: null, condition: null, flags: null, joined: null, choices: [], probes: [], ownership: [] };
|
||||
try { Object.assign(state, JSON.parse(localStorage.getItem(SAVED) || '{}')); } catch (e) {}
|
||||
const keep = () => { try { localStorage.setItem(SAVED, JSON.stringify(state)); } catch (e) {} };
|
||||
const push = (k, x) => {
|
||||
if (!state.keeping) return;
|
||||
state[k].push({ ...x, at: Date.now() });
|
||||
state[k] = state[k].slice(-KEEP);
|
||||
keep();
|
||||
};
|
||||
return {
|
||||
state: () => state,
|
||||
keeping: () => state.keeping,
|
||||
enrolled: () => !!state.digest,
|
||||
setKeeping(v) { state.keeping = !!v; keep(); },
|
||||
choice: (c) => push('choices', c),
|
||||
probe: (p) => push('probes', p),
|
||||
ownership: (o) => push('ownership', o),
|
||||
join(sealed, digest, study, a) { Object.assign(state, { sealed, digest, study, condition: a.condition, flags: a.flags || {}, joined: Date.now(), keeping: true }); keep(); },
|
||||
leave() { Object.assign(state, { sealed: null, digest: null, study: null, condition: null, flags: null, joined: null }); keep(); },
|
||||
forget() { Object.assign(state, { choices: [], probes: [], ownership: [] }); keep(); },
|
||||
};
|
||||
}
|
||||
|
||||
// The engine's best of candidates shown: the fewest T gates, then two-qubit
|
||||
// gates, then depth, then gates.
|
||||
export function engineBest(measures) {
|
||||
const key = (m) => [m?.t_count ?? 0, m?.two_qubit ?? 0, m?.depth ?? 0, m?.gates ?? 0];
|
||||
let best = 0;
|
||||
measures.forEach((m, i) => {
|
||||
const a = key(m), b = key(measures[best]);
|
||||
for (let j = 0; j < a.length; j++) { if (a[j] !== b[j]) { if (a[j] < b[j]) best = i; break; } }
|
||||
});
|
||||
return best;
|
||||
}
|
||||
|
||||
const NAMES = {
|
||||
quality: 'your edits the engine proves equal that made it cheaper',
|
||||
quality_trend: 'that share, later against earlier',
|
||||
selection: 'how often you took the engine’s best of those shown',
|
||||
calibration_off: 'how far your predicted share was from the run’s',
|
||||
calibration_slope: 'the run’s share against your prediction (1 is calibrated)',
|
||||
unassisted: 'your probe score without the assistant (1 matches the reference)',
|
||||
ownership: 'your answers on ownership, 1 to 7',
|
||||
authorship: 'your share of the program’s gates',
|
||||
novelty: 'how far your designs are from the known constructions',
|
||||
diversity: 'how many distinct designs you made (Vendi)',
|
||||
convergence: 'a model’s share of your program’s gates',
|
||||
};
|
||||
|
||||
// The pane. `history()` and `inputs()` read the program's; `signer()` is
|
||||
// this browser's did; `save(name, text, type)` writes a file; `flags` the
|
||||
// features; `organic` pauses the assistant during a probe; `featuresView`
|
||||
// the list of features.
|
||||
export function studyPane({ engine, log, flags, history, inputs, signer, save, organic, featuresView }) {
|
||||
const status = el('p', { class: 'qs-note', id: 'qs-study-status' });
|
||||
const fileIn = el('input', { type: 'file', accept: '.json,application/json', id: 'qs-study-file', hidden: true, 'aria-hidden': 'true' });
|
||||
const loadBtn = el('button', { class: 'qs-btn qs-small', id: 'qs-study-load', type: 'button', text: 'Load a protocol', onclick: () => fileIn.click() });
|
||||
const offer = el('div', { class: 'qs-study-offer', id: 'qs-study-offer' });
|
||||
const keepBox = el('input', { type: 'checkbox', id: 'qs-study-keep', checked: log.keeping() });
|
||||
const keepLabel = el('label', { class: 'qs-check', for: 'qs-study-keep' }, keepBox, el('span', { text: 'Keep a record of how I work, in this browser' }));
|
||||
const tools = el('div', { class: 'qs-study-tools', id: 'qs-study-tools' });
|
||||
const out = el('div', { class: 'qs-study-out', id: 'qs-study-out' });
|
||||
const node = el('section', { class: 'qs-pane qs-study', 'aria-labelledby': 'qs-study-h' },
|
||||
el('div', { class: 'qs-head' }, el('h2', { id: 'qs-study-h', text: 'Study' }), el('span', { class: 'qs-grow' }), loadBtn, fileIn),
|
||||
el('div', { class: 'qs-study-body' }, status, offer, keepLabel, tools, out, featuresView));
|
||||
|
||||
const record = () => {
|
||||
const s = log.state();
|
||||
return { protocol: s.digest, condition: s.condition, flags: s.flags, history: history(), inputs: inputs(), predictions: Predict.load(), choices: s.choices, probes: s.probes, ownership: s.ownership };
|
||||
};
|
||||
|
||||
function show() {
|
||||
const s = log.state();
|
||||
keepBox.checked = log.keeping();
|
||||
keepLabel.hidden = log.enrolled();
|
||||
if (log.enrolled()) {
|
||||
status.textContent = `You take part in “${s.study}”, condition ${s.condition}, set by your key. Your record stays in this browser until you export it.`;
|
||||
} else {
|
||||
status.textContent = 'Take part in a study of how people design with the engine: a researcher gives you a protocol, sealed before any data, and you see what it studies before you choose. Or keep a record for yourself.';
|
||||
}
|
||||
const n = s.choices.length + s.probes.length + s.ownership.length;
|
||||
tools.hidden = !log.keeping();
|
||||
tools.replaceChildren(
|
||||
el('p', { class: 'qs-note', text: `Kept here: ${s.choices.length} choice${s.choices.length === 1 ? '' : 's'} among candidates, ${s.probes.length} probe${s.probes.length === 1 ? '' : 's'}, ${s.ownership.length} answer${s.ownership.length === 1 ? '' : 's'} on ownership, and your predictions and history.` }),
|
||||
el('div', { class: 'qs-actions' },
|
||||
el('button', { class: 'qs-btn qs-small', id: 'qs-study-own', type: 'button', text: 'How much is this yours?', onclick: ownershipForm }),
|
||||
el('button', { class: 'qs-btn qs-small', id: 'qs-study-probe', type: 'button', text: 'A probe task', onclick: probeForm }),
|
||||
el('button', { class: 'qs-btn qs-small', id: 'qs-study-record', type: 'button', text: 'My record', onclick: showRecord }),
|
||||
log.enrolled() ? el('button', { class: 'qs-btn qs-small', id: 'qs-study-export', type: 'button', text: 'Export my record', onclick: exportRecord }) : null,
|
||||
log.enrolled() ? el('button', { class: 'qs-btn qs-small', id: 'qs-study-leave', type: 'button', text: 'Leave the study', onclick: leave }) : null,
|
||||
n ? el('button', { class: 'qs-btn qs-small', id: 'qs-study-forget', type: 'button', text: 'Forget what is kept', onclick: () => { log.forget(); out.replaceChildren(); show(); } }) : null));
|
||||
}
|
||||
|
||||
fileIn.addEventListener('change', async () => {
|
||||
const f = fileIn.files?.[0];
|
||||
fileIn.value = '';
|
||||
if (!f) return;
|
||||
try {
|
||||
const sealed = JSON.parse(await f.text());
|
||||
const p = await engine.study('open_protocol', { sealed });
|
||||
const me = await signer();
|
||||
const a = await engine.study('assign', { sealed, participant: me });
|
||||
offer.replaceChildren(
|
||||
el('h3', { class: 'qs-study-name', text: p.protocol.study }),
|
||||
el('p', { class: 'qs-note', text: 'What it studies, written before any data:' }),
|
||||
el('ul', { class: 'qs-study-hyp' }, ...p.protocol.hypotheses.map((h) => el('li', { text: h }))),
|
||||
el('p', { class: 'qs-note', text: `It compares ${p.protocol.conditions.length} conditions, each a setting of the Studio’s features; your key assigns you one. Its primary measure: ${NAMES[p.protocol.primary] ?? p.protocol.primary}. The protocol is sealed (digest ${p.digest.slice(0, 16)}…), so it cannot change after you join.` }),
|
||||
el('div', { class: 'qs-actions' },
|
||||
el('button', { class: 'qs-btn qs-small qs-primary', id: 'qs-study-join', type: 'button', text: 'Take part', onclick: () => { log.join(sealed, p.digest, p.protocol.study, a); flags.lock(a.flags || {}); offer.replaceChildren(); show(); } }),
|
||||
el('button', { class: 'qs-btn qs-small', type: 'button', text: 'Not now', onclick: () => offer.replaceChildren() })));
|
||||
} catch (e) { offer.replaceChildren(el('p', { class: 'qs-empty', text: `That is not a protocol the engine opens: ${e.message}` })); }
|
||||
});
|
||||
|
||||
function leave() {
|
||||
log.leave();
|
||||
flags.unlock();
|
||||
out.replaceChildren(el('p', { class: 'qs-note', text: 'You left the study. What was kept stays in this browser until you forget it.' }));
|
||||
show();
|
||||
}
|
||||
|
||||
async function ownershipForm() {
|
||||
const items = await engine.study('ownership_items', {});
|
||||
const selects = items.map((text, i) => el('select', { class: 'qs-input qs-small', id: `qs-own-${i}`, 'aria-label': text },
|
||||
el('option', { value: '', text: '–' }), ...[1, 2, 3, 4, 5, 6, 7].map((v) => el('option', { value: String(v), text: String(v) }))));
|
||||
const saveBtn = el('button', { class: 'qs-btn qs-small', id: 'qs-own-save', type: 'button', text: 'Save my answers' });
|
||||
saveBtn.addEventListener('click', () => {
|
||||
const vals = selects.map((s) => Number(s.value));
|
||||
if (vals.some((v) => !v)) { saveBtn.textContent = 'Answer every one'; return; }
|
||||
log.ownership({ items: vals, program: history()?.versions?.at(-1)?.id ?? null });
|
||||
out.replaceChildren(el('p', { class: 'qs-note', id: 'qs-own-saved', text: 'Saved, with the version of the program you answered about.' }));
|
||||
show();
|
||||
});
|
||||
out.replaceChildren(el('p', { class: 'qs-note', text: 'About the program in the editor, from 1 (not at all) to 7 (completely):' }),
|
||||
el('ol', { class: 'qs-own' }, ...items.map((text, i) => el('li', {}, el('span', { text }), selects[i]))), saveBtn);
|
||||
}
|
||||
|
||||
async function probeForm() {
|
||||
const probes = await engine.study('probes', {});
|
||||
const done = new Set(log.state().probes.map((p) => p.probe));
|
||||
const pick = el('select', { class: 'qs-input qs-small', id: 'qs-probe-pick', 'aria-label': 'Which probe' }, ...probes.map((p) => el('option', { value: p.probe, text: `${p.probe}${done.has(p.probe) ? ' (done before)' : ''}` })));
|
||||
const task = el('p', { class: 'qs-study-task', id: 'qs-probe-task' });
|
||||
const answer = el('textarea', { class: 'qs-input qs-probe-answer', id: 'qs-probe-answer', rows: '7', spellcheck: 'false', 'aria-label': 'Your answer, in OpenQASM', placeholder: 'OPENQASM 3.0;\nqubit[2] q;\n…' });
|
||||
const result = el('p', { class: 'qs-note', id: 'qs-probe-result' });
|
||||
const was = organic.on();
|
||||
organic.set(true);
|
||||
const close = () => { organic.set(was); out.replaceChildren(); };
|
||||
const showTask = () => { task.textContent = probes.find((p) => p.probe === pick.value)?.task ?? ''; result.textContent = ''; };
|
||||
pick.addEventListener('change', showTask);
|
||||
const score = el('button', { class: 'qs-btn qs-small', id: 'qs-probe-score', type: 'button', text: 'Score it' });
|
||||
score.addEventListener('click', async () => {
|
||||
try {
|
||||
const s = await engine.study('score_probe', { probe: pick.value, qasm: answer.value });
|
||||
log.probe({ probe: pick.value, source: answer.value });
|
||||
result.textContent = s.verified ? `Verified. Score ${s.score} (the reference scores 1; cost ${s.cost} against ${s.reference_cost}).` : `Not verified: ${s.failure}. Score 0.`;
|
||||
show();
|
||||
} catch (e) { result.textContent = e.message; }
|
||||
});
|
||||
out.replaceChildren(el('p', { class: 'qs-note', text: 'Done without the assistant: it is paused until you close this. The engine scores your answer.' }),
|
||||
el('div', { class: 'qs-actions' }, pick), task, answer,
|
||||
el('div', { class: 'qs-actions' }, score, el('button', { class: 'qs-btn qs-small', id: 'qs-probe-close', type: 'button', text: 'Close', onclick: close })), result);
|
||||
showTask();
|
||||
}
|
||||
|
||||
async function showRecord() {
|
||||
try {
|
||||
const m = await engine.study('measure', { record: record() });
|
||||
const rows = [
|
||||
['quality', m.quality?.share, m.quality ? `${m.quality.improved} of ${m.quality.edits} edits` : ''],
|
||||
['quality_trend', m.quality?.trend, ''],
|
||||
['selection', m.selection?.rate, m.selection?.choices ? `${m.selection.best_taken} of ${m.selection.choices}; by chance ${m.selection.chance}` : ''],
|
||||
['calibration_off', m.calibration?.mean_off, m.calibration?.predictions ? `${m.calibration.predictions} predictions` : ''],
|
||||
['calibration_slope', m.calibration?.slope, ''],
|
||||
['unassisted', m.unassisted?.mean, m.unassisted?.probes ? `${m.unassisted.probes} probes` : ''],
|
||||
['ownership', m.ownership?.mean, m.ownership?.answers ? `${m.ownership.answers} answers` : ''],
|
||||
['authorship', m.authorship, ''],
|
||||
['novelty', m.novelty?.mean, ''],
|
||||
['diversity', m.diversity?.vendi, m.diversity?.designs ? `${m.diversity.designs} designs` : ''],
|
||||
['convergence', m.convergence?.share, ''],
|
||||
];
|
||||
out.replaceChildren(el('table', { class: 'qs-table qs-study-measures', id: 'qs-study-measures' },
|
||||
el('tbody', {}, ...rows.map(([k, v, more]) => el('tr', { 'data-measure': k },
|
||||
el('th', { scope: 'row', text: NAMES[k] }),
|
||||
el('td', { class: 'num', text: v == null ? '—' : String(v) }),
|
||||
el('td', { class: 'qs-note', text: more }))))),
|
||||
el('p', { class: 'qs-note', text: 'Each is the engine’s, computed again from what you made and did; none is a model’s judgement.' }));
|
||||
} catch (e) { out.replaceChildren(el('p', { class: 'qs-empty', text: e.message })); }
|
||||
}
|
||||
|
||||
async function exportRecord() {
|
||||
try {
|
||||
const sealed = await engine.study('seal_record', { record: record() });
|
||||
const me = sealed.record.participant;
|
||||
const where = await save(`study-record-${me.slice(-12)}.json`, JSON.stringify(sealed, null, 2) + '\n', 'application/json');
|
||||
out.replaceChildren(el('p', { class: 'qs-note', id: 'qs-study-exported', text: `Saved ${where}, sealed with your key. Give it to the researcher; nothing was sent.` }));
|
||||
} catch (e) { out.replaceChildren(el('p', { class: 'qs-empty', text: e.message })); }
|
||||
}
|
||||
|
||||
keepBox.addEventListener('change', () => { log.setKeeping(keepBox.checked); show(); });
|
||||
show();
|
||||
return { node, show };
|
||||
}
|
||||
|
|
@ -132,7 +132,9 @@ const byOf = (p) => (p === 'mutate' ? { kind: 'engine', name: 'the blind search'
|
|||
// `by`'s proposal. `found` is the Found pane; `taste` the reader's taste
|
||||
// profiles (taste.js), learned from choices between kept generators.
|
||||
// `paused()` is true while the reader explores on their own: the model is not asked.
|
||||
export function workbenchPane({ engine, draft, model, apply, save, found = null, taste = null, paused = () => false }) {
|
||||
// `features.on(name)` says whether a feature is on (flags.js); `onChoice` hears
|
||||
// which kept generator the reader opened, by its rank in cost.
|
||||
export function workbenchPane({ engine, draft, model, apply, save, found = null, taste = null, paused = () => false, features = { on: () => true }, onChoice = null }) {
|
||||
const startBtn = el('button', { class: 'qs-btn qs-small', id: 'qs-wb-start', type: 'button', text: 'Start', title: 'A new archive, seeded with the program in the editor as the reference' });
|
||||
const task = el('select', { class: 'qs-input qs-small', id: 'qs-wb-task', 'aria-label': 'What each member must do' },
|
||||
el('option', { value: 'prepares', text: 'prepare the same state from |0…0⟩' }), el('option', { value: 'equivalent', text: 'be the same circuit (unitary)' }));
|
||||
|
|
@ -210,10 +212,15 @@ export function workbenchPane({ engine, draft, model, apply, save, found = null,
|
|||
selected = id;
|
||||
for (const p of mapBox.querySelectorAll('.qs-pt')) p.classList.toggle('qs-sel', p.dataset.id === id);
|
||||
const open = el('button', { class: 'qs-btn qs-small', id: 'qs-wb-open', type: 'button', text: 'Open in editor' });
|
||||
open.addEventListener('click', () => { openedFrom = id; apply(e.source, `${id} from the Workbench (${e.proposer}).`, byOf(e.proposer)); });
|
||||
open.addEventListener('click', () => {
|
||||
openedFrom = id;
|
||||
const ranked = elites() ?? [];
|
||||
if (ranked.length >= 2) onChoice?.({ kind: 'workbench', k: ranked.length, picked: Math.max(0, ranked.findIndex((x) => x.name === id)), best: 0 });
|
||||
apply(e.source, `${id} from the Workbench (${e.proposer}).`, byOf(e.proposer));
|
||||
});
|
||||
const chain = lineage(id).map((x, i) => el('button', { class: 'qs-chip', type: 'button', text: `${i ? '← ' : ''}${x.id} ${x.proposer}`, title: x.note ?? '', onclick: () => pick(x.id) }));
|
||||
const novel = el('div', { class: 'qs-wb-novelty', id: 'qs-wb-novelty' });
|
||||
engine.discoverNovelty?.(spec, archive, id).then((v) => {
|
||||
(features.on('novelty') ? engine.discoverNovelty?.(spec, archive, id) : null)?.then((v) => {
|
||||
if (selected !== id) return;
|
||||
const nv = noveltyLabel(v, { verified: true });
|
||||
const u = unaskedText(v, MEASURES);
|
||||
|
|
@ -268,6 +275,7 @@ export function workbenchPane({ engine, draft, model, apply, save, found = null,
|
|||
|
||||
// What is new in a kept entry, said and flagged when it is worth a second look.
|
||||
async function novelty(id, who, by) {
|
||||
if (!features.on('novelty')) return;
|
||||
const v = await engine.discoverNovelty?.(spec, archive, id).catch(() => null);
|
||||
const nv = noveltyLabel(v, { verified: true });
|
||||
if (!nv) return;
|
||||
|
|
@ -278,7 +286,7 @@ export function workbenchPane({ engine, draft, model, apply, save, found = null,
|
|||
// What a judged candidate adds to its outcome: an aberration's invitation to
|
||||
// revise a rule, and what is new in it once kept.
|
||||
async function after(o, who, source, by) {
|
||||
const ab = o.evaluation?.aberration;
|
||||
const ab = features.on('novelty') ? o.evaluation?.aberration : null;
|
||||
if (ab) {
|
||||
const broke = ab.broke?.kind === 'gates' ? `it uses ${ab.broke.gate}` : ab.broke?.kind === 'connectivity' ? 'it joins qubits that are not neighbours' : 'it breaks a rule';
|
||||
const why = `It breaks only one rule (${ab.rule}); without it, it would be verified (${ab.strength}) at cost ${num(ab.cost)}, ${MEASURES[spec.axes[0]]} ${ab.axes[0]}, ${MEASURES[spec.axes[1]]} ${ab.axes[1]}.`;
|
||||
|
|
@ -306,7 +314,7 @@ export function workbenchPane({ engine, draft, model, apply, save, found = null,
|
|||
say(`The blind search: ${total} edits judged, ${summary.stats.kept - before} kept.`);
|
||||
persist();
|
||||
// Only what is worth a second look is said of the search's finds.
|
||||
for (const id of kept) {
|
||||
for (const id of features.on('novelty') ? kept : []) {
|
||||
const nv = noveltyLabel(await engine.discoverNovelty?.(spec, archive, id).catch(() => null), { verified: true });
|
||||
if (nv?.flag) { say(`${id} (the blind search): ${nv.text}.`, 'find'); found?.flag({ kind: nv.flag, title: `${id} (the blind search): ${nv.text}`, why: nv.why, where: 'Workbench', source: entryById(id)?.source, by: byOf('mutate') }); }
|
||||
}
|
||||
|
|
|
|||
|
|
@ -51,6 +51,7 @@ self.onmessage = async (ev) => {
|
|||
case 'history': text = e().history(ev.data.name, JSON.stringify(ev.data.args ?? {})); break;
|
||||
case 'locks': text = mod.engine_locks(ev.data.name, JSON.stringify(ev.data.args ?? {})); break;
|
||||
case 'novelty': text = mod.engine_novelty(JSON.stringify(ev.data.args ?? {})); break;
|
||||
case 'study': text = e().study(ev.data.name, JSON.stringify(ev.data.args ?? {})); break;
|
||||
case 'converge': text = mod.engine_converge(ev.data.name, JSON.stringify(ev.data.args ?? {})); break;
|
||||
case 'rewrite': text = mod.engine_rewrite(ev.data.name, JSON.stringify(ev.data.args ?? {})); break;
|
||||
case 'taste': text = mod.engine_taste(ev.data.name, JSON.stringify(ev.data.args ?? {})); break;
|
||||
|
|
|
|||
|
|
@ -31,6 +31,11 @@
|
|||
// one taken as the engine's proposal, and the list following the program;
|
||||
// - your own way: a while on one's own, kept in this browser, Explore waiting for it, a
|
||||
// version made then noted so, the convergence line, and its end;
|
||||
// - the study: a feature turned off and on, nothing kept until asked, a choice among
|
||||
// Improve's candidates kept with the engine's best, an ownership answer, a probe done with
|
||||
// the assistant paused and scored by the engine, the measures, a sealed protocol joined
|
||||
// (its condition set by the key, the features held), the record exported sealed, and
|
||||
// leaving it;
|
||||
// - the classical-shortcut lens: a Clifford ring with a closing rotation layer priced by
|
||||
// the cut-rank sum, a Clifford circuit easy at any width, a contraction searched only
|
||||
// when thorough, and a check marked stale once the program changes;
|
||||
|
|
@ -302,6 +307,71 @@ const chip = (k) => page.$eval(`#qs-measures [data-k=${k}]`, (c) => c.textConten
|
|||
check(await page.isHidden('#qs-organic-chip') && /^Explore on my own/.test(await page.textContent('#qs-organic')), 'ended, the model may be asked again');
|
||||
}
|
||||
|
||||
// The study: features, a record kept, an ownership answer, a probe, a protocol joined, the record exported.
|
||||
{
|
||||
const SEALED = {"protocol":{"study":"the Studio's e2e study","hypotheses":["Without the taste profile, unassisted probe scores do not change."],"primary":"unassisted","conditions":[{"name":"everything"},{"name":"no taste","flags":{"taste":false}}],"control":"everything","analysis":{"resamples":1000,"seed":1,"level":0.95}},"seal":{"alg":"ed25519","canon":"rfc8785","key":"197f6b23e16c8532c6abc838facd5ea789be0c76b2920334039bfa8b3d368d61","digest":"a3d028f00e263d6ac33bfe61283f86e193316812f0f6bd16727108f937e39257","signature":"3f957f63a602925c49fa7fa010e29cea055ba4051180818f41d1d84539051027c4db992b5598ac36dd1145d6a0f408830743080b59ca6e80c129dbbdb0069d0a"}};
|
||||
if (!(await page.$eval('#qs-features', (d) => d.open))) await page.click('#qs-features > summary');
|
||||
await page.uncheck('#qs-flag-rewrites');
|
||||
check(await page.isHidden('#qs-rewrites-panel'), 'a feature turned off is not there: the rewrite lens');
|
||||
await page.check('#qs-flag-rewrites');
|
||||
check(await page.isVisible('#qs-rewrites-panel'), 'and turned on, it is back');
|
||||
check(await page.isHidden('#qs-study-tools'), 'nothing is kept until the reader asks');
|
||||
await page.check('#qs-study-keep');
|
||||
await page.waitForSelector('#qs-study-tools:not([hidden]) #qs-study-own', { timeout: LONG });
|
||||
// A choice among Improve's candidates is kept, with the engine's best of them.
|
||||
await setProgram('OPENQASM 3.0;\nqubit[2] q;\nh q[0];\nh q[0];\nx q[1];\nx q[1];\ncx q[0], q[1];\n');
|
||||
await page.click('#qs-improve');
|
||||
await page.waitForFunction(() => document.querySelectorAll('.qs-cand').length >= 2 && !/looking/.test(document.querySelector('#qs-alt-body')?.textContent ?? ''), null, { timeout: LONG });
|
||||
await page.click('.qs-cand >> nth=0 >> .qs-apply');
|
||||
await page.waitForSelector('#qs-review:not([hidden]) #qs-review-apply', { timeout: LONG });
|
||||
await page.click('#qs-review-apply');
|
||||
const kept = await page.evaluate(() => JSON.parse(localStorage.getItem('wai.quantum-studio.study')).choices);
|
||||
check(kept.length === 1 && kept[0].kind === 'alternatives' && kept[0].k >= 2 && kept[0].picked === 0 && Number.isInteger(kept[0].best), `the choice is kept, with how many were shown and the engine's best (${JSON.stringify(kept)})`);
|
||||
await page.click('#qs-study-own');
|
||||
await page.waitForSelector('#qs-own-4', { timeout: LONG });
|
||||
for (let i = 0; i < 5; i++) await page.selectOption(`#qs-own-${i}`, '6');
|
||||
await page.click('#qs-own-save');
|
||||
await page.waitForSelector('#qs-own-saved', { timeout: LONG });
|
||||
check(true, 'an answer on ownership is kept');
|
||||
await page.click('#qs-study-probe');
|
||||
await page.waitForSelector('#qs-probe-answer', { timeout: LONG });
|
||||
check(await page.isVisible('#qs-organic-chip'), 'during a probe the assistant is paused');
|
||||
await page.selectOption('#qs-probe-pick', 'swap');
|
||||
await page.fill('#qs-probe-answer', 'OPENQASM 3.0;\nqubit[2] q;\ncx q[0], q[1];\ncx q[1], q[0];\ncx q[0], q[1];\n');
|
||||
await page.click('#qs-probe-score');
|
||||
await page.waitForFunction(() => /^Verified\. Score 1\b/.test(document.querySelector('#qs-probe-result')?.textContent ?? ''), null, { timeout: LONG });
|
||||
check(true, `the engine scores the probe (${await page.textContent('#qs-probe-result')})`);
|
||||
await page.click('#qs-probe-close');
|
||||
check(await page.isHidden('#qs-organic-chip'), 'closed, the assistant is back');
|
||||
await page.click('#qs-study-record');
|
||||
await page.waitForSelector('#qs-study-measures', { timeout: LONG });
|
||||
const cell = (k) => page.textContent(`#qs-study-measures tr[data-measure=${k}] td.num`);
|
||||
check((await cell('ownership')) === '6' && (await cell('unassisted')) === '1' && /^(0|1|0\.5)$/.test(await cell('selection')), `the record's measures are the engine's (ownership ${await cell('ownership')}, unassisted ${await cell('unassisted')}, selection ${await cell('selection')})`);
|
||||
// A study: the protocol is read, joined, and its condition holds the features.
|
||||
await page.setInputFiles('#qs-study-file', { name: 'study.sealed.json', mimeType: 'application/json', buffer: Buffer.from(JSON.stringify(SEALED)) });
|
||||
await page.waitForSelector('#qs-study-join', { timeout: LONG });
|
||||
const offer = await page.textContent('#qs-study-offer');
|
||||
check(/Without the taste profile/.test(offer) && /digest a3d028f00e263d6a/.test(offer), 'the protocol is shown before joining: what it studies, and its digest');
|
||||
await page.click('#qs-study-join');
|
||||
await page.waitForFunction(() => /You take part in/.test(document.querySelector('#qs-study-status')?.textContent ?? ''), null, { timeout: LONG });
|
||||
const cond = (await page.textContent('#qs-study-status')).match(/condition (everything|no taste)/)?.[1];
|
||||
check(cond && (cond === 'no taste') === (await page.isDisabled('#qs-flag-taste')) && (cond === 'no taste') === (await page.isHidden('#qs-taste')) && !(await page.isDisabled('#qs-flag-rewrites')) && !(await page.isChecked('#qs-predict-first')),
|
||||
`joined: the key's condition (${cond}) sets the features it names and holds them; the others stay the reader's`);
|
||||
const files = [];
|
||||
page.on('download', (d) => files.push(d));
|
||||
await page.click('#qs-study-export');
|
||||
await page.waitForSelector('#qs-study-exported', { timeout: LONG });
|
||||
await page.waitForTimeout(500);
|
||||
const sealedRecord = files.length ? JSON.parse(readFileSync(await files.at(-1).path(), 'utf8')) : null;
|
||||
const r = sealedRecord?.record;
|
||||
check(r?.protocol === SEALED.seal.digest && /^did:key:z6Mk/.test(r?.participant ?? '') && r.condition === cond && sealedRecord.seal?.signature?.length === 128 && r.ownership.length === 1 && r.probes.length === 1 && r.choices.length >= 1,
|
||||
`Export my record saves it sealed with this browser's key, for the researcher (${files.at(-1)?.suggestedFilename()})`);
|
||||
await page.click('#qs-study-leave');
|
||||
check(!(await page.isDisabled('#qs-flag-taste')) && await page.isVisible('#qs-taste') && !(await page.isChecked('#qs-predict-first')), 'left, the features are the reader’s again, as they set them');
|
||||
// The Study pane is the last of the side column: back to the top, where the editor's lines are drawn.
|
||||
await page.evaluate(() => window.scrollTo(0, 0));
|
||||
}
|
||||
|
||||
// Versions: each change and who made it; a proposal taken change by change;
|
||||
// whose hand each line and gate is in; the run's receipt binding the count; a
|
||||
// line of work from an older version, brought back in; and all of it kept.
|
||||
|
|
@ -329,6 +399,9 @@ const chip = (k) => page.$eval(`#qs-measures [data-k=${k}]`, (c) => c.textConten
|
|||
check(/proposed by the assistant \(scripted\)/.test(await here()) && /1 of 2 taken/.test(await here()), `only the change taken reaches the program, recorded as the assistant's (${await here()})`);
|
||||
check(/1 of 2 changes taken; the rest were not taken/.test(await page.textContent('#qs-vs-detail')), 'the version keeps what was rejected');
|
||||
await page.check('#qs-vs-hands');
|
||||
// The editor draws gutter marks only for the lines in view, and it is as tall as the
|
||||
// side column: its first lines are brought into view.
|
||||
await page.evaluate(() => document.querySelector('.cm-editor').scrollIntoView({ block: 'start' }));
|
||||
await page.waitForFunction(() => document.querySelectorAll('.qs-hand-gutter .qs-hand:not(.qs-hand-spacer)').length === 7, null, { timeout: LONG });
|
||||
const gutter = await page.$$eval('.qs-hand-gutter .qs-hand:not(.qs-hand-spacer)', (ms) => ms.map((m) => m.className.replace('qs-hand qs-hand-', '')));
|
||||
await page.waitForFunction(() => document.querySelectorAll('#qs-diagram .qc-item.qc-by-assistant').length === 1, null, { timeout: LONG });
|
||||
|
|
@ -532,6 +605,9 @@ const wbIdle = () => page.waitForFunction(() => !document.querySelector('#qs-wb-
|
|||
check(/GHZ, by doubling/.test(novel) && /Better than the reference on what the cost does not weigh: two-qubit depth \d+ against \d+/.test(novel), `the entry says what it is and what it beats the reference on unasked (${novel})`);
|
||||
// Your taste: two kept generators, which the engine proves make the same state, compared.
|
||||
if (!(await page.$eval('#qs-taste', (d) => d.open))) await page.click('#qs-taste > summary');
|
||||
// The choices the profile already holds: the alternatives taken whole above.
|
||||
const tasteBefore = Number((await page.textContent('#qs-taste-count')) || 0);
|
||||
const tasteNow = String(tasteBefore + 1);
|
||||
await page.click('#qs-taste-compare');
|
||||
await page.waitForSelector('.qs-taste-pick', { timeout: LONG });
|
||||
const pair = await page.$$eval('.qs-taste-side', (ss) => ss.map((s) => s.dataset.name));
|
||||
|
|
@ -540,8 +616,8 @@ const wbIdle = () => page.waitForFunction(() => !document.querySelector('#qs-wb-
|
|||
await page.click('.qs-taste-side >> nth=0 >> .qs-taste-pick');
|
||||
await page.waitForSelector('#qs-taste-chosen', { timeout: LONG });
|
||||
check(new RegExp(`^Kept: ${pair[0]} over ${pair[1]}, for its economy`).test(await page.textContent('#qs-taste-chosen')), `the choice is kept, with what it was about (${await page.textContent('#qs-taste-chosen')})`);
|
||||
await page.waitForFunction(() => document.querySelector('#qs-taste-count')?.textContent === '2', null, { timeout: LONG });
|
||||
check(/^2 choices \(economy 1\)/.test(await page.textContent('#qs-taste-summary')), `the profile counts its choices and their parts (${await page.textContent('#qs-taste-summary')})`);
|
||||
await page.waitForFunction((n) => document.querySelector('#qs-taste-count')?.textContent === n, tasteNow, { timeout: LONG });
|
||||
check(new RegExp(`^${tasteNow} choices \\(economy 1\\)`).test(await page.textContent('#qs-taste-summary')), `the profile counts its choices and their parts (${await page.textContent('#qs-taste-summary')})`);
|
||||
await page.click('#qs-taste-rank');
|
||||
await page.waitForSelector('#qs-taste-list li', { timeout: LONG });
|
||||
const ranked = await page.$$eval('#qs-taste-list li', (ls) => ls.map((l) => l.textContent));
|
||||
|
|
@ -559,7 +635,7 @@ const wbIdle = () => page.waitForFunction(() => !document.querySelector('#qs-wb-
|
|||
check((await page.textContent('#qs-taste-count')) === '', 'a second profile is kept apart: it has no choices');
|
||||
await page.fill('#qs-taste-profile', 'mine');
|
||||
await page.dispatchEvent('#qs-taste-profile', 'change');
|
||||
await page.waitForFunction(() => document.querySelector('#qs-taste-count')?.textContent === '2', null, { timeout: LONG });
|
||||
await page.waitForFunction((n) => document.querySelector('#qs-taste-count')?.textContent === n, tasteNow, { timeout: LONG });
|
||||
check(true, 'and the first comes back with its choices');
|
||||
await setProgram(CHAIN);
|
||||
await page.click('#qs-wb-open');
|
||||
|
|
@ -584,7 +660,7 @@ const wbIdle = () => page.waitForFunction(() => !document.querySelector('#qs-wb-
|
|||
await page.waitForFunction(() => /Taken up again: 102 evaluated/.test(document.querySelector('#qs-wb-log')?.textContent ?? ''), null, { timeout: LONG });
|
||||
check(/by person/.test(await page.textContent('#qs-wb-stats')) && (await page.inputValue('#qs-wb-sizes')) === 'n = 3, 5, 8', 'the archive and its specification come back with the page');
|
||||
await lintSays(/qubits/);
|
||||
check((await page.textContent('#qs-taste-count')) === '2' && (await page.inputValue('#qs-taste-profile')) === 'mine', 'the taste profiles come back with the page');
|
||||
check((await page.textContent('#qs-taste-count')) === tasteNow && (await page.inputValue('#qs-taste-profile')) === 'mine', 'the taste profiles come back with the page');
|
||||
check(/^\d+ flagged · 1 saved$/.test(await page.textContent('#qs-found-count')) && (await page.inputValue('#qs-found-list .qs-found-item.saved input')) === 'the ladder, read backwards', 'what Found keeps comes back with the page, with its note');
|
||||
// On a line, the tree breaks only the rule that gates join neighbours: an aberration.
|
||||
if (!(await page.$eval('#qs-wb-spec', (d) => d.open))) await page.click('#qs-wb-spec > summary');
|
||||
|
|
|
|||
|
|
@ -155,6 +155,7 @@ A cheap route proves a design classically easy. None being cheap proves nothing
|
|||
- `novelty_qasm` and `discover_novelty` (what is new, below)
|
||||
- `rewrite_qasm` (rewrites, below)
|
||||
- `converge_cohort` and `converge_history` (convergence, below)
|
||||
- `study_measure` and `study_cohort` (a study, below)
|
||||
- `taste_rank` and `taste_next` (your taste, below)
|
||||
- `history_commit`, `history_changes`, `history_blame` and `history_log` (a program's history, below)
|
||||
- `list_methods`
|
||||
|
|
@ -289,6 +290,33 @@ wai-quantum converge cohort designs/*.qasm --input n=6 # the Vendi score, the
|
|||
|
||||
Everything is read from the files given, and nothing is sent anywhere.
|
||||
|
||||
## A study
|
||||
|
||||
`wai-quantum study` measures whether the co-design works, with every measure computed by the engine:
|
||||
|
||||
```sh
|
||||
wai-quantum study catalog # the flags a condition sets, the measures, the probe tasks
|
||||
wai-quantum study new protocol.json # check it and seal it with your key, before any data
|
||||
wai-quantum study assign protocol.sealed.json # the condition your key is assigned
|
||||
wai-quantum study measure record.json # one participant's measures
|
||||
wai-quantum study cohort protocol.sealed.json records/*.json --out report.json
|
||||
```
|
||||
|
||||
- **The protocol** names the study, its hypotheses, the primary measure, two conditions or more (each a setting of the Studio's features), the control, and the analysis (resamples, seed, level). It is sealed before any data, so it cannot change once people take part.
|
||||
- **Assignment** is the protocol's digest and the participant's key hashed together. It is fixed before they begin, and anyone can check it.
|
||||
- **A record** is what the Studio's "Export my record" saves, sealed with the participant's key. It holds their program's history, predictions, choices among candidates, probe tasks done without the assistant, and answers on ownership.
|
||||
- **The measures:**
|
||||
- quality: the edits the engine proves equal to what they replaced, and how many made it cheaper, with its trend over time;
|
||||
- diversity (Vendi) and novelty;
|
||||
- selection skill: how often the engine's best of the candidates shown was taken;
|
||||
- calibration of predictions against the runs;
|
||||
- unassisted skill: the probes, scored again by the engine;
|
||||
- ownership answers, beside the authorship share by the history;
|
||||
- convergence on a model's proposals.
|
||||
- **The cohort report** checks every record against the protocol (its digest, its seal, the condition its key is assigned) and refuses one that fails, with the reason. It gives the measures per condition, each condition's difference from the control on the primary measure with a seeded bootstrap interval, Cronbach's alpha of the ownership items, and the Vendi score of each condition's final designs. `--out` saves it sealed with your key.
|
||||
|
||||
Records are given to whoever runs the study; nothing is sent anywhere.
|
||||
|
||||
## Your taste
|
||||
|
||||
`wai-quantum taste` learns your taste in circuits from your own choices:
|
||||
|
|
|
|||
|
|
@ -46,6 +46,7 @@ mod lab;
|
|||
mod locks;
|
||||
mod mcp;
|
||||
mod rewrite;
|
||||
mod study;
|
||||
mod taste;
|
||||
|
||||
use clap::{Args, CommandFactory, Parser, Subcommand};
|
||||
|
|
@ -219,6 +220,11 @@ enum Command {
|
|||
#[command(subcommand)]
|
||||
cmd: locks::Cmd,
|
||||
},
|
||||
/// Measuring whether the co-design works: a protocol sealed before any data, conditions by key, records, and the cohort report (`wai-quantum study --help`)
|
||||
Study {
|
||||
#[command(subcommand)]
|
||||
cmd: study::Cmd,
|
||||
},
|
||||
/// Whether your designs converge on a model's proposals, and how diverse a set of programs is (`wai-quantum converge --help`)
|
||||
Converge {
|
||||
#[command(subcommand)]
|
||||
|
|
@ -740,6 +746,7 @@ fn main() {
|
|||
Command::Taste { cmd } => taste::main(cmd),
|
||||
Command::Rewrite { cmd } => rewrite::main(cmd),
|
||||
Command::Converge { cmd } => converge::main(cmd),
|
||||
Command::Study { cmd } => study::main(cmd),
|
||||
Command::Novelty { file, against, reference, weigh, json } => {
|
||||
let inputs = INPUTS.get().cloned().unwrap_or_default();
|
||||
let text = read_text(file.as_deref()).unwrap_or_else(|e| die(&e));
|
||||
|
|
|
|||
|
|
@ -150,6 +150,18 @@ fn tools() -> Value {
|
|||
"description": "Along a program's history (the path of NAME.history.json): for each version the person made after a model proposed, how far it is built from the proposals and the share of its gates in a model's hand; whether that is converging, and why; and the versions made exploring on their own set beside the rest. When it converges, propose less and ask more.",
|
||||
"inputSchema": { "type": "object", "required": ["history"], "properties": { "history": { "type": "string", "description": "The history file's path." }, "inputs": inputs_schema() } },
|
||||
},
|
||||
{
|
||||
"name": "study_measure",
|
||||
"title": "A study participant's measures",
|
||||
"description": "The engine's measures of one participant's record (the file the Studio's \"Export my record\" saves): quality (the edits the engine proves equal to what they replaced, and whether they made it cheaper) and its trend, diversity (Vendi), novelty, selection skill, calibration of predictions, unassisted probe scores (scored again by the engine), ownership answers and authorship share, and convergence on a model's proposals.",
|
||||
"inputSchema": { "type": "object", "required": ["record"], "properties": { "record": { "type": "string", "description": "The record file's path." } } },
|
||||
},
|
||||
{
|
||||
"name": "study_cohort",
|
||||
"title": "A study's report",
|
||||
"description": "A study's report from its sealed protocol and the participants' sealed records: each record checked (the protocol's digest, its seal, the condition its key is assigned) and refused with the reason if it fails; the measures per condition; each condition's difference from the control on the primary measure with a seeded bootstrap interval; Cronbach's alpha of the ownership items; the Vendi score of each condition's final designs. The analysis is the one the protocol fixed before any data.",
|
||||
"inputSchema": { "type": "object", "required": ["protocol", "records"], "properties": { "protocol": { "type": "string", "description": "The sealed protocol's path." }, "records": { "type": "array", "items": { "type": "string" }, "description": "The sealed records' paths." } } },
|
||||
},
|
||||
{
|
||||
"name": "taste_rank",
|
||||
"title": "Rank programs by the person's taste",
|
||||
|
|
@ -247,6 +259,20 @@ fn call(name: &str, args: &Value, memory_gb: u64, key: &SigningKey, client: &Cli
|
|||
"list_methods" => Ok(json!({ "methods": catalog::describe(memory_gb << 30) })),
|
||||
"novelty_qasm" => wai_quantum::engine::novelty::call(raw),
|
||||
"converge_cohort" => wai_quantum::engine::converge::call("cohort", raw),
|
||||
"study_measure" | "study_cohort" => {
|
||||
let read = |p: &str| -> Result<Value, String> {
|
||||
let t = std::fs::read_to_string(p).map_err(|e| format!("{p}: {e}"))?;
|
||||
serde_json::from_str(&t).map_err(|e| format!("{p}: not JSON: {e}"))
|
||||
};
|
||||
if name == "study_measure" {
|
||||
let v = read(raw.get("record").and_then(Value::as_str).ok_or("give \"record\": the record file's path")?)?;
|
||||
wai_quantum::engine::study::measure(v.get("record").unwrap_or(&v))
|
||||
} else {
|
||||
let sealed = read(raw.get("protocol").and_then(Value::as_str).ok_or("give \"protocol\": the sealed protocol's path")?)?;
|
||||
let records = raw.get("records").and_then(Value::as_array).ok_or("give \"records\": the sealed records' paths")?.iter().map(|p| read(p.as_str().unwrap_or_default())).collect::<Result<Vec<_>, String>>()?;
|
||||
wai_quantum::engine::study::cohort(&sealed, &records)
|
||||
}
|
||||
}
|
||||
"converge_history" => {
|
||||
let path = raw.get("history").and_then(Value::as_str).ok_or("give \"history\": the history file's path")?;
|
||||
let h = crate::history::load(std::path::Path::new(path))?;
|
||||
|
|
@ -470,7 +496,7 @@ mod tests {
|
|||
assert!(handle(&json!({ "jsonrpc": "2.0", "method": "notifications/initialized" }), 1, &key(), &mut Client::default()).is_none());
|
||||
let list = handle(&json!({ "jsonrpc": "2.0", "id": 2, "method": "tools/list" }), 1, &key(), &mut Client::default()).unwrap();
|
||||
let names: Vec<&str> = list["result"]["tools"].as_array().unwrap().iter().map(|t| t["name"].as_str().unwrap()).collect();
|
||||
assert_eq!(names, ["run_quantum_circuit", "plan_quantum_circuit", "lint_qasm", "critique_qasm", "check_equivalent", "improve_qasm", "classical_shortcuts", "draw_circuit", "verify_quantum_result", "discover_evaluate", "discover_submit", "discover_archive", "novelty_qasm", "discover_novelty", "rewrite_qasm", "converge_cohort", "converge_history", "taste_rank", "taste_next", "history_commit", "history_changes", "history_blame", "history_log", "list_methods"]);
|
||||
assert_eq!(names, ["run_quantum_circuit", "plan_quantum_circuit", "lint_qasm", "critique_qasm", "check_equivalent", "improve_qasm", "classical_shortcuts", "draw_circuit", "verify_quantum_result", "discover_evaluate", "discover_submit", "discover_archive", "novelty_qasm", "discover_novelty", "rewrite_qasm", "converge_cohort", "converge_history", "study_measure", "study_cohort", "taste_rank", "taste_next", "history_commit", "history_changes", "history_blame", "history_log", "list_methods"]);
|
||||
let run = handle(
|
||||
&json!({ "jsonrpc": "2.0", "id": 3, "method": "tools/call", "params": { "name": "run_quantum_circuit", "arguments": { "qasm": "OPENQASM 3.0; qubit[2] q; h q[0]; cx q[0], q[1];", "observables": ["Z0 Z1"] } } }),
|
||||
1,
|
||||
|
|
@ -530,6 +556,18 @@ mod tests {
|
|||
assert!(call("lint_qasm", &json!({ "qasm": star }), 1, &key(), &Client::default()).unwrap_err().contains("give n a value"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_agent_reads_a_participant_s_measures() {
|
||||
let dir = std::env::temp_dir().join(format!("wai-quantum-mcp-study-{}", std::process::id()));
|
||||
let _ = std::fs::create_dir_all(&dir);
|
||||
let rec = dir.join("record.json");
|
||||
std::fs::write(&rec, json!({ "record": { "choices": [{ "k": 2, "picked": 0, "best": 0 }], "ownership": [{ "items": [6, 6, 6, 6, 6] }] } }).to_string()).unwrap();
|
||||
let m = call("study_measure", &json!({ "record": rec.to_str().unwrap() }), 1, &key(), &Client::default()).unwrap();
|
||||
assert_eq!((m["selection"]["rate"].as_f64(), m["ownership"]["mean"].as_f64()), (Some(1.0), Some(6.0)), "{m}");
|
||||
assert!(call("study_cohort", &json!({ "protocol": dir.join("none.json").to_str().unwrap(), "records": [] }), 1, &key(), &Client::default()).is_err());
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_agent_reads_a_cohort_s_diversity() {
|
||||
let a = "OPENQASM 3.0;\nqubit[1] q;\nh q[0];\n";
|
||||
|
|
|
|||
139
wai/wai-quantum-cli/src/study.rs
Normal file
139
wai/wai-quantum-cli/src/study.rs
Normal file
|
|
@ -0,0 +1,139 @@
|
|||
//! `wai-quantum study`: measuring whether the co-design works
|
||||
//! ([`wai_quantum::engine::study`]). A researcher writes a protocol and seals
|
||||
//! it before any data; each participant's condition follows from their key;
|
||||
//! participants give their sealed records (the Studio's "Export my record");
|
||||
//! the cohort report checks each against the protocol and computes the
|
||||
//! measures here. Nothing is sent anywhere.
|
||||
|
||||
use clap::Subcommand;
|
||||
use serde_json::{Value, json};
|
||||
use std::path::{Path, PathBuf};
|
||||
use wai_quantum::engine::study;
|
||||
|
||||
#[derive(Subcommand)]
|
||||
pub enum Cmd {
|
||||
/// Check a protocol and seal it with your key, before any data: NAME.sealed.json beside it
|
||||
New {
|
||||
protocol: PathBuf,
|
||||
#[command(flatten)]
|
||||
key: super::KeyArgs,
|
||||
},
|
||||
/// The condition a participant is assigned: yours (your key's did) by default
|
||||
Assign {
|
||||
sealed: PathBuf,
|
||||
/// A participant's did:key
|
||||
#[arg(long)]
|
||||
participant: Option<String>,
|
||||
#[command(flatten)]
|
||||
key: super::KeyArgs,
|
||||
},
|
||||
/// The flags a condition may set, the measures a protocol may name, and the probe tasks
|
||||
Catalog,
|
||||
/// The engine's measures of one record (sealed or not)
|
||||
Measure {
|
||||
record: PathBuf,
|
||||
#[arg(long)]
|
||||
json: bool,
|
||||
},
|
||||
/// The study's report: every record checked against the sealed protocol, the measures per condition, and the primary measure against the control
|
||||
Cohort {
|
||||
sealed: PathBuf,
|
||||
records: Vec<PathBuf>,
|
||||
/// Write the report here, sealed with your key
|
||||
#[arg(long)]
|
||||
out: Option<PathBuf>,
|
||||
#[command(flatten)]
|
||||
key: super::KeyArgs,
|
||||
#[arg(long)]
|
||||
json: bool,
|
||||
},
|
||||
}
|
||||
|
||||
fn read_json(p: &Path) -> Result<Value, String> {
|
||||
let t = std::fs::read_to_string(p).map_err(|e| format!("{}: {e}", p.display()))?;
|
||||
serde_json::from_str(&t).map_err(|e| format!("{}: not JSON: {e}", p.display()))
|
||||
}
|
||||
|
||||
fn write_json(p: &Path, v: &Value) -> Result<(), String> {
|
||||
std::fs::write(p, serde_json::to_string_pretty(v).expect("JSON") + "\n").map_err(|e| format!("{}: {e}", p.display()))
|
||||
}
|
||||
|
||||
/// A report in words.
|
||||
pub fn cohort_text(r: &Value) -> String {
|
||||
let mut out = format!("{} (protocol {}): {} of {} records included\n", r["study"].as_str().unwrap_or(""), &r["protocol"].as_str().unwrap_or("")[..16.min(r["protocol"].as_str().unwrap_or("").len())], r["included"], r["records"]);
|
||||
for x in r["refused"].as_array().into_iter().flatten() {
|
||||
out += &format!(" refused record {}: {}\n", x["record"], x["why"].as_str().unwrap_or(""));
|
||||
}
|
||||
let primary = r["primary"].as_str().unwrap_or("");
|
||||
for c in r["conditions"].as_array().into_iter().flatten() {
|
||||
let m = &c["means"][primary];
|
||||
out += &format!("{:<20} {} participants, mean {primary} {}", c["condition"].as_str().unwrap_or(""), c["participants"], if m.is_null() { "—".to_string() } else { m.to_string() });
|
||||
if let Some(d) = c.get("primary").filter(|d| !d["difference"].is_null()) {
|
||||
out += &format!(", against {}: {:+} ({} to {}){}", r["control"].as_str().unwrap_or(""), d["difference"].as_f64().unwrap_or(0.0), d["low"], d["high"], if d["excludes_zero"] == true { ", the interval excludes 0" } else { "" });
|
||||
}
|
||||
out += "\n";
|
||||
}
|
||||
if let Some(a) = r["ownership_alpha"].as_f64() {
|
||||
out += &format!("ownership items: Cronbach's alpha {a} over {} answers\n", r["ownership_answers"]);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
pub fn main(cmd: Cmd) {
|
||||
let r = (|| match cmd {
|
||||
Cmd::New { protocol, key } => {
|
||||
let p = study::Protocol::from_json(&read_json(&protocol)?)?;
|
||||
let k = super::load_key(&key);
|
||||
let sealed = study::seal_protocol(&p, &k.key);
|
||||
let out = protocol.with_extension("sealed.json");
|
||||
write_json(&out, &sealed)?;
|
||||
println!("{}: sealed, digest {}", out.display(), sealed["seal"]["digest"].as_str().unwrap_or(""));
|
||||
Ok(())
|
||||
}
|
||||
Cmd::Assign { sealed, participant, key } => {
|
||||
let (p, digest) = study::open_protocol(&read_json(&sealed)?)?;
|
||||
let who = match participant {
|
||||
Some(d) => d,
|
||||
None => wai_quantum::quantum_receipt::did_key(&super::load_key(&key).key.verifying_key()),
|
||||
};
|
||||
let a = study::assign(&p, &digest, &who);
|
||||
println!("{who}: {} {}", a["condition"].as_str().unwrap_or(""), a["flags"]);
|
||||
Ok(())
|
||||
}
|
||||
Cmd::Catalog => {
|
||||
for k in ["flags", "measures", "probes"] {
|
||||
println!("{k}:");
|
||||
for x in study::call(k, &json!({}))?.as_array().into_iter().flatten() {
|
||||
let (name, what) = (x.get("flag").or(x.get("measure")).or(x.get("probe")), x.get("what").or(x.get("task")));
|
||||
println!(" {:<18} {}", name.and_then(Value::as_str).unwrap_or(""), what.and_then(Value::as_str).unwrap_or(""));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
Cmd::Measure { record, json } => {
|
||||
let v = read_json(&record)?;
|
||||
let m = study::measure(v.get("record").unwrap_or(&v))?;
|
||||
if json {
|
||||
println!("{m}");
|
||||
} else {
|
||||
for (k, x) in study::scalars(&m) {
|
||||
println!("{k:<18} {x}");
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
Cmd::Cohort { sealed, records, out, key, json } => {
|
||||
let rs = records.iter().map(|p| read_json(p)).collect::<Result<Vec<_>, String>>()?;
|
||||
let report = study::cohort(&read_json(&sealed)?, &rs)?;
|
||||
if let Some(o) = out {
|
||||
let k = super::load_key(&key);
|
||||
let seal = wai_quantum::engine::receipt::seal(&k.key, &json!({ "study_report": report["protocol"] }), &report);
|
||||
write_json(&o, &json!({ "report": report, "seal": seal }))?;
|
||||
eprintln!("{}: the report, sealed", o.display());
|
||||
}
|
||||
if json { println!("{report}") } else { print!("{}", cohort_text(&report)) }
|
||||
Ok(())
|
||||
}
|
||||
})();
|
||||
r.unwrap_or_else(|e: String| super::die(&e));
|
||||
}
|
||||
|
|
@ -26,7 +26,7 @@ use std::process::{Child, Command, Stdio};
|
|||
use std::sync::{Arc, Mutex};
|
||||
use wai_quantum::ed25519_dalek::SigningKey;
|
||||
use wai_quantum::engine::discover::{self, Archive, Budget, Evaluator, SearchOptions, Spec, search};
|
||||
use wai_quantum::engine::{catalog, converge, critique, draw, energy, equiv, job, locks, novelty, rewrite, shortcuts, taste, versions};
|
||||
use wai_quantum::engine::{catalog, converge, critique, draw, energy, equiv, job, locks, novelty, rewrite, shortcuts, study, taste, versions};
|
||||
|
||||
/// What a result says of its energy here.
|
||||
const UNMETERED: &str = "the desktop app does not meter a run's energy";
|
||||
|
|
@ -223,6 +223,17 @@ fn novelty(args: String) -> String {
|
|||
.unwrap_or_else(error_json)
|
||||
}
|
||||
|
||||
/// `name` on the study ([`study::call_sealing`]), sealing with the app's key
|
||||
/// where it seals: the answer, or `{error}`.
|
||||
#[tauri::command]
|
||||
fn study(studio: tauri::State<'_, Studio>, name: String, args: String) -> String {
|
||||
serde_json::from_str::<Value>(&args)
|
||||
.map_err(|e| format!("{name}: the arguments are not JSON: {e}"))
|
||||
.and_then(|a| study::call_sealing(&name, &a, &studio.key))
|
||||
.map(|v| v.to_string())
|
||||
.unwrap_or_else(error_json)
|
||||
}
|
||||
|
||||
/// `name` on the convergence monitor ([`converge::call`]): the answer, or
|
||||
/// `{error}`.
|
||||
#[tauri::command]
|
||||
|
|
@ -464,7 +475,7 @@ fn main() {
|
|||
};
|
||||
tauri::Builder::default()
|
||||
.manage(Studio { key, job_bytes: job_bytes(), running: Arc::new(Mutex::new(None)), cancelled: Arc::new(Mutex::new(false)) })
|
||||
.invoke_handler(tauri::generate_handler![run_job, cancel, plan_job, methods, lint, critique, equivalent, variants, shortcuts, draw, layout, verify, signer, save_job, save_program, chat, models, discover_start, discover_submit, discover_search, discover_evaluate, discover_novelty, instantiate, history, locks, novelty, rewrite, taste, converge])
|
||||
.invoke_handler(tauri::generate_handler![run_job, cancel, plan_job, methods, lint, critique, equivalent, variants, shortcuts, draw, layout, verify, signer, save_job, save_program, chat, models, discover_start, discover_submit, discover_search, discover_evaluate, discover_novelty, instantiate, history, locks, novelty, rewrite, taste, converge, study])
|
||||
.run(tauri::generate_context!())
|
||||
.expect("the window opens");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -68,7 +68,7 @@ never creates information.
|
|||
| `quantum` | byte-exact statevector simulation (dyadic Clifford+T+P(k)), as `wai.quantum.circuit2` by default or as the frozen `wai.quantum.circuit` |
|
||||
| `quantum_toolchain` | algorithm library, backend recommendation, **OpenQASM 2/3 ingest of what the mainstream toolchains actually export** — both register and both measurement spellings, `rx`/`ry`/`rz`/`sx`/`u`/`u2`/`u3`/`crz`, and exact rational-multiple-of-pi angles (non-dyadic angles are refused, never rounded); **device-ready OpenQASM 2/3** (`device::device_qasm`): every gate lowered exactly to its dialect's standard library, multi-controlled gates included, every qubit measured into `c[i]`, and the service's counts read back onto the crate's basis; **native programs** (`native::compile`): a processor's own basis (`rz`/`sx`/`x`/`cz`; `prx`/`cz`; or `gpi`/`gpi2`/`zz` with angles in turns, written exactly; the Z rotations carried in the phases of the last two) on its own qubits, routed over its coupling map, and checked against the circuit by simulation before it is sent — Bloch / entanglement / purity analysis |
|
||||
| `quantum_qasm` | **the OpenQASM reader the job service runs**, for every surface that takes a program: OpenQASM 3 and the OpenQASM 2 forms in common use, every register, a gate on a whole register, gate definitions with parameters, `ctrl`/`negctrl` modifiers, angle expressions, comments, and measurement in every spelling. A program that measures and then acts, resets after a gate or uses `if` is kept as dynamic steps. Generators: integer `input`s, `const`s and `for` loops (ranges, steps, sets, nested), with integer arithmetic in indices and sizes and `if` on integers decided on reading, so one program describes a family of circuits; `qasm::parse_with` reads it at given inputs and `qasm::instantiate` writes that circuit out as plain OpenQASM. `while`, `inv`, `pow` and anything else not listed are refused with their line. Lowered to each method's input exactly up to a global phase. `qasm::dyadic_circuit` gives the exact state vector's circuit (dyadic angles) for a program whose measurements all come at the end, and refuses any other with the reason |
|
||||
| `quantum_engine` | **the quantum job service's engine, for every host**: a job (an OpenQASM program and what is asked of it: shots, Pauli expectation values, amplitudes, noise) planned onto the strongest method it admits, with the reason and the memory and work it will take; run; answered in the service's JSON with its accuracy class; and sealed over the request and the result in their canonical JSON (`engine::job`, `engine::receipt`), so a result from the service, the command line or a browser tab checks the same way. Also a program's circuit drawn as text (`engine::draw`); the specification-and-evaluator workbench (`engine::discover`: a specification, the engine's verdict on a generator at every size, and an archive of the cheapest verified generator in every cell of two measures, sealed); a program's history, attributed (`engine::versions`: versions in a sealed chain, each with who made it and how, proposals reviewed change by change, and each line and gate in the hand that wrote it, which a job's receipt can bind); the person's rules for proposals (`engine::locks`: held blocks, the gates allowed, the qubit declarations, what to avoid and a region, checked change by change); what is new in a program (`engine::novelty`: its structural distance to the known constructions and to programs given, each known construction it is proved to be and whether by a different route, and the strengths a cost did not ask for; a workbench candidate that breaks only a rule is an aberration, with the cost it would have); convergence (`engine::converge`: along a program's history, how far the person's versions are built from a model's proposals and the model's share of the gates, with the versions made on their own set beside the rest; and a cohort's diversity, its Vendi score, centroid and narrowing over time); rewrites that keep a circuit's meaning (`engine::rewrite`: cancellations, merges, Hadamard conjugations and gates carried past what they commute with, each checked on the qubits it touches, so a chain of them proves two programs equal at any width, explains an optimisation, and offers a person one rewrite at a time); a person's taste (`engine::taste`: a Bradley–Terry model of their own choices between programs the engine proves equal, part by part of a rubric, with typicality shown beside it and moved only by a dial, and how well it predicts them measured); and the CPU energy the kernel counts for one process (`engine::energy`). It builds for `wasm32-unknown-unknown`, where it runs on one thread and claims no time |
|
||||
| `quantum_engine` | **the quantum job service's engine, for every host**: a job (an OpenQASM program and what is asked of it: shots, Pauli expectation values, amplitudes, noise) planned onto the strongest method it admits, with the reason and the memory and work it will take; run; answered in the service's JSON with its accuracy class; and sealed over the request and the result in their canonical JSON (`engine::job`, `engine::receipt`), so a result from the service, the command line or a browser tab checks the same way. Also a program's circuit drawn as text (`engine::draw`); the specification-and-evaluator workbench (`engine::discover`: a specification, the engine's verdict on a generator at every size, and an archive of the cheapest verified generator in every cell of two measures, sealed); a program's history, attributed (`engine::versions`: versions in a sealed chain, each with who made it and how, proposals reviewed change by change, and each line and gate in the hand that wrote it, which a job's receipt can bind); the person's rules for proposals (`engine::locks`: held blocks, the gates allowed, the qubit declarations, what to avoid and a region, checked change by change); what is new in a program (`engine::novelty`: its structural distance to the known constructions and to programs given, each known construction it is proved to be and whether by a different route, and the strengths a cost did not ask for; a workbench candidate that breaks only a rule is an aberration, with the cost it would have); a study (`engine::study`: a protocol sealed before any data, each participant's condition fixed by their key, their record sealed with it, the measures computed by the engine from what they made, and a cohort report with seeded bootstrap intervals and Cronbach's alpha); convergence (`engine::converge`: along a program's history, how far the person's versions are built from a model's proposals and the model's share of the gates, with the versions made on their own set beside the rest; and a cohort's diversity, its Vendi score, centroid and narrowing over time); rewrites that keep a circuit's meaning (`engine::rewrite`: cancellations, merges, Hadamard conjugations and gates carried past what they commute with, each checked on the qubits it touches, so a chain of them proves two programs equal at any width, explains an optimisation, and offers a person one rewrite at a time); a person's taste (`engine::taste`: a Bradley–Terry model of their own choices between programs the engine proves equal, part by part of a rubric, with typicality shown beside it and moved only by a dial, and how well it predicts them measured); and the CPU energy the kernel counts for one process (`engine::energy`). It builds for `wasm32-unknown-unknown`, where it runs on one thread and claims no time |
|
||||
| `quantum_stabilizer` | stabilizer (CHP) tableau — `O(n²)`, scales far past statevector |
|
||||
| `quantum_mps` | tensor-network (matrix-product-state) backend |
|
||||
| `quantum_pauli` | sparse-Pauli / Heisenberg observable propagation over the dyadic gate set (up to 64 qubits) |
|
||||
|
|
|
|||
|
|
@ -53,6 +53,11 @@
|
|||
//! caller's own circuits), each known construction the engine proves it to
|
||||
//! be found again (and whether by a different route), and the strengths a
|
||||
//! cost did not ask for.
|
||||
//! - [`study`] (with `quantum_receipt`): measuring whether the co-design
|
||||
//! works. A protocol sealed before any data, each participant's condition
|
||||
//! fixed by their key, their record sealed with it, the report's eight
|
||||
//! measures computed by the engine, and a cohort report with bootstrap
|
||||
//! intervals.
|
||||
//! - [`taste`] (with `quantum_receipt`): a person's taste in circuits,
|
||||
//! learned from their own choices between programs the engine proves
|
||||
//! equal, part by part of a rubric; typicality shown beside it and moved
|
||||
|
|
@ -89,6 +94,8 @@ pub mod rewrite;
|
|||
pub mod run;
|
||||
pub mod shortcuts;
|
||||
#[cfg(feature = "quantum_receipt")]
|
||||
pub mod study;
|
||||
#[cfg(feature = "quantum_receipt")]
|
||||
pub mod taste;
|
||||
#[cfg(feature = "quantum_receipt")]
|
||||
pub mod versions;
|
||||
|
|
|
|||
707
wai/wai-quantum/src/engine/study.rs
Normal file
707
wai/wai-quantum/src/engine/study.rs
Normal file
|
|
@ -0,0 +1,707 @@
|
|||
//! Measuring whether the co-design works: a study people take part in, with
|
||||
//! every measure computed by the engine and none by a model.
|
||||
//!
|
||||
//! - **A protocol** ([`Protocol`]), written and sealed before any data: the
|
||||
//! hypotheses, the primary measure, the conditions (each a setting of the
|
||||
//! Studio's [`FLAGS`]) and the analysis (resamples, seed, level).
|
||||
//! - **Assignment** ([`assign`]): a participant's condition is the protocol's
|
||||
//! digest and the participant's key hashed together, so it is fixed before
|
||||
//! they begin and anyone can check it.
|
||||
//! - **A record** ([`Record`]): what one participant made and did, sealed
|
||||
//! with their key: the program's history, their predictions, their choices
|
||||
//! among candidates, the probe tasks they did without the assistant, and
|
||||
//! their answers on ownership.
|
||||
//! - **The measures** ([`measure`]), the report's eight: quality (engine
|
||||
//! measures of the edits the engine proves equal to what they replaced) and
|
||||
//! its trend over time; diversity (the Vendi score of the participant's
|
||||
//! designs); novelty (distance from the known constructions); selection
|
||||
//! skill (how often they took the engine's best of the candidates shown);
|
||||
//! calibration (their predictions against the runs); unassisted skill (the
|
||||
//! probes, scored again by the engine); ownership (their answers, beside
|
||||
//! their share of the gates by the history); and convergence on a model's
|
||||
//! proposals.
|
||||
//! - **The cohort** ([`cohort`]): every record checked against the protocol
|
||||
//! (its digest, its seal, its condition) and refused with the reason if it
|
||||
//! fails; per condition the mean of each measure; each condition's
|
||||
//! difference from the control on the primary measure, with a seeded
|
||||
//! bootstrap interval; Cronbach's alpha of the ownership items; and the
|
||||
//! Vendi score of each condition's final designs.
|
||||
//!
|
||||
//! Nothing here is sent anywhere: a participant gives their record to whoever
|
||||
//! runs the study, and the report is computed where it is run.
|
||||
|
||||
use super::converge::{self, Design};
|
||||
use super::discover::{Evaluator, Spec};
|
||||
use super::versions::{Hand, History, Kind};
|
||||
use super::{critique, equiv, novelty, receipt};
|
||||
use crate::qasm::{self, Inputs};
|
||||
use ed25519_dalek::SigningKey;
|
||||
use serde_json::{Value, json};
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
|
||||
/// The form of a protocol and a record this engine reads and writes.
|
||||
pub const FORM: u32 = 1;
|
||||
|
||||
/// The Studio's features a condition sets, each on or off, with what it is.
|
||||
pub const FLAGS: &[(&str, &str)] = &[
|
||||
("critique", "the engine's critique on the live diagram"),
|
||||
("assistant_asks", "the assistant asks before it proposes, and speaks only when asked"),
|
||||
("alternatives", "Alternatives: improve and explore, laid out on the engine's axes"),
|
||||
("predict_first", "predict before a run"),
|
||||
("workbench", "the Workbench: a family searched, every proposal judged"),
|
||||
("rules", "the person's rules for proposals"),
|
||||
("shortcuts", "the classical-shortcut lens"),
|
||||
("novelty", "what is new: labels, Found"),
|
||||
("taste", "the taste profile: comparisons, ranking"),
|
||||
("rewrites", "the rewrite lens"),
|
||||
("convergence", "your own way: convergence and exploring on your own"),
|
||||
];
|
||||
|
||||
/// The measures a protocol may name as its primary one.
|
||||
pub const MEASURES: &[(&str, &str)] = &[
|
||||
("quality", "the share of the participant's edits, proved equal to what they replaced, that made it cheaper"),
|
||||
("quality_trend", "how that share moved over their history (later half less earlier half)"),
|
||||
("selection", "how often they took the engine's best of the candidates shown"),
|
||||
("calibration_off", "how far their predicted share was from the run's, on average"),
|
||||
("calibration_slope", "the slope of the run's share on their prediction (1 is calibrated)"),
|
||||
("unassisted", "their mean probe score, done without the assistant (1 matches the reference)"),
|
||||
("ownership", "their mean answer on the ownership items, 1 to 7"),
|
||||
("authorship", "their share of the program's gates, by the history"),
|
||||
("novelty", "the mean distance of their designs from the known constructions"),
|
||||
("diversity", "the Vendi score of their distinct designs"),
|
||||
("convergence", "a model's share of their program's gates"),
|
||||
];
|
||||
|
||||
/// The ownership items, each answered from 1 (not at all) to 7 (completely).
|
||||
pub const OWNERSHIP: &[&str] = &[
|
||||
"This program is mine.",
|
||||
"I decided how it is built.",
|
||||
"I could explain every part of it.",
|
||||
"I would put my name to it.",
|
||||
"I made it more than any tool did.",
|
||||
];
|
||||
|
||||
/// The probe tasks, done without the assistant and scored by the engine:
|
||||
/// `(id, what to do, the specification it is judged by)`. A score is the
|
||||
/// reference's cost over the answer's, so 1 matches the reference and more
|
||||
/// beats it; an answer the engine does not verify scores 0.
|
||||
pub fn probes() -> Vec<(&'static str, &'static str, Value)> {
|
||||
vec![
|
||||
(
|
||||
"ghz4",
|
||||
"Prepare (|0000⟩ + |1111⟩)/√2 on four qubits with h and cx, as shallow as you can.",
|
||||
json!({ "name": "probe ghz4", "task": "prepares", "reference": "OPENQASM 3.0;\nqubit[4] q;\nh q[0];\ncx q[0], q[1];\ncx q[1], q[2];\ncx q[2], q[3];\n", "inputs": {}, "gates": ["h", "cx"], "cost": { "depth": 1.0, "two_qubit": 0.1 }, "axes": ["two_qubit", "depth"] }),
|
||||
),
|
||||
(
|
||||
"bell_cz",
|
||||
"Prepare the Bell state (|00⟩ + |11⟩)/√2 using only h and cz.",
|
||||
json!({ "name": "probe bell_cz", "task": "prepares", "reference": "OPENQASM 3.0;\nqubit[2] q;\nh q[0];\nh q[1];\ncz q[0], q[1];\nh q[1];\n", "inputs": {}, "gates": ["h", "cz"], "cost": { "gates": 1.0 }, "axes": ["two_qubit", "depth"] }),
|
||||
),
|
||||
(
|
||||
"swap",
|
||||
"Write a circuit equal to swap q[0], q[1] using only cx.",
|
||||
json!({ "name": "probe swap", "task": "equivalent", "reference": "OPENQASM 3.0;\nqubit[2] q;\ncx q[0], q[1];\ncx q[1], q[0];\ncx q[0], q[1];\n", "inputs": {}, "gates": ["cx"], "cost": { "gates": 1.0 }, "axes": ["two_qubit", "depth"] }),
|
||||
),
|
||||
(
|
||||
"tidy",
|
||||
"Write the same circuit as h q[0]; x q[1]; x q[1]; cx q[0], q[1]; rz(0.25) q[0]; rz(0.5) q[0]; with as few gates as you can.",
|
||||
json!({ "name": "probe tidy", "task": "equivalent", "reference": "OPENQASM 3.0;\nqubit[2] q;\nh q[0];\nx q[1];\nx q[1];\ncx q[0], q[1];\nrz(0.25) q[0];\nrz(0.5) q[0];\n", "inputs": {}, "cost": { "gates": 1.0 }, "axes": ["two_qubit", "depth"] }),
|
||||
),
|
||||
]
|
||||
}
|
||||
|
||||
/// The engine's score of `source` as an answer to the probe `id`.
|
||||
pub fn score_probe(id: &str, source: &str) -> Result<Value, String> {
|
||||
let (_, what, spec) = probes().into_iter().find(|(p, _, _)| *p == id).ok_or_else(|| format!("{id}: no such probe"))?;
|
||||
let ev = Evaluator::new(Spec::from_json(&spec)?)?;
|
||||
let reference = ev.evaluate(spec["reference"].as_str().unwrap_or_default());
|
||||
let answer = ev.evaluate(source);
|
||||
let rc = reference.cost.ok_or("the probe's reference does not verify")?;
|
||||
let score = match (answer.verified, answer.cost) {
|
||||
(true, Some(c)) if c > 0.0 => rc / c,
|
||||
(true, Some(_)) => 1.0,
|
||||
_ => 0.0,
|
||||
};
|
||||
Ok(json!({ "probe": id, "task": what, "verified": answer.verified, "cost": answer.cost, "reference_cost": rc, "score": round(score), "failure": answer.failure }))
|
||||
}
|
||||
|
||||
/// A study, before any data.
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub struct Protocol {
|
||||
pub value: Value,
|
||||
}
|
||||
|
||||
fn round(x: f64) -> f64 {
|
||||
(x * 1000.0).round() / 1000.0
|
||||
}
|
||||
|
||||
impl Protocol {
|
||||
/// A protocol from its JSON, checked: a study name, at least one
|
||||
/// hypothesis, two conditions or more (each with a name and only known
|
||||
/// flags), a control among them, a primary measure the engine knows, and
|
||||
/// an analysis with resamples, a seed and a level.
|
||||
pub fn from_json(v: &Value) -> Result<Protocol, String> {
|
||||
let need = |k: &str| v.get(k).ok_or_else(|| format!("the protocol needs \"{k}\""));
|
||||
need("study")?.as_str().filter(|s| !s.is_empty()).ok_or("the study needs a name")?;
|
||||
if need("hypotheses")?.as_array().is_none_or(|h| h.is_empty() || h.iter().any(|x| !x.is_string())) {
|
||||
return Err("the protocol needs hypotheses: one sentence or more, written before any data".into());
|
||||
}
|
||||
let primary = need("primary")?.as_str().ok_or("primary: a measure's name")?;
|
||||
if !MEASURES.iter().any(|(m, _)| *m == primary) {
|
||||
return Err(format!("{primary}: the primary measure is one of {}", MEASURES.iter().map(|(m, _)| *m).collect::<Vec<_>>().join(", ")));
|
||||
}
|
||||
let conds = need("conditions")?.as_array().ok_or("conditions: a list")?;
|
||||
if conds.len() < 2 {
|
||||
return Err("a study compares two conditions or more".into());
|
||||
}
|
||||
let mut names = BTreeSet::new();
|
||||
for c in conds {
|
||||
let name = c.get("name").and_then(Value::as_str).ok_or("each condition needs a name")?;
|
||||
if !names.insert(name) {
|
||||
return Err(format!("{name}: two conditions have this name"));
|
||||
}
|
||||
for (flag, on) in c.get("flags").and_then(Value::as_object).into_iter().flatten() {
|
||||
if !FLAGS.iter().any(|(f, _)| f == flag) || !on.is_boolean() {
|
||||
return Err(format!("{name}: {flag} is not a flag (flags: {}), or is not true or false", FLAGS.iter().map(|(f, _)| *f).collect::<Vec<_>>().join(", ")));
|
||||
}
|
||||
}
|
||||
}
|
||||
let control = need("control")?.as_str().ok_or("control: a condition's name")?;
|
||||
if !names.contains(control) {
|
||||
return Err(format!("{control}: the control is not one of the conditions"));
|
||||
}
|
||||
let a = need("analysis")?;
|
||||
let resamples = a.get("resamples").and_then(Value::as_u64).ok_or("analysis.resamples: a whole number")?;
|
||||
if !(100..=1_000_000).contains(&resamples) {
|
||||
return Err("analysis.resamples: 100 to 1,000,000".into());
|
||||
}
|
||||
a.get("seed").and_then(Value::as_u64).ok_or("analysis.seed: a whole number")?;
|
||||
let level = a.get("level").and_then(Value::as_f64).ok_or("analysis.level: as 0.95")?;
|
||||
if !(0.5..1.0).contains(&level) {
|
||||
return Err("analysis.level: between 0.5 and 1".into());
|
||||
}
|
||||
Ok(Protocol { value: v.clone() })
|
||||
}
|
||||
|
||||
fn conditions(&self) -> Vec<&Value> {
|
||||
self.value["conditions"].as_array().map(|c| c.iter().collect()).unwrap_or_default()
|
||||
}
|
||||
}
|
||||
|
||||
fn protocol_request() -> Value {
|
||||
json!({ "study_protocol": FORM })
|
||||
}
|
||||
|
||||
/// A protocol sealed with the researcher's key, before any data: `{protocol,
|
||||
/// seal}`. Its digest names the study in every record.
|
||||
pub fn seal_protocol(p: &Protocol, key: &SigningKey) -> Value {
|
||||
json!({ "protocol": p.value, "seal": receipt::seal(key, &protocol_request(), &p.value) })
|
||||
}
|
||||
|
||||
/// A sealed protocol checked: the protocol and its digest.
|
||||
pub fn open_protocol(sealed: &Value) -> Result<(Protocol, String), String> {
|
||||
let p = Protocol::from_json(sealed.get("protocol").ok_or("a sealed protocol is {protocol, seal}")?)?;
|
||||
let seal = sealed.get("seal").ok_or("the protocol is not sealed")?;
|
||||
if !receipt::verify(&protocol_request(), &p.value, seal) {
|
||||
return Err("the protocol's seal does not verify: it changed after it was sealed".into());
|
||||
}
|
||||
Ok((p, seal["digest"].as_str().unwrap_or_default().to_string()))
|
||||
}
|
||||
|
||||
/// The condition `participant` (a did:key) is assigned in the study whose
|
||||
/// sealed protocol has `digest`: fixed by the two, before any data.
|
||||
pub fn assign(p: &Protocol, digest: &str, participant: &str) -> Value {
|
||||
let h = blake3::hash(format!("{digest}|{participant}").as_bytes());
|
||||
let n = p.conditions().len() as u64;
|
||||
let i = (u64::from_le_bytes(h.as_bytes()[..8].try_into().expect("8 bytes")) % n.max(1)) as usize;
|
||||
let c = p.conditions()[i];
|
||||
json!({ "condition": c["name"], "flags": c.get("flags").cloned().unwrap_or(json!({})) })
|
||||
}
|
||||
|
||||
fn record_request(digest: &str, participant: &str) -> Value {
|
||||
json!({ "study_record": digest, "participant": participant })
|
||||
}
|
||||
|
||||
/// A participant's record sealed with their key: `{record, seal}`. The
|
||||
/// record names the protocol's digest, the participant (the key's did) and
|
||||
/// their condition, and holds `history`, `predictions`, `choices`, `probes`
|
||||
/// and `ownership` as the Studio keeps them.
|
||||
pub fn seal_record(record: &Value, key: &SigningKey) -> Result<Value, String> {
|
||||
let did = crate::quantum_receipt::did_key(&key.verifying_key());
|
||||
let mut r = record.clone();
|
||||
r["record"] = json!(FORM);
|
||||
r["participant"] = json!(did);
|
||||
let digest = r.get("protocol").and_then(Value::as_str).ok_or("the record names no protocol")?.to_string();
|
||||
Ok(json!({ "record": r, "seal": receipt::seal(key, &record_request(&digest, &did), &r) }))
|
||||
}
|
||||
|
||||
/// A sealed record checked against a protocol: the seal holds, the key is the
|
||||
/// participant's, the protocol is this one, and the condition is the one
|
||||
/// assigned. The record, or why it is refused.
|
||||
pub fn open_record(sealed: &Value, p: &Protocol, digest: &str) -> Result<Value, String> {
|
||||
let r = sealed.get("record").ok_or("a sealed record is {record, seal}")?;
|
||||
let seal = sealed.get("seal").ok_or("the record is not sealed")?;
|
||||
let did = r.get("participant").and_then(Value::as_str).ok_or("the record names no participant")?;
|
||||
if r.get("protocol").and_then(Value::as_str) != Some(digest) {
|
||||
return Err("it was made for another study".into());
|
||||
}
|
||||
if !receipt::verify(&record_request(digest, did), r, seal) {
|
||||
return Err("its seal does not verify: it changed after it was sealed".into());
|
||||
}
|
||||
if seal.get("key").and_then(Value::as_str).and_then(super::job::did_of).as_deref() != Some(did) {
|
||||
return Err("it is sealed by a key other than its participant's".into());
|
||||
}
|
||||
let assigned = assign(p, digest, did);
|
||||
if r.get("condition") != assigned.get("condition") {
|
||||
return Err(format!("its condition is {}, and the participant is assigned {}", r["condition"], assigned["condition"]));
|
||||
}
|
||||
Ok(r.clone())
|
||||
}
|
||||
|
||||
fn slope(ys: &[f64]) -> Option<f64> {
|
||||
let n = ys.len();
|
||||
if n < 3 {
|
||||
return None;
|
||||
}
|
||||
let mx = (n - 1) as f64 / 2.0;
|
||||
let my = ys.iter().sum::<f64>() / n as f64;
|
||||
let num: f64 = ys.iter().enumerate().map(|(i, y)| (i as f64 - mx) * (y - my)).sum();
|
||||
let den: f64 = (0..n).map(|i| (i as f64 - mx).powi(2)).sum();
|
||||
Some(num / den)
|
||||
}
|
||||
|
||||
fn mean(v: &[f64]) -> Option<f64> {
|
||||
(!v.is_empty()).then(|| v.iter().sum::<f64>() / v.len() as f64)
|
||||
}
|
||||
|
||||
/// The engine's measures of one participant, from their record: each one
|
||||
/// computed again here from what they made and did.
|
||||
pub fn measure(record: &Value) -> Result<Value, String> {
|
||||
let inputs: Inputs = match record.get("inputs") {
|
||||
None | Some(Value::Null) => Inputs::new(),
|
||||
Some(v) => serde_json::from_value(v.clone()).map_err(|e| format!("inputs: {e}"))?,
|
||||
};
|
||||
let history = match record.get("history") {
|
||||
None | Some(Value::Null) => None,
|
||||
Some(v) => Some(History::from_json(v)?),
|
||||
};
|
||||
let mut out = json!({ "participant": record.get("participant"), "condition": record.get("condition") });
|
||||
|
||||
// Quality: the person's edits the engine proves equal to their parent.
|
||||
if let Some(h) = &history {
|
||||
let at: BTreeMap<&str, usize> = h.versions.iter().enumerate().map(|(i, v)| (v.id.as_str(), i)).collect();
|
||||
let mut scores = Vec::new();
|
||||
let mut deltas: BTreeMap<&str, Vec<f64>> = BTreeMap::new();
|
||||
for v in h.versions.iter().filter(|v| v.by.kind == Kind::Person) {
|
||||
let Some(parent) = v.parents.first().and_then(|p| at.get(p.as_str())).map(|&i| &h.versions[i]) else { continue };
|
||||
let (Ok(a), Ok(b)) = (qasm::parse_with(&parent.source, &inputs), qasm::parse_with(&v.source, &inputs)) else { continue };
|
||||
let e = equiv::equivalent_programs(&a, &b);
|
||||
if e["verdict"] != "equivalent" || e["how"] == "random" {
|
||||
continue;
|
||||
}
|
||||
let (ma, mb) = (critique::measures(&a), critique::measures(&b));
|
||||
let keys = ["two_qubit", "depth", "t_count", "gates"];
|
||||
let d: Vec<f64> = keys.iter().map(|k| ma[*k].as_f64().unwrap_or(0.0) - mb[*k].as_f64().unwrap_or(0.0)).collect();
|
||||
for (k, x) in keys.iter().zip(&d) {
|
||||
deltas.entry(*k).or_default().push(*x);
|
||||
}
|
||||
// Cheaper in one measure and dearer in none is better; the reverse, worse.
|
||||
let s = if d.iter().all(|x| *x >= 0.0) && d.iter().any(|x| *x > 0.0) { 1.0 } else if d.iter().all(|x| *x <= 0.0) && d.iter().any(|x| *x < 0.0) { -1.0 } else { 0.0 };
|
||||
scores.push(s);
|
||||
}
|
||||
let improved = scores.iter().filter(|s| **s > 0.0).count();
|
||||
let half = scores.len() / 2;
|
||||
let trend = (scores.len() >= 4).then(|| mean(&scores[half..]).unwrap_or(0.0) - mean(&scores[..half]).unwrap_or(0.0));
|
||||
out["quality"] = json!({
|
||||
"edits": scores.len(),
|
||||
"improved": improved,
|
||||
"worsened": scores.iter().filter(|s| **s < 0.0).count(),
|
||||
"share": (!scores.is_empty()).then(|| round(improved as f64 / scores.len() as f64)),
|
||||
"trend": trend.map(round),
|
||||
"mean_saving": deltas.iter().map(|(k, v)| (k.to_string(), json!(mean(v).map(round)))).collect::<serde_json::Map<_, _>>(),
|
||||
});
|
||||
// Diversity and novelty of their distinct designs.
|
||||
let mut seen = BTreeSet::new();
|
||||
let designs: Vec<Design> = h.versions.iter().filter(|v| v.by.kind == Kind::Person && seen.insert(v.source.trim().to_string())).map(|v| Design { name: v.id.clone(), source: v.source.clone(), at: Some(v.at) }).collect();
|
||||
let readable: Vec<Design> = designs.into_iter().filter(|d| qasm::parse_with(&d.source, &inputs).is_ok()).collect();
|
||||
out["diversity"] = if readable.len() >= 2 { converge::cohort(&readable, &inputs, 0).map(|c| json!({ "designs": readable.len(), "vendi": c["vendi"] }))? } else { json!({ "designs": readable.len(), "vendi": null }) };
|
||||
let nov: Vec<f64> = readable.iter().filter_map(|d| novelty::assess(&d.source, &inputs, &[], None, &[])["novelty"].as_f64()).collect();
|
||||
out["novelty"] = json!({ "designs": nov.len(), "mean": mean(&nov).map(round) });
|
||||
// Authorship, and convergence on a model's proposals.
|
||||
if let Ok(head) = h.head()
|
||||
&& let Ok(a) = h.authorship(&head.id, &inputs)
|
||||
&& a.of.gates > 0
|
||||
{
|
||||
let g = |k: Hand| a.gates.get(&k).copied().unwrap_or(0) as f64;
|
||||
let person = g(Hand::Person) + 0.5 * (g(Hand::AssistantThenPerson) + g(Hand::AgentThenPerson));
|
||||
out["authorship"] = json!(round(person / a.of.gates as f64));
|
||||
}
|
||||
if let Ok(c) = converge::history(h, &inputs) {
|
||||
out["convergence"] = json!({ "share": c["share"], "converging": c["converging"], "organic": c["organic"], "assisted": c["assisted"] });
|
||||
}
|
||||
}
|
||||
|
||||
// Selection skill: the engine's best of the candidates, taken or not.
|
||||
let choices: Vec<&Value> = record.get("choices").and_then(Value::as_array).map(|c| c.iter().collect()).unwrap_or_default();
|
||||
let shown: Vec<(u64, u64, u64)> = choices.iter().filter_map(|c| Some((c.get("k")?.as_u64()?, c.get("picked")?.as_u64()?, c.get("best")?.as_u64()?))).filter(|(k, p, b)| *k >= 2 && p < k && b < k).collect();
|
||||
out["selection"] = json!({
|
||||
"choices": shown.len(),
|
||||
"best_taken": shown.iter().filter(|(_, p, b)| p == b).count(),
|
||||
"rate": (!shown.is_empty()).then(|| round(shown.iter().filter(|(_, p, b)| p == b).count() as f64 / shown.len() as f64)),
|
||||
"chance": mean(&shown.iter().map(|(k, _, _)| 1.0 / *k as f64).collect::<Vec<_>>()).map(round),
|
||||
});
|
||||
|
||||
// Calibration: the share they predicted against the run's.
|
||||
let preds: Vec<(f64, f64, bool)> = record.get("predictions").and_then(Value::as_array).into_iter().flatten().filter_map(|p| Some((p.get("share")?.as_f64()?, p.get("got")?.as_f64()?, p.get("right").and_then(Value::as_bool).unwrap_or(false)))).filter(|(s, g, _)| (0.0..=1.0).contains(s) && (0.0..=1.0).contains(g)).collect();
|
||||
let (xs, ys): (Vec<f64>, Vec<f64>) = preds.iter().map(|(s, g, _)| (*s, *g)).unzip();
|
||||
let fit = (xs.len() >= 3).then(|| {
|
||||
let (mx, my) = (mean(&xs).unwrap_or(0.0), mean(&ys).unwrap_or(0.0));
|
||||
let sxx: f64 = xs.iter().map(|x| (x - mx).powi(2)).sum();
|
||||
let sxy: f64 = xs.iter().zip(&ys).map(|(x, y)| (x - mx) * (y - my)).sum();
|
||||
(sxx > 1e-12).then(|| (sxy / sxx, my - sxy / sxx * mx))
|
||||
}).flatten();
|
||||
out["calibration"] = json!({
|
||||
"predictions": preds.len(),
|
||||
"top_right": preds.iter().filter(|p| p.2).count(),
|
||||
"mean_off": mean(&preds.iter().map(|(s, g, _)| (s - g).abs()).collect::<Vec<_>>()).map(round),
|
||||
"slope": fit.map(|f| round(f.0)),
|
||||
"intercept": fit.map(|f| round(f.1)),
|
||||
});
|
||||
|
||||
// Unassisted skill: every probe scored again by the engine.
|
||||
let mut probe_scores = Vec::new();
|
||||
let mut by_probe: BTreeMap<String, Vec<f64>> = BTreeMap::new();
|
||||
for p in record.get("probes").and_then(Value::as_array).into_iter().flatten() {
|
||||
let (Some(id), Some(src)) = (p.get("probe").and_then(Value::as_str), p.get("source").and_then(Value::as_str)) else { continue };
|
||||
if let Ok(s) = score_probe(id, src) {
|
||||
let x = s["score"].as_f64().unwrap_or(0.0);
|
||||
probe_scores.push(x);
|
||||
by_probe.entry(id.to_string()).or_default().push(x);
|
||||
}
|
||||
}
|
||||
out["unassisted"] = json!({
|
||||
"probes": probe_scores.len(),
|
||||
"mean": mean(&probe_scores).map(round),
|
||||
"trend": slope(&probe_scores).map(round),
|
||||
"by_probe": by_probe.iter().map(|(k, v)| (k.clone(), json!(mean(v).map(round)))).collect::<serde_json::Map<_, _>>(),
|
||||
});
|
||||
|
||||
// Ownership: the answers, each 1 to 7.
|
||||
let answers: Vec<Vec<f64>> = record.get("ownership").and_then(Value::as_array).into_iter().flatten().filter_map(|o| {
|
||||
let items: Vec<f64> = o.get("items")?.as_array()?.iter().filter_map(Value::as_f64).collect();
|
||||
(items.len() == OWNERSHIP.len() && items.iter().all(|x| (1.0..=7.0).contains(x))).then_some(items)
|
||||
}).collect();
|
||||
out["ownership"] = json!({
|
||||
"answers": answers.len(),
|
||||
"mean": mean(&answers.iter().filter_map(|a| mean(a)).collect::<Vec<_>>()).map(round),
|
||||
"latest": answers.last().and_then(|a| mean(a)).map(round),
|
||||
});
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// The measures as one number each, by the names in [`MEASURES`], where the
|
||||
/// record gives one.
|
||||
pub fn scalars(m: &Value) -> BTreeMap<&'static str, f64> {
|
||||
let pick = [
|
||||
("quality", m.pointer("/quality/share")),
|
||||
("quality_trend", m.pointer("/quality/trend")),
|
||||
("selection", m.pointer("/selection/rate")),
|
||||
("calibration_off", m.pointer("/calibration/mean_off")),
|
||||
("calibration_slope", m.pointer("/calibration/slope")),
|
||||
("unassisted", m.pointer("/unassisted/mean")),
|
||||
("ownership", m.pointer("/ownership/mean")),
|
||||
("authorship", m.get("authorship")),
|
||||
("novelty", m.pointer("/novelty/mean")),
|
||||
("diversity", m.pointer("/diversity/vendi")),
|
||||
("convergence", m.pointer("/convergence/share")),
|
||||
];
|
||||
pick.into_iter().filter_map(|(k, v)| Some((k, v?.as_f64()?))).collect()
|
||||
}
|
||||
|
||||
/// splitmix64: the bootstrap's resampling, the same from a seed everywhere.
|
||||
struct Rng(u64);
|
||||
|
||||
impl Rng {
|
||||
fn next(&mut self) -> u64 {
|
||||
self.0 = self.0.wrapping_add(0x9e37_79b9_7f4a_7c15);
|
||||
let mut z = self.0;
|
||||
z = (z ^ (z >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9);
|
||||
z = (z ^ (z >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb);
|
||||
z ^ (z >> 31)
|
||||
}
|
||||
fn below(&mut self, n: usize) -> usize {
|
||||
(self.next() % n as u64) as usize
|
||||
}
|
||||
}
|
||||
|
||||
/// The difference of means `b − a` and its bootstrap interval at `level`,
|
||||
/// resampling each group's participants `resamples` times from `seed`.
|
||||
fn bootstrap(a: &[f64], b: &[f64], resamples: u64, seed: u64, level: f64) -> Value {
|
||||
if a.is_empty() || b.is_empty() {
|
||||
return json!({ "difference": null, "low": null, "high": null, "why": "a group has no participant with this measure" });
|
||||
}
|
||||
let diff = mean(b).unwrap_or(0.0) - mean(a).unwrap_or(0.0);
|
||||
let mut rng = Rng(seed);
|
||||
let mut ds: Vec<f64> = (0..resamples)
|
||||
.map(|_| {
|
||||
let ma = (0..a.len()).map(|_| a[rng.below(a.len())]).sum::<f64>() / a.len() as f64;
|
||||
let mb = (0..b.len()).map(|_| b[rng.below(b.len())]).sum::<f64>() / b.len() as f64;
|
||||
mb - ma
|
||||
})
|
||||
.collect();
|
||||
ds.sort_by(f64::total_cmp);
|
||||
let q = |p: f64| ds[((p * (ds.len() - 1) as f64).round() as usize).min(ds.len() - 1)];
|
||||
let tail = (1.0 - level) / 2.0;
|
||||
json!({ "difference": round(diff), "low": round(q(tail)), "high": round(q(1.0 - tail)), "excludes_zero": q(tail) > 0.0 || q(1.0 - tail) < 0.0 })
|
||||
}
|
||||
|
||||
/// Cronbach's alpha of `responses`, each the same number of items.
|
||||
fn cronbach(responses: &[Vec<f64>]) -> Option<f64> {
|
||||
let k = responses.first()?.len();
|
||||
if k < 2 || responses.len() < 2 {
|
||||
return None;
|
||||
}
|
||||
let var = |xs: &[f64]| {
|
||||
let m = xs.iter().sum::<f64>() / xs.len() as f64;
|
||||
xs.iter().map(|x| (x - m).powi(2)).sum::<f64>() / (xs.len() - 1) as f64
|
||||
};
|
||||
let items: f64 = (0..k).map(|j| var(&responses.iter().map(|r| r[j]).collect::<Vec<_>>())).sum();
|
||||
let total = var(&responses.iter().map(|r| r.iter().sum()).collect::<Vec<_>>());
|
||||
(total > 1e-12).then(|| k as f64 / (k - 1) as f64 * (1.0 - items / total))
|
||||
}
|
||||
|
||||
/// The study's report: each record checked against the sealed protocol,
|
||||
/// those refused with the reason, the measures per condition, each
|
||||
/// condition's difference from the control on the primary measure with its
|
||||
/// bootstrap interval, Cronbach's alpha of the ownership items, and the
|
||||
/// Vendi score of each condition's final designs.
|
||||
pub fn cohort(sealed_protocol: &Value, records: &[Value]) -> Result<Value, String> {
|
||||
let (p, digest) = open_protocol(sealed_protocol)?;
|
||||
let mut refused = Vec::new();
|
||||
let mut by_cond: BTreeMap<String, Vec<(Value, Value)>> = BTreeMap::new();
|
||||
let mut seen = BTreeSet::new();
|
||||
for (i, s) in records.iter().enumerate() {
|
||||
match open_record(s, &p, &digest).and_then(|r| measure(&r).map(|m| (r, m))) {
|
||||
Ok((r, m)) => {
|
||||
let who = r["participant"].as_str().unwrap_or_default().to_string();
|
||||
if !seen.insert(who.clone()) {
|
||||
refused.push(json!({ "record": i, "why": format!("a second record from {who}: one record per participant") }));
|
||||
continue;
|
||||
}
|
||||
by_cond.entry(r["condition"].as_str().unwrap_or_default().to_string()).or_default().push((r, m));
|
||||
}
|
||||
Err(e) => refused.push(json!({ "record": i, "why": e })),
|
||||
}
|
||||
}
|
||||
let primary = p.value["primary"].as_str().unwrap_or_default();
|
||||
let control = p.value["control"].as_str().unwrap_or_default();
|
||||
let a = &p.value["analysis"];
|
||||
let (resamples, seed, level) = (a["resamples"].as_u64().unwrap_or(10_000), a["seed"].as_u64().unwrap_or(0), a["level"].as_f64().unwrap_or(0.95));
|
||||
let values = |cond: &str, k: &str| -> Vec<f64> { by_cond.get(cond).into_iter().flatten().filter_map(|(_, m)| scalars(m).get(k).copied()).collect() };
|
||||
let mut conditions = Vec::new();
|
||||
for c in p.conditions() {
|
||||
let name = c["name"].as_str().unwrap_or_default();
|
||||
let ms: BTreeMap<&str, Value> = MEASURES.iter().map(|(k, _)| (*k, json!(mean(&values(name, k)).map(round)))).collect();
|
||||
let finals: Vec<Design> = by_cond.get(name).into_iter().flatten().filter_map(|(r, _)| {
|
||||
let h = History::from_json(r.get("history")?).ok()?;
|
||||
let head = h.head().ok()?;
|
||||
Some(Design { name: r["participant"].as_str()?.to_string(), source: head.source.clone(), at: Some(head.at) })
|
||||
}).collect();
|
||||
let vendi = if finals.len() >= 2 { converge::cohort(&finals, &Inputs::new(), 0).ok().map(|v| v["vendi"].clone()) } else { None };
|
||||
let mut entry = json!({ "condition": name, "participants": by_cond.get(name).map_or(0, Vec::len), "means": ms, "final_designs_vendi": vendi });
|
||||
if name != control {
|
||||
entry["primary"] = bootstrap(&values(control, primary), &values(name, primary), resamples, seed, level);
|
||||
}
|
||||
conditions.push(entry);
|
||||
}
|
||||
let answers: Vec<Vec<f64>> = by_cond.values().flatten().flat_map(|(r, _)| {
|
||||
r.get("ownership").and_then(Value::as_array).cloned().unwrap_or_default().into_iter().filter_map(|o| {
|
||||
let items: Vec<f64> = o.get("items")?.as_array()?.iter().filter_map(Value::as_f64).collect();
|
||||
(items.len() == OWNERSHIP.len()).then_some(items)
|
||||
})
|
||||
}).collect();
|
||||
Ok(json!({
|
||||
"study": p.value["study"],
|
||||
"protocol": digest,
|
||||
"hypotheses": p.value["hypotheses"],
|
||||
"primary": primary,
|
||||
"control": control,
|
||||
"analysis": { "resamples": resamples, "seed": seed, "level": level, "method": "the difference of means, each group's participants resampled with replacement" },
|
||||
"records": records.len(),
|
||||
"included": by_cond.values().map(Vec::len).sum::<usize>(),
|
||||
"refused": refused,
|
||||
"conditions": conditions,
|
||||
"ownership_alpha": cronbach(&answers).map(round),
|
||||
"ownership_answers": answers.len(),
|
||||
}))
|
||||
}
|
||||
|
||||
/// The study as a page, an app or an agent calls it, with JSON arguments:
|
||||
/// `flags` and `measures` (what a protocol may name), `probes` (the probe
|
||||
/// tasks), `score_probe` (`probe`, `qasm`), `check_protocol` (`protocol`),
|
||||
/// `open_protocol` (`sealed`), `assign` (`sealed`, `participant`), `measure`
|
||||
/// (`record`, unsealed) and `cohort` (`sealed`, `records`). Sealing takes a
|
||||
/// key, which the host holds: [`seal_protocol`] and [`seal_record`].
|
||||
pub fn call(op: &str, args: &Value) -> Result<Value, String> {
|
||||
let s = |k: &str| args.get(k).ok_or_else(|| format!("{op}: give \"{k}\""));
|
||||
match op {
|
||||
"flags" => Ok(json!(FLAGS.iter().map(|(k, w)| json!({ "flag": k, "what": w })).collect::<Vec<_>>())),
|
||||
"measures" => Ok(json!(MEASURES.iter().map(|(k, w)| json!({ "measure": k, "what": w })).collect::<Vec<_>>())),
|
||||
"probes" => Ok(json!(probes().into_iter().map(|(id, what, _)| json!({ "probe": id, "task": what })).collect::<Vec<_>>())),
|
||||
"ownership_items" => Ok(json!(OWNERSHIP)),
|
||||
"score_probe" => score_probe(s("probe")?.as_str().ok_or("probe: an id")?, s("qasm")?.as_str().ok_or("qasm: the answer")?),
|
||||
"check_protocol" => Protocol::from_json(s("protocol")?).map(|_| json!({ "ok": true })),
|
||||
"open_protocol" => open_protocol(s("sealed")?).map(|(p, d)| json!({ "protocol": p.value, "digest": d })),
|
||||
"assign" => {
|
||||
let (p, d) = open_protocol(s("sealed")?)?;
|
||||
Ok(assign(&p, &d, s("participant")?.as_str().ok_or("participant: a did:key")?))
|
||||
}
|
||||
"measure" => measure(s("record")?),
|
||||
"cohort" => cohort(s("sealed")?, s("records")?.as_array().ok_or("records: a list of sealed records")?),
|
||||
_ => Err(format!("{op}: no such call on the study")),
|
||||
}
|
||||
}
|
||||
|
||||
/// Seal a protocol or a record with `key`, for hosts that hold one:
|
||||
/// `seal_protocol` (`protocol`) or `seal_record` (`record`).
|
||||
pub fn call_sealing(op: &str, args: &Value, key: &SigningKey) -> Result<Value, String> {
|
||||
match op {
|
||||
"seal_protocol" => Ok(seal_protocol(&Protocol::from_json(args.get("protocol").ok_or("give \"protocol\"")?)?, key)),
|
||||
"seal_record" => seal_record(args.get("record").ok_or("give \"record\"")?, key),
|
||||
_ => call(op, args),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::engine::versions::By;
|
||||
|
||||
fn key(b: u8) -> SigningKey {
|
||||
SigningKey::from_bytes(&[b; 32])
|
||||
}
|
||||
|
||||
fn protocol() -> Value {
|
||||
json!({
|
||||
"study": "the assistant asks first",
|
||||
"hypotheses": ["Asking first raises unassisted skill against proposing first."],
|
||||
"primary": "unassisted",
|
||||
"conditions": [
|
||||
{ "name": "proposes", "flags": { "assistant_asks": false } },
|
||||
{ "name": "asks", "flags": { "assistant_asks": true } },
|
||||
],
|
||||
"control": "proposes",
|
||||
"analysis": { "resamples": 2000, "seed": 7, "level": 0.95 },
|
||||
})
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_protocol_is_checked_sealed_and_assigns_by_key() {
|
||||
assert!(Protocol::from_json(&json!({ "study": "x" })).unwrap_err().contains("hypotheses"));
|
||||
let mut bad = protocol();
|
||||
bad["primary"] = json!("beauty");
|
||||
assert!(Protocol::from_json(&bad).unwrap_err().contains("primary measure is one of"));
|
||||
bad = protocol();
|
||||
bad["conditions"][1]["flags"] = json!({ "telepathy": true });
|
||||
assert!(Protocol::from_json(&bad).unwrap_err().contains("not a flag"));
|
||||
let p = Protocol::from_json(&protocol()).unwrap();
|
||||
let sealed = seal_protocol(&p, &key(1));
|
||||
let (back, digest) = open_protocol(&sealed).unwrap();
|
||||
assert_eq!(back, p);
|
||||
let mut tampered = sealed.clone();
|
||||
tampered["protocol"]["primary"] = json!("quality");
|
||||
assert!(open_protocol(&tampered).unwrap_err().contains("changed after it was sealed"));
|
||||
// Assignment is fixed by the protocol and the key, and spreads participants.
|
||||
let conds: BTreeSet<String> = (0..20u8).map(|b| assign(&p, &digest, &crate::quantum_receipt::did_key(&key(b + 10).verifying_key()))["condition"].as_str().unwrap().to_string()).collect();
|
||||
assert_eq!(conds.len(), 2, "{conds:?}");
|
||||
let did = crate::quantum_receipt::did_key(&key(3).verifying_key());
|
||||
assert_eq!(assign(&p, &digest, &did), assign(&p, &digest, &did));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn probes_are_scored_by_the_engine() {
|
||||
let tree = "OPENQASM 3.0;\nqubit[4] q;\nh q[0];\ncx q[0], q[1];\ncx q[0], q[2];\ncx q[1], q[3];\n";
|
||||
let s = score_probe("ghz4", tree).unwrap();
|
||||
assert_eq!(s["verified"], true, "{s}");
|
||||
assert!(s["score"].as_f64().unwrap() > 1.0, "a shallower tree beats the chain: {s}");
|
||||
assert_eq!(score_probe("ghz4", "OPENQASM 3.0;\nqubit[4] q;\nh q[0];\n").unwrap()["score"], 0.0);
|
||||
assert_eq!(score_probe("swap", "OPENQASM 3.0;\nqubit[2] q;\ncx q[0], q[1];\ncx q[1], q[0];\ncx q[0], q[1];\n").unwrap()["score"], 1.0);
|
||||
let tidy = score_probe("tidy", "OPENQASM 3.0;\nqubit[2] q;\nh q[0];\ncx q[0], q[1];\nrz(0.75) q[0];\n").unwrap();
|
||||
assert_eq!(tidy["score"], 2.0, "6 gates to 3: {tidy}");
|
||||
assert!(score_probe("nonsense", "").unwrap_err().contains("no such probe"));
|
||||
}
|
||||
|
||||
fn record_for(k: &SigningKey, digest: &str, p: &Protocol, unassisted: &str) -> Value {
|
||||
let did = crate::quantum_receipt::did_key(&k.verifying_key());
|
||||
let cond = assign(p, digest, &did)["condition"].clone();
|
||||
let bell = "OPENQASM 3.0;\nqubit[2] q;\nh q[0];\nx q[1];\nx q[1];\ncx q[0], q[1];\n";
|
||||
let mut h = History::start(bell, By::new(Kind::Person, None), None, 1, None).unwrap();
|
||||
h.commit("v0", "OPENQASM 3.0;\nqubit[2] q;\nh q[0];\ncx q[0], q[1];\n", By::new(Kind::Person, None), None, 2, None).unwrap();
|
||||
let r = json!({
|
||||
"protocol": digest,
|
||||
"condition": cond,
|
||||
"history": h,
|
||||
"predictions": [{ "share": 0.5, "got": 0.49, "right": true }, { "share": 0.7, "got": 0.52, "right": true }, { "share": 0.2, "got": 0.1, "right": false }],
|
||||
"choices": [{ "k": 3, "picked": 0, "best": 0 }, { "k": 2, "picked": 1, "best": 0 }],
|
||||
"probes": [{ "probe": "ghz4", "source": unassisted }],
|
||||
"ownership": [{ "items": [6, 6, 5, 7, 6] }, { "items": [5, 6, 5, 6, 5] }],
|
||||
});
|
||||
seal_record(&r, k).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_record_is_measured_by_the_engine() {
|
||||
let p = Protocol::from_json(&protocol()).unwrap();
|
||||
let (_, digest) = open_protocol(&seal_protocol(&p, &key(1))).unwrap();
|
||||
let chain = "OPENQASM 3.0;\nqubit[4] q;\nh q[0];\ncx q[0], q[1];\ncx q[1], q[2];\ncx q[2], q[3];\n";
|
||||
let sealed = record_for(&key(20), &digest, &p, chain);
|
||||
let r = open_record(&sealed, &p, &digest).unwrap();
|
||||
let m = measure(&r).unwrap();
|
||||
assert_eq!((m["quality"]["edits"].as_u64(), m["quality"]["improved"].as_u64()), (Some(1), Some(1)), "removing x x is an improvement the engine proves: {m}");
|
||||
assert_eq!((m["selection"]["choices"].as_u64(), m["selection"]["rate"].as_f64()), (Some(2), Some(0.5)), "{m}");
|
||||
assert_eq!(m["calibration"]["predictions"], 3);
|
||||
assert!(m["calibration"]["slope"].is_number(), "{m}");
|
||||
assert_eq!(m["unassisted"]["mean"], 1.0, "the chain is the reference: {m}");
|
||||
assert_eq!(m["ownership"]["answers"], 2);
|
||||
assert_eq!(m["authorship"], 1.0, "every gate is the person's: {m}");
|
||||
let sc = scalars(&m);
|
||||
assert!(sc.contains_key("quality") && sc.contains_key("unassisted") && sc.contains_key("ownership"), "{sc:?}");
|
||||
// A changed record, another study, another key: refused.
|
||||
let mut changed = sealed.clone();
|
||||
changed["record"]["probes"][0]["source"] = json!("OPENQASM 3.0;\nqubit[4] q;\n");
|
||||
assert!(open_record(&changed, &p, &digest).unwrap_err().contains("changed after it was sealed"));
|
||||
assert!(open_record(&sealed, &p, "00").unwrap_err().contains("another study"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_cohort_is_checked_compared_and_alpha_computed() {
|
||||
let p = Protocol::from_json(&protocol()).unwrap();
|
||||
let sp = seal_protocol(&p, &key(1));
|
||||
let (_, digest) = open_protocol(&sp).unwrap();
|
||||
let chain = "OPENQASM 3.0;\nqubit[4] q;\nh q[0];\ncx q[0], q[1];\ncx q[1], q[2];\ncx q[2], q[3];\n";
|
||||
let tree = "OPENQASM 3.0;\nqubit[4] q;\nh q[0];\ncx q[0], q[1];\ncx q[0], q[2];\ncx q[1], q[3];\n";
|
||||
let mut records = Vec::new();
|
||||
for b in 30..46u8 {
|
||||
let k = key(b);
|
||||
let did = crate::quantum_receipt::did_key(&k.verifying_key());
|
||||
// Those who are asked first answer the probe with the tree.
|
||||
let asks = assign(&p, &digest, &did)["condition"] == "asks";
|
||||
records.push(record_for(&k, &digest, &p, if asks { tree } else { chain }));
|
||||
}
|
||||
records.push(records[0].clone());
|
||||
let mut wrong = records[1].clone();
|
||||
let flip = if wrong["record"]["condition"] == "asks" { "proposes" } else { "asks" };
|
||||
wrong["record"]["condition"] = json!(flip);
|
||||
records.push(wrong);
|
||||
let r = cohort(&sp, &records).unwrap();
|
||||
assert_eq!((r["records"].as_u64(), r["included"].as_u64()), (Some(18), Some(16)), "{r}");
|
||||
let why: Vec<&str> = r["refused"].as_array().unwrap().iter().map(|x| x["why"].as_str().unwrap()).collect();
|
||||
assert!(why.iter().any(|w| w.contains("second record")) && why.iter().any(|w| w.contains("changed after it was sealed") || w.contains("is assigned")), "{why:?}");
|
||||
let asks = r["conditions"].as_array().unwrap().iter().find(|c| c["condition"] == "asks").unwrap();
|
||||
let d = &asks["primary"];
|
||||
assert!(d["difference"].as_f64().unwrap() > 0.0 && d["excludes_zero"] == true, "the tree's score beats the chain's: {r}");
|
||||
assert!(r["ownership_alpha"].is_number(), "{r}");
|
||||
// The same inputs give the same report.
|
||||
assert_eq!(cohort(&sp, &records).unwrap(), r);
|
||||
}
|
||||
}
|
||||
|
|
@ -8,7 +8,7 @@
|
|||
use ed25519_dalek::SigningKey;
|
||||
use serde_json::{Value, json};
|
||||
use wai::engine::discover::{self, Archive, Budget, Evaluator, SearchOptions, Spec, search};
|
||||
use wai::engine::{catalog, converge, critique, draw, equiv, job, locks, novelty, rewrite, shortcuts, taste, versions};
|
||||
use wai::engine::{catalog, converge, critique, draw, equiv, job, locks, novelty, rewrite, shortcuts, study, taste, versions};
|
||||
use wasm_bindgen::prelude::*;
|
||||
|
||||
/// What a result run in a tab says of its energy.
|
||||
|
|
@ -197,6 +197,21 @@ impl QuantumEngine {
|
|||
}
|
||||
}
|
||||
|
||||
/// A study in a page ([`wai::engine::study`]): the participant's record
|
||||
/// sealed with the page's key, and the protocol a researcher seals here too.
|
||||
#[wasm_bindgen]
|
||||
impl QuantumEngine {
|
||||
/// `op` on the study, sealing with the page's key where it seals
|
||||
/// (`seal_record`, `seal_protocol`): the answer, or `{error}`.
|
||||
pub fn study(&self, op: &str, args: &str) -> String {
|
||||
serde_json::from_str::<Value>(args)
|
||||
.map_err(|e| format!("{op}: the arguments are not JSON: {e}"))
|
||||
.and_then(|a| study::call_sealing(op, &a, &self.key))
|
||||
.map(|v| v.to_string())
|
||||
.unwrap_or_else(error_json)
|
||||
}
|
||||
}
|
||||
|
||||
/// A program's history in a page ([`wai::engine::versions`]), kept by the
|
||||
/// page as JSON: who made each version and how, whose hand each line and gate
|
||||
/// is in, and each version recorded sealed with the page's key.
|
||||
|
|
@ -416,6 +431,24 @@ mod tests {
|
|||
assert!(engine_discover_novelty(&spec, &sub["archive"].to_string(), "e99").contains("no such entry"));
|
||||
}
|
||||
|
||||
/// A study in the page: a protocol sealed, the page's condition, a probe
|
||||
/// scored and a record sealed with the page's key and measured.
|
||||
#[test]
|
||||
fn a_study_runs_in_the_page() {
|
||||
let e = QuantumEngine::new(&[9; 32], 64).unwrap();
|
||||
let protocol = json!({ "study": "s", "hypotheses": ["h"], "primary": "unassisted", "conditions": [{ "name": "a" }, { "name": "b", "flags": { "taste": false } }], "control": "a", "analysis": { "resamples": 200, "seed": 1, "level": 0.95 } });
|
||||
let sealed: Value = serde_json::from_str(&e.study("seal_protocol", &json!({ "protocol": protocol }).to_string())).unwrap();
|
||||
let open: Value = serde_json::from_str(&e.study("open_protocol", &json!({ "sealed": sealed }).to_string())).unwrap();
|
||||
let digest = open["digest"].as_str().unwrap().to_string();
|
||||
let me: Value = serde_json::from_str(&e.study("assign", &json!({ "sealed": sealed, "participant": e.signer_did() }).to_string())).unwrap();
|
||||
let s: Value = serde_json::from_str(&e.study("score_probe", &json!({ "probe": "swap", "qasm": "OPENQASM 3.0;\nqubit[2] q;\ncx q[0], q[1];\ncx q[1], q[0];\ncx q[0], q[1];\n" }).to_string())).unwrap();
|
||||
assert_eq!(s["score"], 1.0, "{s}");
|
||||
let rec: Value = serde_json::from_str(&e.study("seal_record", &json!({ "record": { "protocol": digest, "condition": me["condition"], "probes": [{ "probe": "swap", "source": "OPENQASM 3.0;\nqubit[2] q;\ncx q[0], q[1];\ncx q[1], q[0];\ncx q[0], q[1];\n" }] } }).to_string())).unwrap();
|
||||
let report: Value = serde_json::from_str(&e.study("cohort", &json!({ "sealed": sealed, "records": [rec] }).to_string())).unwrap();
|
||||
assert_eq!(report["included"], 1, "{report}");
|
||||
assert!(e.study("nonsense", "{}").contains("no such call"));
|
||||
}
|
||||
|
||||
/// The convergence monitor in the page, on a history the page keeps.
|
||||
#[test]
|
||||
fn convergence_is_measured_in_the_page() {
|
||||
|
|
|
|||
Loading…
Reference in a new issue