Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 15 Packages Wiki Activity Actions 152
open-standards/wai/quantum-conformance/eval/token-cost.json
Transaction Science 00cfcf0530 wai-quantum: a model scored by the engine on a task set, and each answer counted in tokens four ways 
`wai-quantum eval run` asks a model each task of a set, once and at
temperature 0. The model is asked either for the program alone or for an
answer in the answer view layer. The engine then scores what it writes:
whether the answer reads, the verdict's level against the task's intent,
whether the task is solved, and the joules per solved task where the model
server's process energy can be read.

Repair rounds send the engine's codes back for a corrected answer, and the
report keeps the first try's counts. A model server that is neither on
this machine nor on a private network is refused unless allowed. The
report ranks nothing, and no report is committed.

The task set (wai/quantum-conformance/eval/tasks.json) is fourteen tasks
from the intent conformance set and the worked examples. Each has a
reference program that a test holds to its intent by proof.

`eval corpus` writes each task's answer four ways: the program alone, the
layer whole, the layer with the request alone, and a job's JSON. Noise
that two calls share is named once. token_cost.py counts each form with
a model's own vocabulary. Its report on three open-weight vocabularies is
committed beside the corpus.

The smoke checks run a scripted model through the harness, and CI names
each harness test.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-28 08:42:54 -04:00

382 lines
8.4 KiB
JSON
Raw Permalink Blame History

{
"format": "wai.quantum.token-cost/1",
"about": "The tokens a task set's answers take, each written four ways: plain (the program alone, in a fenced block), views (the answer view layer: the program, its circuit, a sample when it measures, a verdict for each intent, a seal), views_request (the layer with the sample and the verdicts alone) and job (the request an agent sends a run tool, as JSON). views_request and job carry the same program and the same request; views adds the circuit and the seal; plain carries the program alone.",
"answers": 14,
"bytes": {
"plain": 1402,
"views": 5119,
"views_request": 3932,
"job": 2857
},
"tokenizers": [
{
"name": "A",
"vocabulary": 49159,
"totals": {
"plain": 841,
"views": 2365,
"views_request": 1967,
"job": 1492
},
"mean": {
"plain": 60.1,
"views": 168.9,
"views_request": 140.5,
"job": 106.6
},
"against_plain": {
"plain": 1.0,
"views": 2.812,
"views_request": 2.339,
"job": 1.774
},
"views_request_against_job": 1.318,
"per_answer": [
{
"name": "plus",
"plain": 42,
"views": 120,
"views_request": 92,
"job": 64
},
{
"name": "bell",
"plain": 61,
"views": 141,
"views_request": 113,
"job": 85
},
{
"name": "bell-odd-parity",
"plain": 58,
"views": 138,
"views_request": 110,
"job": 82
},
{
"name": "ghz-3",
"plain": 62,
"views": 144,
"views_request": 116,
"job": 88
},
{
"name": "ghz-4-prepares",
"plain": 81,
"views": 280,
"views_request": 246,
"job": 189
},
{
"name": "uniform-3",
"plain": 40,
"views": 198,
"views_request": 170,
"job": 130
},
{
"name": "grover-2",
"plain": 85,
"views": 156,
"views_request": 128,
"job": 102
},
{
"name": "rotation-three-quarters",
"plain": 56,
"views": 136,
"views_request": 108,
"job": 80
},
{
"name": "w-3",
"plain": 98,
"views": 248,
"views_request": 220,
"job": 183
},
{
"name": "flip-the-last",
"plain": 42,
"views": 114,
"views_request": 86,
"job": 60
},
{
"name": "cz-from-cx",
"plain": 45,
"views": 147,
"views_request": 119,
"job": 83
},
{
"name": "swap-from-cx",
"plain": 53,
"views": 155,
"views_request": 127,
"job": 91
},
{
"name": "merged-rotations",
"plain": 57,
"views": 194,
"views_request": 166,
"job": 130
},
{
"name": "noisy-bell",
"plain": 61,
"views": 194,
"views_request": 166,
"job": 125
}
]
},
{
"name": "B",
"vocabulary": 248320,
"totals": {
"plain": 834,
"views": 2312,
"views_request": 1928,
"job": 1491
},
"mean": {
"plain": 59.6,
"views": 165.1,
"views_request": 137.7,
"job": 106.5
},
"against_plain": {
"plain": 1.0,
"views": 2.772,
"views_request": 2.312,
"job": 1.788
},
"views_request_against_job": 1.293,
"per_answer": [
{
"name": "plus",
"plain": 42,
"views": 117,
"views_request": 90,
"job": 64
},
{
"name": "bell",
"plain": 60,
"views": 137,
"views_request": 110,
"job": 84
},
{
"name": "bell-odd-parity",
"plain": 58,
"views": 135,
"views_request": 108,
"job": 82
},
{
"name": "ghz-3",
"plain": 62,
"views": 141,
"views_request": 114,
"job": 88
},
{
"name": "ghz-4-prepares",
"plain": 80,
"views": 273,
"views_request": 240,
"job": 186
},
{
"name": "uniform-3",
"plain": 40,
"views": 195,
"views_request": 168,
"job": 130
},
{
"name": "grover-2",
"plain": 84,
"views": 152,
"views_request": 125,
"job": 101
},
{
"name": "rotation-three-quarters",
"plain": 54,
"views": 131,
"views_request": 104,
"job": 78
},
{
"name": "w-3",
"plain": 98,
"views": 245,
"views_request": 218,
"job": 186
},
{
"name": "flip-the-last",
"plain": 42,
"views": 111,
"views_request": 84,
"job": 60
},
{
"name": "cz-from-cx",
"plain": 45,
"views": 144,
"views_request": 117,
"job": 84
},
{
"name": "swap-from-cx",
"plain": 53,
"views": 152,
"views_request": 125,
"job": 92
},
{
"name": "merged-rotations",
"plain": 56,
"views": 189,
"views_request": 162,
"job": 129
},
{
"name": "noisy-bell",
"plain": 60,
"views": 190,
"views_request": 163,
"job": 127
}
]
},
{
"name": "C",
"vocabulary": 262144,
"totals": {
"plain": 835,
"views": 2282,
"views_request": 1940,
"job": 1483
},
"mean": {
"plain": 59.6,
"views": 163.0,
"views_request": 138.6,
"job": 105.9
},
"against_plain": {
"plain": 1.0,
"views": 2.733,
"views_request": 2.323,
"job": 1.776
},
"views_request_against_job": 1.308,
"per_answer": [
{
"name": "plus",
"plain": 42,
"views": 115,
"views_request": 91,
"job": 64
},
{
"name": "bell",
"plain": 60,
"views": 135,
"views_request": 111,
"job": 84
},
{
"name": "bell-odd-parity",
"plain": 58,
"views": 133,
"views_request": 109,
"job": 82
},
{
"name": "ghz-3",
"plain": 62,
"views": 139,
"views_request": 115,
"job": 88
},
{
"name": "ghz-4-prepares",
"plain": 80,
"views": 271,
"views_request": 241,
"job": 186
},
{
"name": "uniform-3",
"plain": 40,
"views": 193,
"views_request": 169,
"job": 130
},
{
"name": "grover-2",
"plain": 84,
"views": 150,
"views_request": 126,
"job": 101
},
{
"name": "rotation-three-quarters",
"plain": 55,
"views": 130,
"views_request": 106,
"job": 79
},
{
"name": "w-3",
"plain": 98,
"views": 243,
"views_request": 219,
"job": 183
},
{
"name": "flip-the-last",
"plain": 42,
"views": 109,
"views_request": 85,
"job": 60
},
{
"name": "cz-from-cx",
"plain": 45,
"views": 141,
"views_request": 117,
"job": 83
},
{
"name": "swap-from-cx",
"plain": 53,
"views": 149,
"views_request": 125,
"job": 91
},
{
"name": "merged-rotations",
"plain": 56,
"views": 186,
"views_request": 162,
"job": 128
},
{
"name": "noisy-bell",
"plain": 60,
"views": 188,
"views_request": 164,
"job": 124
}
]
}
]
}
Reference in a new issue View git blame Copy permalink