00cfcf0530
`wai-quantum eval run` asks a model each task of a set, once and at temperature 0. The model is asked either for the program alone or for an answer in the answer view layer. The engine then scores what it writes: whether the answer reads, the verdict's level against the task's intent, whether the task is solved, and the joules per solved task where the model server's process energy can be read. Repair rounds send the engine's codes back for a corrected answer, and the report keeps the first try's counts. A model server that is neither on this machine nor on a private network is refused unless allowed. The report ranks nothing, and no report is committed. The task set (wai/quantum-conformance/eval/tasks.json) is fourteen tasks from the intent conformance set and the worked examples. Each has a reference program that a test holds to its intent by proof. `eval corpus` writes each task's answer four ways: the program alone, the layer whole, the layer with the request alone, and a job's JSON. Noise that two calls share is named once. token_cost.py counts each form with a model's own vocabulary. Its report on three open-weight vocabularies is committed beside the corpus. The smoke checks run a scripted model through the harness, and CI names each harness test. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1.8 KiB
1.8 KiB
The model harness's task set
What wai-quantum eval run asks a model, and the corpus the answer view
layer's token cost is measured on.
| Path | What |
|---|---|
tasks.json |
The tasks, in the conformance format: each what a model is asked (ask), the intent its program is held to, the run, and a reference program a test holds to the intent by proof. Built from the intent conformance set's cases (../intents/) and the worked examples (wai_quantum::engine::guide::examples), each named in from, with a few more of the first states of a circuits course. |
answers.json |
Each task's answer written four ways (wai-quantum eval corpus): the program alone in a fenced block, the answer view layer whole (the circuit, a sample, a verdict for each intent, a seal) and with the request alone (the sample and the verdicts), and the request an agent sends a run tool as JSON. The last two carry the same program and the same request. A test holds it to the task set, and every answer in the layer reads with no diagnostic and runs to its intents. |
token_cost.py |
Counts the tokens of each answer in each form with a model's own vocabulary (read from its GGUF file by a tokenizer tool), and reports the totals, the means and each form against the program alone. Standard library only. |
token-cost.json |
Its report on answers.json for three open-weight models' vocabularies; the answer view layer's specification records it (§9). |
wai-quantum eval run --tasks tasks.json --model http://127.0.0.1:8080/v1 --energy-pid PID --out report.json
wai-quantum eval corpus --tasks tasks.json --out answers.json
python3 -I token_cost.py answers.json --tokenizer A=MODEL.gguf --tokenizer B=… --tool TOKENIZER --out token-cost.json
A report is one model on one task set and ranks nothing; reports are kept where they are written, not here.