open-standards/wai/extensions/neural-dispatch.md

WAI Extension: Neural Dispatch

Status: Draft. Says which unit of a sink's device — its neural processor, its GPU or its CPU — runs a neural capability's networks, under the determinism tier the decode needs (determinism-tiers), and what the sink reports about the choice. A path that computes in floating point, a neural processor's included, never runs a decode that needs decode-equivalence. Defines the report wai-neural-dispatch/1 and seven refusal codes. No manifest member and no wire format: the envelope is unchanged, and the choice is the sink's. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

1. Scope

A neural capability's decode has an entropy stage, which turns the payload into symbols and a latent, and networks, which turn the latent into the medium. A sink can run the networks on more than one unit of its device:

• a neural processor (npu), driven through the platform's model runtime or through a browser's WebNN;
• a GPU (gpu);
• the CPU (cpu).

A path is a unit, the backend that drives it, and the arithmetic the networks are computed in there:

backend	what drives the unit
model-runtime	the platform's model runtime: it compiles a program for the device, and states in a compute plan the device it prefers for each operation
webnn	a browser's WebNN context, created for the unit
onnx-runtime	ONNX Runtime, the reference sink's path for every float neural capability
wai-ops	the integer operator set's interpreter (integer-operators), the path of every NeuralIntegerExact capability
wgsl	the integer operator set's WGSL kernels on a GPU (determinism-tiers §5, Claiming the tier on a GPU)

The arithmetic is integer (the integer arithmetic the capability's registration fixes, end to end) or float (floating point, at whatever precision, order and fusion the backend chooses: a neural processor computes in reduced precision, a GPU and a CPU reorder and fuse).

2. The tier a decode needs

A decode of capability C needs decode-equivalence when any of these holds:

1. C is registered at decode-equivalence (its registry class is NeuralIntegerExact, determinism-tiers §3);
2. the envelope's intent is replicate (SPEC §7);
3. the pin over C's parameter set states tier: "decode-equivalence" (SPEC §3.1).

Otherwise it needs C's registered tier: entropy-consistency for a NeuralEntropyExact capability, none for every other. A pin's other tokens, and a token the vocabulary does not register, add nothing. A pin applies to the capability it pins; a decode through the fallback reads no pin of the primary's.

3. Which paths may run it

• A float path MUST NOT run a decode that needs decode-equivalence (dispatch_tier_refused), whichever unit it names, the CPU included. Determinism-tiers §1 says why: the result of a float network depends on what the backend chooses per device.
• A float path MAY run any other decode. The decode holds C's registered tier, never more: a neural processor's output is the reconstruction of that device, and no tier over the reconstruction is claimed for it.
• An integer path runs only a capability registered at decode-equivalence, whose networks are integer; offered for another, it is refused (dispatch_arithmetic_mismatch). It holds decode-equivalence where determinism-tiers §5 says the device keeps the integer arithmetic: on the CPU always, on a GPU under Claiming the tier on a GPU.

A neural processor that runs integer graphs exactly, as wai/wai-ops-npu does for wai-ops/1-npu (integer-operators §9.2), would be an integer path; no decoder runs its graphs there in this revision (determinism-tiers §5), so no integer path names npu here.

The entropy stage is not a path's: every path runs it on the CPU, in the integer arithmetic the capability's registration fixes (§6.3). So at entropy-consistency a decode's symbols are the same whichever path runs its networks.

4. The choice

A sink offers its paths in an order of its choosing; the reference sink offers the neural processor first, then the CPU, then the integer interpreter (§6). For each offered path in order, the sink:

1. applies §3; a refused path is not tried;
2. asks whether the device can run it: whether the unit is there (dispatch_unit_absent), whether the parameter set holds a program for the backend (dispatch_program_absent), whether that program takes this payload's shape (dispatch_shape_mismatch), and, where the backend states a plan, whether the plan places any operation on the unit (dispatch_not_placed, §5);
3. runs the decode's networks there; a backend error refuses the path (dispatch_backend_failed).

The first path that runs is the decode's. When every offered path is refused, the decode fails with the report (§5), and SPEC §4 treats the capability as one this sink cannot decode for this envelope.

5. The report

A sink that dispatches a neural decode SHOULD give a report, a JSON object:

member	value
kind	"wai-neural-dispatch/1"
capability	the capability the payload decoded through
needed	the tier the decode needs (§2), as its token
ran	the path that ran, or null
refused	every path refused, in the order offered

A path is {"unit", "backend", "arithmetic"} with the tokens of §1. The path in ran adds tier, the tier the decode holds (§3), and placement: where the backend states a plan, {"npu", "gpu", "cpu", "unknown"}, how many of the program's operations the plan puts on each unit, constants left out; where it states none, null. A refused path adds reason, a code of §7.

Placement is the backend's statement. A model runtime that compiles a program for a device chooses, per operation, among the devices the program may use, and states its choice in a compute plan. A plan that puts no operation on the unit a path names is not a run there: the path is refused (dispatch_not_placed) and the next is tried. A plan that puts some operations elsewhere is reported as it is. WebNN states no placement: a page reports the unit it created its context for, with placement null, and the report says no more than that.

A report states which unit ran and at which tier. It is not an energy figure, and is never read as one (energy-measurement).

A sink of a fleet carries its reports in its run report, which holds each decode's set and tier to what the operator's model policy lets the sink's class run, and refuses a report whose float path claims decode-equivalence (model-governance §6.2).

6. The paths of the reference implementation

wai-rs/src/neural_dispatch.rs holds §2–§5 as pure functions: demand, admit, choose and Report::to_json. A page runs the same functions (wai-web's neural_npu bundle). wai.neural.bmshj2018 has a path on each unit the reference reaches; every other neural capability has the one path it always had, which the report states, with the neural processor refused (dispatch_tier_refused where the decode needs decode-equivalence, dispatch_program_absent otherwise).

6.1 The platform's model runtime (native, macOS)

Feature neural_npu of wai-rs (src/neural/npu.rs, src/neural/npu_runtime.rs). A parameter set holds a program for the runtime as two files: decoder.mlmodel, an ML Program's specification, and decoder.weight.bin, its weight file. The sink places the bytes it read from the set — the bytes a pin verified, for a pinned decode — in a package of its own, compiles the package with the runtime, loads it for the CPU and the neural processor, and reads the runtime's compute plan. The program's one input fixes the latent's shape; a payload of another shape is dispatch_shape_mismatch. neural::npu::decode_envelope_dispatched and decode_payload_dispatched decode with an offer (Offer::native(): the neural processor, then ONNX Runtime on the CPU, then the integer interpreter) and return the report beside the output; neural::npu::npu_plan gives the plan without decoding. On another platform, or without the feature, the path is dispatch_unit_absent. Every other entry point decodes on the CPU, as before.

tools/wai_bmshj2018_npu_set.py writes such a set for wai.neural.bmshj2018 from CompressAI's pretrained bmshj2018-factorized model: decoder.onnx and cdfs_q3.json as the CPU path reads them, the program computed in float16 with its input fixed at the latent of a 256 × 256 picture, the WebNN form of §6.2, and the set's usage statement (wai-usage.json, prior-carriage §18.1), which records the licence the weights are published under, BSD-3-Clause-Clear. It also writes one payload, a picture it draws from a seed and codes with the model's own encoder. GDN's reparameterised parameters are taken as values, so the program holds plain convolutions, multiplications and square roots; the tool refuses to write a set whose plain synthesis differs from CompressAI's by any amount.

6.2 WebNN (a page)

wai-web/demo/npu.html, with the neural_npu bundle. The page decodes the entropy stage through the bundle (bmshj2018Latent), builds the synthesis as a WebNN graph from decoder.webnn.json (the layers, each a transposed convolution or an inverse GDN, as offsets into decoder.f32, the float32 weights), and tries the contexts of the neural processor, the GPU and the CPU in that order (createContext with the deviceType the browser of §8 takes). A transposed convolution is built with no padding and followed by the slice its padding and output padding select, the same values: one browser's WebNN backend refuses explicit padding on a transposed convolution. npuDispatch applies §2–§4 to what the page found and gives the report. A browser with no navigator.ml gives every WebNN path dispatch_unit_absent; the page has no other path for the networks. A decode that needs decode-equivalence is refused before any context is created.

6.3 The entropy stage

neural_dispatch::bmshj2018::latent: the payload's header, the rANS stream decoded against the set's tables (cdfs_q<N>.json) to the symbols, and each symbol plus its channel's median, a value the set carries, as the latent. Integer and table reads only, so the symbols and the latent are the same on every machine and in every page. Their digest (symbols_digest) is the BLAKE3 of the symbols, each as four bytes little-endian.

6.4 Agreement with the reference

A float path's decode agrees with the reference at its tier when:

1. the symbols are the reference decode's (§6.3: the same digest); and
2. at none, which promises nothing of the reconstruction, the implementation holds its own bound: the RGB8 reconstruction within 40 dB PSNR of the reference sink's decode on the CPU (ONNX Runtime), on the set's payload. The bound is the implementation's check that the path decodes the capability, not a property of the capability.

At entropy-consistency the first condition is the tier's criterion (determinism-tiers §2). Nothing here compares reconstructions across devices for a tier: no tier above the registered one is claimed for a float path's output.

7. Codes

code	meaning
dispatch_tier_refused	the decode needs decode-equivalence and the path computes in floating point (§3)
dispatch_arithmetic_mismatch	an integer path offered for a capability whose networks are not integer (§3)
dispatch_unit_absent	the device has no such unit, or the backend is absent (§4)
dispatch_program_absent	the parameter set holds no program for the backend (§4)
dispatch_shape_mismatch	the set's program does not take this payload's shape (§4)
dispatch_not_placed	the backend's plan places no operation on the path's unit (§5)
dispatch_backend_failed	the backend failed the decode (§4)

8. Evidence

claim	evidence	where
§2–§5 as functions: the tier a decode needs; no float path for decode-equivalence; the integer path for an integer capability; a plan with nothing on the unit moves on; the report's members	library unit tests, run with --features neural_dispatch	wai-rs/src/neural_dispatch.rs tests
wai.neural.bmshj2018 decodes on the neural processor through the platform's model runtime: the plan puts all 21 operations of the program on it; the symbols are the CPU reference's (digest 417452fc…1dd1, 49 152 symbols); the RGB8 reconstruction is within the bound of §6.4	measured 2026-11-09 with rustc 1.99.1 on macOS 27.2 arm64, on two hosts whose neural processors are of different generations, with the set and payload tools/wai_bmshj2018_npu_set.py wrote (quality 3, seed 20261109): 56.85 dB and 56.69 dB against the CPU reference, at most 2 apart in any sample, 26 377 and 27 424 of 196 608 samples differing. The set is not committed (its weights are CompressAI's); the test reads it from WAI_NPU_SET and, with WAI_REQUIRE_NPU_SET=1, fails without it	wai-rs/tests/neural_npu.rs
a decode that needs decode-equivalence refuses every float path: a pin that states the tier and intent = replicate leave wai.neural.bmshj2018 no path; wai.neural.int_synth refuses the neural processor and ONNX Runtime and decodes on the integer interpreter to the corpus hash	the same test file; the integer capability's test reads the committed neural-conformance corpus and runs wherever the feature builds	wai-rs/tests/neural_npu.rs
WebNN in a page, where the browser gives it: the symbols are the native sink's, the synthesis runs on the context created for the neural processor (ran npu/webnn, tier none, placement null), within the bound of §6.4 of the native CPU reference; intent = replicate is refused on every WebNN path with no context created; and where the browser gives no WebNN, every path is dispatch_unit_absent and the symbols are still the native sink's	measured 2026-11-09 on macOS 27.2 arm64 with Playwright 1.60's browsers, the same set and payload: Chromium 148 with WebNN enabled (--enable-features=WebMachineLearningNeuralNetwork) ran the synthesis on the context it gave for the neural processor: 89.61 dB against the CPU reference, at most 1 apart in any sample, 14 of 196 608 samples differing. That is closer to the CPU than the model runtime's float16 program came, and WebNN states no placement, so this row claims the context the page asked for, not the unit that computed. Chromium 148 as it ships, a Chromium-based browser at 156, and Playwright's WebKit 26.4 and Gecko 150 builds expose no navigator.ml: there the page reports WebNN absent and decodes the entropy stage only. The verifier fails unless one browser gives WebNN. Not run in CI: the set is not committed	wai-web/demo/npu.html, npu.js, verify_npu.mjs