open-standards/wai/extensions/neural-dispatch.md

# WAI Extension: Neural Dispatch

> Status: Draft. Says which unit of a sink's device — its neural processor, its GPU or its
> CPU — runs a neural capability's networks, under the determinism tier the decode needs
> ([determinism-tiers](determinism-tiers.md)), and what the sink reports about the choice.
> A path that computes in floating point, a neural processor's included, never runs a decode
> that needs `decode-equivalence`. Defines the report `wai-neural-dispatch/1` and seven
> refusal codes. No manifest member and no wire format: the envelope is unchanged, and the
> choice is the sink's.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.

## 1. Scope

A neural capability's decode has an entropy stage, which turns the payload into symbols and
a latent, and networks, which turn the latent into the medium. A sink can run the networks on
more than one unit of its device:

- a **neural processor** (`npu`), driven through the platform's model runtime or through a
  browser's WebNN;
- a **GPU** (`gpu`);
- the **CPU** (`cpu`).

A **path** is a unit, the **backend** that drives it, and the **arithmetic** the networks are
computed in there:

| backend | what drives the unit |
|---|---|
| `model-runtime` | the platform's model runtime: it compiles a program for the device, and states in a compute plan the device it prefers for each operation |
| `webnn` | a browser's WebNN context, created for the unit |
| `onnx-runtime` | ONNX Runtime, the reference sink's path for every float neural capability |
| `wai-ops` | the integer operator set's interpreter ([integer-operators](integer-operators.md)), the path of every `NeuralIntegerExact` capability |
| `wgsl` | the integer operator set's WGSL kernels on a GPU (determinism-tiers §5, *Claiming the tier on a GPU*) |

The arithmetic is `integer` (the integer arithmetic the capability's registration fixes, end
to end) or `float` (floating point, at whatever precision, order and fusion the backend
chooses: a neural processor computes in reduced precision, a GPU and a CPU reorder and fuse).

## 2. The tier a decode needs

A decode of capability `C` **needs** `decode-equivalence` when any of these holds:

1. `C` is registered at `decode-equivalence` (its registry class is `NeuralIntegerExact`,
   determinism-tiers §3);
2. the envelope's `intent` is `replicate` (SPEC §7);
3. the pin over `C`'s parameter set states `tier: "decode-equivalence"` (SPEC §3.1).

Otherwise it needs `C`'s registered tier: `entropy-consistency` for a `NeuralEntropyExact`
capability, `none` for every other. A pin's other tokens, and a token the vocabulary does not
register, add nothing. A pin applies to the capability it pins; a decode through the
fallback reads no pin of the primary's.

## 3. Which paths may run it

- A `float` path **MUST NOT** run a decode that needs `decode-equivalence`
  (`dispatch_tier_refused`), whichever unit it names, the CPU included. Determinism-tiers §1
  says why: the result of a float network depends on what the backend chooses per device.
- A `float` path MAY run any other decode. The decode holds `C`'s registered tier, never
  more: a neural processor's output is the reconstruction of that device, and no tier over
  the reconstruction is claimed for it.
- An `integer` path runs only a capability registered at `decode-equivalence`, whose networks
  are integer; offered for another, it is refused (`dispatch_arithmetic_mismatch`). It holds
  `decode-equivalence` where determinism-tiers §5 says the device keeps the integer
  arithmetic: on the CPU always, on a GPU under *Claiming the tier on a GPU*.

A neural processor that runs integer graphs exactly, as `wai/wai-ops-npu` does for
`wai-ops/1-npu` (integer-operators §9.2), would be an `integer` path; no decoder runs its
graphs there in this revision (determinism-tiers §5), so no `integer` path names `npu` here.

The entropy stage is not a path's: every path runs it on the CPU, in the integer arithmetic
the capability's registration fixes (§6.3). So at `entropy-consistency` a decode's symbols
are the same whichever path runs its networks.

## 4. The choice

A sink offers its paths in an order of its choosing; the reference sink offers the neural
processor first, then the CPU, then the integer interpreter (§6). For each offered path in
order, the sink:

1. applies §3; a refused path is not tried;
2. asks whether the device can run it: whether the unit is there (`dispatch_unit_absent`),
   whether the parameter set holds a program for the backend (`dispatch_program_absent`),
   whether that program takes this payload's shape (`dispatch_shape_mismatch`), and, where
   the backend states a plan, whether the plan places any operation on the unit
   (`dispatch_not_placed`, §5);
3. runs the decode's networks there; a backend error refuses the path
   (`dispatch_backend_failed`).

The first path that runs is the decode's. When every offered path is refused, the decode
fails with the report (§5), and SPEC §4 treats the capability as one this sink cannot decode
for this envelope.

## 5. The report

A sink that dispatches a neural decode **SHOULD** give a report, a JSON object:

| member | value |
|---|---|
| `kind` | `"wai-neural-dispatch/1"` |
| `capability` | the capability the payload decoded through |
| `needed` | the tier the decode needs (§2), as its token |
| `ran` | the path that ran, or `null` |
| `refused` | every path refused, in the order offered |

A path is `{"unit", "backend", "arithmetic"}` with the tokens of §1. The path in `ran` adds
`tier`, the tier the decode holds (§3), and `placement`: where the backend states a plan,
`{"npu", "gpu", "cpu", "unknown"}`, how many of the program's operations the plan puts on
each unit, constants left out; where it states none, `null`. A refused path adds `reason`,
a code of §7.

**Placement is the backend's statement.** A model runtime that compiles a program for a
device chooses, per operation, among the devices the program may use, and states its choice
in a compute plan. A plan that puts no operation on the unit a path names is not a run there:
the path is refused (`dispatch_not_placed`) and the next is tried. A plan that puts some
operations elsewhere is reported as it is. WebNN states no placement: a page reports the unit
it created its context for, with `placement` `null`, and the report says no more than that.

A report states which unit ran and at which tier. It is not an energy figure, and is never
read as one ([energy-measurement](energy-measurement.md)).

## 6. The paths of the reference implementation

`wai-rs/src/neural_dispatch.rs` holds §2–§5 as pure functions: `demand`, `admit`, `choose`
and `Report::to_json`. A page runs the same functions (`wai-web`'s `neural_npu` bundle).
`wai.neural.bmshj2018` has a path on each unit the reference reaches; every other neural
capability has the one path it always had, which the report states, with the neural
processor refused (`dispatch_tier_refused` where the decode needs `decode-equivalence`,
`dispatch_program_absent` otherwise).

### 6.1 The platform's model runtime (native, macOS)

Feature `neural_npu` of `wai-rs` (`src/neural/npu.rs`, `src/neural/npu_runtime.rs`). A
parameter set holds a program for the runtime as two files: `decoder.mlmodel`, an ML Program's
specification, and `decoder.weight.bin`, its weight file. The sink places the bytes it read
from the set — the bytes a pin verified, for a pinned decode — in a package of its own,
compiles the package with the runtime, loads it for the CPU and the neural processor, and
reads the runtime's compute plan. The program's one input fixes the latent's shape; a
payload of another shape is `dispatch_shape_mismatch`. `neural::npu::decode_envelope_dispatched`
and `decode_payload_dispatched` decode with an offer (`Offer::native()`: the neural processor,
then ONNX Runtime on the CPU, then the integer interpreter) and return the report beside the
output; `neural::npu::npu_plan` gives the plan without decoding. On another platform, or
without the feature, the path is `dispatch_unit_absent`. Every other entry point decodes on
the CPU, as before.

`tools/wai_bmshj2018_npu_set.py` writes such a set for `wai.neural.bmshj2018` from
CompressAI's pretrained bmshj2018-factorized model: `decoder.onnx` and `cdfs_q3.json` as the
CPU path reads them, the program computed in float16 with its input fixed at the latent of a
256 × 256 picture, the WebNN form of §6.2, and the set's usage statement (`wai-usage.json`,
[prior-carriage](prior-carriage.md) §18.1), which records the licence the weights are
published under, BSD-3-Clause-Clear. It also writes one payload, a picture it draws from a
seed and codes with the model's own encoder. GDN's reparameterised parameters are taken as
values, so the program holds plain convolutions, multiplications and square roots; the tool
refuses to write a set whose plain synthesis differs from CompressAI's by any amount.

### 6.2 WebNN (a page)

`wai-web/demo/npu.html`, with the `neural_npu` bundle. The page decodes the entropy stage
through the bundle (`bmshj2018Latent`), builds the synthesis as a WebNN graph from
`decoder.webnn.json` (the layers, each a transposed convolution or an inverse GDN, as offsets
into `decoder.f32`, the float32 weights), and tries the contexts of the neural processor, the
GPU and the CPU in that order (`createContext` with the `deviceType` the browser of §8 takes).
A transposed convolution is built with no padding and followed by the slice its padding and
output padding select, the same values: one browser's WebNN backend refuses explicit padding
on a transposed convolution.
`npuDispatch` applies §2–§4 to what the page found and gives the report. A browser with no
`navigator.ml` gives every WebNN path `dispatch_unit_absent`; the page has no other path for
the networks. A decode that needs `decode-equivalence` is refused before any context is
created.

### 6.3 The entropy stage

`neural_dispatch::bmshj2018::latent`: the payload's header, the rANS stream decoded against
the set's tables (`cdfs_q<N>.json`) to the symbols, and each symbol plus its channel's
median, a value the set carries, as the latent. Integer and table reads only, so the symbols
and the latent are the same on every machine and in every page. Their digest
(`symbols_digest`) is the BLAKE3 of the symbols, each as four bytes little-endian.

### 6.4 Agreement with the reference

A float path's decode agrees with the reference at its tier when:

1. the symbols are the reference decode's (§6.3: the same digest); and
2. at `none`, which promises nothing of the reconstruction, the implementation holds its own
   bound: the RGB8 reconstruction within 40 dB PSNR of the reference sink's decode on the
   CPU (ONNX Runtime), on the set's payload. The bound is the implementation's check that the
   path decodes the capability, not a property of the capability.

At `entropy-consistency` the first condition is the tier's criterion (determinism-tiers §2).
Nothing here compares reconstructions across devices for a tier: no tier above the
registered one is claimed for a float path's output.

## 7. Codes

| code | meaning |
|---|---|
| `dispatch_tier_refused` | the decode needs `decode-equivalence` and the path computes in floating point (§3) |
| `dispatch_arithmetic_mismatch` | an integer path offered for a capability whose networks are not integer (§3) |
| `dispatch_unit_absent` | the device has no such unit, or the backend is absent (§4) |
| `dispatch_program_absent` | the parameter set holds no program for the backend (§4) |
| `dispatch_shape_mismatch` | the set's program does not take this payload's shape (§4) |
| `dispatch_not_placed` | the backend's plan places no operation on the path's unit (§5) |
| `dispatch_backend_failed` | the backend failed the decode (§4) |

## 8. Evidence

| claim | evidence | where |
|---|---|---|
| §2–§5 as functions: the tier a decode needs; no float path for `decode-equivalence`; the integer path for an integer capability; a plan with nothing on the unit moves on; the report's members | library unit tests, run with `--features neural_dispatch` | `wai-rs/src/neural_dispatch.rs` tests |
| `wai.neural.bmshj2018` decodes on the neural processor through the platform's model runtime: the plan puts all 21 operations of the program on it; the symbols are the CPU reference's (digest `417452fc…1dd1`, 49 152 symbols); the RGB8 reconstruction is within the bound of §6.4 | measured 2026-11-09 with rustc 1.99.1 on macOS 27.2 arm64, on two hosts whose neural processors are of different generations, with the set and payload `tools/wai_bmshj2018_npu_set.py` wrote (quality 3, seed 20261109): 56.85 dB and 56.69 dB against the CPU reference, at most 2 apart in any sample, 26 377 and 27 424 of 196 608 samples differing. The set is not committed (its weights are CompressAI's); the test reads it from `WAI_NPU_SET` and, with `WAI_REQUIRE_NPU_SET=1`, fails without it | `wai-rs/tests/neural_npu.rs` |
| a decode that needs `decode-equivalence` refuses every float path: a pin that states the tier and `intent = replicate` leave `wai.neural.bmshj2018` no path; `wai.neural.int_synth` refuses the neural processor and ONNX Runtime and decodes on the integer interpreter to the corpus hash | the same test file; the integer capability's test reads the committed `neural-conformance` corpus and runs wherever the feature builds | `wai-rs/tests/neural_npu.rs` |
| WebNN in a page, where the browser gives it: the symbols are the native sink's, the synthesis runs on the context created for the neural processor (`ran` `npu`/`webnn`, tier `none`, `placement` `null`), within the bound of §6.4 of the native CPU reference; `intent = replicate` is refused on every WebNN path with no context created; and where the browser gives no WebNN, every path is `dispatch_unit_absent` and the symbols are still the native sink's | measured 2026-11-09 on macOS 27.2 arm64 with Playwright 1.60's browsers, the same set and payload: Chromium 148 with WebNN enabled (`--enable-features=WebMachineLearningNeuralNetwork`) ran the synthesis on the context it gave for the neural processor: 89.61 dB against the CPU reference, at most 1 apart in any sample, 14 of 196 608 samples differing. That is closer to the CPU than the model runtime's float16 program came, and WebNN states no placement, so this row claims the context the page asked for, not the unit that computed. Chromium 148 as it ships, a Chromium-based browser at 156, and Playwright's WebKit 26.4 and Gecko 150 builds expose no `navigator.ml`: there the page reports WebNN absent and decodes the entropy stage only. The verifier fails unless one browser gives WebNN. Not run in CI: the set is not committed | `wai-web/demo/npu.html`, `npu.js`, `verify_npu.mjs` |