open-standards/wai/research/cross-fabric.md

# WAI across fabrics: research for the packer and the verifiers

> **Status.** Research note, informative, not normative. It records what decided the change
> that lets each step of a WAI pack or verify run on the fabric that does it with the least
> measured energy (the CPU, a GPU, a hardware media engine, or a neural processor):
> [packaging](../extensions/packaging.md) §11.6, [jwp-receipts](../extensions/jwp-receipts.md)
> §2.4.8 and [energy-measurement](../extensions/energy-measurement.md) §6.3. Nothing here changes
> a byte of a package or a verdict.

**Method.** Every statement about an interface or a specification was checked against that
document on 2026-10-22. Items marked **[measured]** were run for this note (the setup is in §9);
items marked **[local]** were read from the platform's SDK headers on the machine that ran them.
Exact counts and digests are exact. Times and energies are indicative: each table in §9 is **one
run**, on a shared host whose load is given, and none is a sample of anything.

## 1. The hardware the measurements ran on

Hardware is named as the rest of the repository names it: by architecture, PCI vendor and device
id, and operating-system version.

- **arm64 desktops** (macOS 27.0.1), each with an integrated GPU and a media engine that decodes
  AV1 in hardware and encodes no AV1. The measurements of §9 ran on one of them, shared with other
  work. Every one reports the platform's current GPU interface.
- **A Windows x86_64 host** with an integrated GPU of PCI vendor 0x8086, device 0x9a49, and a
  discrete GPU of PCI vendor 0x10de, device 0x1f15. Its vendor's published support matrix gives
  that discrete GPU's architecture neither AV1 decode nor AV1 encode; the integrated GPU decodes
  AV1, with a film-grain path its vendor does not state as bit-exact (§6.1).
- **Linux x86_64 hosts** with GPUs of PCI vendor 0x10de, device 0x28e1, of vendor 0x8086, device
  0xa7a0, and of vendor 0x1002, device 0x1586, as determinism-tiers §5 records them. Their media
  engines were not probed for this note.

## 2. The steps, and the fabrics that can run each

| Step | CPU | GPU | Media engine | Neural processor | Placed on |
|---|---|---|---|---|---|
| `source-decode` (a coded source to frames, before a pack) | dav1d | | AV1 decode where the engine has it | | either, on equal pictures (§6) |
| `source-read`, `segment` | yes | | | | CPU (sequential, cheap) |
| `base-search` (the integer motion search) | reference | exact integer kernel | none: no engine runs §5.5's search, and the platform's hardware motion estimation is not it (§3) | unsound (FP16, §5) | GPU where it qualifies |
| `base-code`, `refine-code` (residuals, rANS) | reference | | none: no engine codes `WIV1` or `WIR1` | | CPU (rANS is sequential) |
| `sse` | reference | exact (partial sums under 2^32) | | unsound | by measured cost |
| `object-hash` (BLAKE3) | reference | exact (chunks on the device, parents on the host) | | | by measured cost |
| `digest` (SHA-256) | yes | no: one message is one chain (FIPS 180-4 §6.2.2) | | | CPU |
| `floor-read` (AV1 and CMAF structure) | yes | | | | CPU |
| `floor-decode` | dav1d | | AV1 decode, as above | | informative only (§6.3) |
| `index`, `io` | yes | | | | CPU |

What this decides: the GPU takes the one step that dominates a pack's work, the motion search (at
least 96.8% of the CPU-only pack's CPU time in the run of §9.2), plus SSE and BLAKE3 where the
measured cost says so, which on the host of §9 it does not (§9.3). No hardware encoder computes any
WAI step. The media engine decodes AV1 sources and floors. The neural processor computes nothing
exact.

## 3. Media engines

- **AV1 decode, and no AV1 encode, on the arm64 desktops.** Every published specification of their
  media engines lists AV1 decode and no AV1 encode, and the platform's compression headers name no
  AV1 encoder **[local]**. ffmpeg's encoders for the platform's framework cover H.264, HEVC and
  ProRes only.
- **What the decode needs.** A decode session takes the sequence header OBU from an `av1C` record;
  a low-overhead OBU stream (`-f obu`) gives it none up front, so it does not decode there, while
  the ladder's CMAF (with `av1C`) does **[measured]**. The reference tool therefore decodes floors
  from the package's CMAF files.
- **What it gives back [measured]** (ffmpeg 9.0.2, macOS 27.0.1):
  - The ladder of §9 is AV1 Main profile, **10-bit** (each rung's `av1C`). The engine's hardware
    frames are P010; a download as 8-bit NV12 is refused.
  - Its raw P010 frames equal, frame by frame, dav1d's planar 10-bit output repacked to P010. But
    ffmpeg's P010-to-8-bit conversion of them does not equal dav1d's own 10-to-8-bit conversion:
    that conversion rounds and dithers, so a comparison stays at the stream's bit depth or
    converts exactly.
  - Earlier comparisons hashed `-f framemd5` outputs with stderr piped into the hash; the
    "difference" they showed was ffmpeg's error text. Every comparison in §9 hashes the output
    file, not the console.
- **Hardware motion estimation.** The platform offers a motion-estimation session returning 4×4 or
  16×16 vectors, and documents neither the engine, its cost metric, its search window nor its
  determinism. It cannot compute §5.5's normative search.
- **Other platforms.** One vendor's published support matrix gives AV1 decode from one GPU
  generation and AV1 encode from a later one, and states that its decoder runs independently of
  the compute engine. Another's media driver decodes AV1 from its twelfth graphics generation, in
  its full-feature build. A third's kernel driver decodes AV1 on its third video-core generation
  and encodes from its fourth. Vulkan defines `VK_KHR_video_decode_av1` from Vulkan 1.3.277 and
  `VK_KHR_video_encode_av1` from 1.3.302.

## 4. GPU compute

- **WGSL integer semantics** (WGSL, W3C Candidate Recommendation Draft): `i32` operations "use
  two's complement arithmetic and have defined overflow behavior"; `>>` on a signed value inserts
  copies of the sign bit (§8.10); an integer division by zero evaluates to `e1` (§8.8); an
  out-of-bounds access is a dynamic error the implementation does not perform as written (§6.5.7).
  So a WGSL kernel that keeps every value inside its type and every index inside its buffer has
  one result on every conformant device. The WAI kernels do: a block's cost is at most
  `block² · 765`, refused on the host where that passes 2^32 − 1. What a translator does with the
  cases WGSL defines and its target languages do not is determinism-tiers §5's subject.
- **The backend languages.** MSL leaves signed integer overflow undefined and its atomics relaxed,
  so only commutative integer reductions are order-free there. The WAI kernels use no atomics:
  each block's minimum is a tree reduction over a total order, so it does not depend on which
  thread visits which candidate.
- **BLAKE3 on GPUs.** The specification splits input into 1 KiB chunks hashed independently.
  Published GPU ports measured slower than one CPU core on a discrete GPU, or than eight CPU
  threads on an integrated one, and none reported energy. Measured here in §9.3.

## 5. Neural processors

The neural processor of the arm64 desktops computes in FP16, which the platform's machine-learning
framework can permit but not require, and its documentation states that its capabilities differ by
hardware and software generation. FP16 represents integers exactly only up to 2048, and one 8-bit
squared difference is 65025. **No WAI step can run on it.** The fabric term is `npu`, the term
determinism-tiers uses for `wai-ops/1-npu`.

## 6. Determinism and conformance

### 6.1 AV1 decoding is exact, film grain aside

AV1 Bitstream & Decoding Process Specification, Version 1.0.0 with Errata 1, §7.2: "When
film_grain_params_present is equal to 0, decoders shall produce output frames that are identical
in all respects and have the same output order as those produced by the decoding process
specified herein." With film grain, a conformant decoder may instead match only the pre-grain
frames of §7.18.2 and add grain with no "perceptually significant differences". dav1d states that
it passes the AV1 conformance streams with grain on. One GPU vendor's media driver states that its
film-grain path does not guarantee a bit match, and AV1 decode defects have been fixed in more than
one vendor's driver.

**Tested [measured]:** the media engine decoded each rung of the ladder of §9, and its pictures
were compared with dav1d's (§9.1). The ladder carries no film grain: dav1d gives the same frames
with grain applied and with it exported.

### 6.2 Hardware encoders are not reproducible

No vendor states that its hardware encoder is deterministic. One vendor's encoder guide documents
split-frame encoding that turns on with the number of engines and the frame height, and another's
low-power AV1 rate control runs in firmware. Software encoders are reproducible only under pinned
conditions.

### 6.3 What it means for WAI

- **Receipts.** A package's bytes do not depend on where its steps ran, because every step is
  integer and the GPU kernels compute the same integers: the receipt, the index and every object
  are the CPU's (§9.1). Where the steps ran is recorded only in a metered package's measurement
  claim, which §11.2 already exempts, so an unmetered package records nothing machine-specific.
- **§11.2 on non-deterministic hardware.** No WAI step may be computed by a hardware encoder. A
  floor a packager is given is an input; a floor encoded on a hardware encoder is one §7.5 covers,
  and §11.2 does not hold its bytes.
- **Verifiers' independence.** A check whose expected value the package states (an object's
  content hash) may run anywhere: a match shows the fabric computed it, and a mismatch is computed
  again on the CPU before it is reported. A floor states no decoded-output digest, so a hardware
  decode never decides a floor verdict, and a hardware decoder may conceal an error that dav1d
  reports. The Python verifier stays on the CPU and shares no code with wai-rs; the GPU path is
  wai-pack's.

## 7. Measuring energy per fabric without privileges

### 7.1 macOS 27

- **IOReport** is a private interface (the SDK ships a stub and no header **[local]**). On macOS 27
  its Energy Model CPU and neural-processor channels stay visible and stop updating for a reader
  without a private entitlement. **[measured]** on the host of §9: of 645 Energy Model channels,
  only `GPU Energy` (nJ) moved over a 6 s span during a hardware decode, a dav1d decode and idle;
  the CPU, per-core, neural-processor, video-encoder, memory and display channels read zero, and
  there is no decoder energy channel. So **the media engine and the neural processor have no
  unprivileged counter there**; a privileged sampler reads the neural processor, and nothing in
  this change runs privileged.
- **CPU, per process:** `proc_pid_rusage(RUSAGE_INFO_V6)` gives the kernel's account of the task's
  own CPU energy (`ri_energy_nj`) and the energy it billed to the task and serviced for others
  (`ri_billed_energy`, `ri_serviced_energy`), readable for one's own process or a child. A task's
  `task` share is `ri_energy_nj + ri_billed_energy − ri_serviced_energy`, the definition the
  sink-metering profiles use, so energy a system service spent for the process is in it. Other
  load on the host enters it only through contention.
- **GPU, per process:** the accelerator's user clients in the I/O registry carry each creator's pid
  and its accumulated GPU time (ns), readable by any process **[measured]**. The repository's own
  measurement found that dividing the shared rail by GPU time charged a decode 3 to 15 times its own
  GPU energy, and that a client disappears with its process. So the rail is attributed to a process
  only when it was the rail's sole user: the clients are sampled every 200 ms through the span, each
  client's GPU time differenced between samples, and the span refused when a client vanishes, a
  difference is negative, or the process held under 99% of the GPU time the samples saw. The rail
  is then sealed whole, its idle and static draw over the span included, never divided by busy
  time. The rail holds a reading for hundreds of milliseconds and then catches up in one step, so
  it is read on its next republish at each end, inside the sampled window.

### 7.2 Linux and Windows (not run here)

- RAPL's `energy_uj` became root-only in Linux 5.10, and perf's RAPL events are system-wide only,
  so reading CPU energy there takes an owner's act (`cap_perfmon`, or a group-readable
  `energy_uj`).
- NVML's total-energy query (mJ) is not marked as needing privilege; the in-kernel hwmon power and
  energy attributes of two other GPU drivers are world-readable.
- Windows' Energy Meter Interface is readable unelevated.

## 8. The design choices, and what decided them

| Choice | Decided by |
|---|---|
| The GPU computes the motion search, SSE and BLAKE3 chunks, in WGSL integer kernels through wgpu, not in one platform's shading language | WGSL's defined integer arithmetic (§4) gives one result on every conformant device; the repository's GPU precedent (`wai-ops-gpu`) is wgpu. Exactness is claimed only on the adapters a run showed (`wai-pack-gpu`'s crate documentation lists them), with qualification at open as the runtime gate |
| A backend runs a step only after it gives the reference's output on qualifying inputs; any call it cannot take runs on the CPU | §11.2 must hold on every fabric; a device or driver can be wrong (§6.1's driver defects) |
| Placement by a measured cost table (µJ per call + nJ per unit), the CPU on a tie or where unmeasured | **[measured]** the GPU's fixed cost per call (1.7 mJ for a hash, 3.1 mJ for an SSE) exceeds what a pack's calls save, while the search saves 14× per unit (§9.3); placing by fabric alone made the floor-only pack dearer (§9.2) |
| The fabric record lives in the measurement claim, not the receipt | the receipt and `LISTING` would otherwise differ by machine, voiding §11.2 for every package; the claim is already §11.2's exemption |
| `fabrics` is absent rather than `null` | every existing claim, and every package that carries one, keeps its bytes |
| Attributions `task` and `rail` only; a `rail` share is the whole rail, sealed only for its sole user | the repository's measurement of a 3–15× overcharge from dividing by busy time (§7.1) |
| The figure is an `OnChipCounter` reading with a declared uncertainty over its span | both readings are counters (energy-measurement §2, §2.9); a counter reading is not a model |
| Hardware AV1 decode for sources and for informative floor decodes; never a floor verdict | AV1 §7.2 (exact without grain); floors state no digest; hardware may conceal errors |
| Hardware frames read from P010 exactly, never through ffmpeg's 16-to-8-bit conversion | **[measured]**: that conversion differs from dav1d's (§3) |
| No hardware encoder computes a WAI step | §6.2 |
| No neural-processor backend | §5 |

## 9. Measurements

**Setup.** A 596 s, 24 fps open-licence film, 14,315 frames: a 256×144 integer proxy and the
ladder's four 10-bit AV1 rungs (1080, 720, 540 and 360 lines). One arm64 desktop (macOS 27.0.1),
shared with other work; each run's load average is in its report. wai-pack at this change,
release, features `gpu,fabric-meter`; ffmpeg 9.0.2 with dav1d 1.5.4. Each run was metered by an
external sampler that read the kernel's account of the child's CPU energy before reaping it (and of
every decode service the run started: hardware decodes run out of process, and a 2,000-frame
decode billed 1.20 J to ffmpeg and 1.09 J to its service), and the IOReport `GPU Energy` rail and
every client's GPU time every 200 ms, with idle brackets before and after.

**Class and scope of every figure below.** CPU figures are `OnChipCounter` readings of the
process's own task energy. GPU figures are `OnChipCounter` readings of the whole GPU rail over the
run, given only where the run held at least 99% of all GPU time; otherwise there is no GPU figure.
The media engine has no counter, so a media-engine run's figure is its CPU side alone, a lower
bound. **Each table is one run.** None was repeated, so none is a median, and none carries a
measured spread; they are indicative.

### 9.1 Exact results

- **Packages are the same bytes on the GPU.** A 360-line rung's policy packed with
  `--fabric gpu`: the CPU pack's digest (`7454acc6…1118`), every file the same (`diff -r`), in
  three runs, and with `--fabric auto:<costs>` too. A four-rung pack over the ladder's own CMAF,
  the base coded once: pack digest `3912bd7c…b9b7f` and receipt `e0256cd9…`, the CPU's. A
  floor-only pack: `f8871444…e032`, the CPU's.
- **The media engine decodes what dav1d decodes.** Each of the four rungs, all 14,315 frames: the
  SHA-256 of every frame in P010 equal between the engine's hardware frames and dav1d's output.
  And `wai-pack floor-decode` of the four-rung package: the same digest from both decoders for each
  of its 600 pieces, in the stream's own 10-bit form, the P010 words read back exactly.
- **A source decoded by the media engine is the same source.** The proxy decoded from the 1080-line
  rung on the engine, then by the same scaler: SHA-256 `8d236f5b…b56d`, the CPU decode's proxy byte
  for byte.
- **Placement.** In a 12-piece pack, 24,915 of 24,930 motion-search calls ran on the GPU before the
  map-wait fix. The other 15 were handed back to the CPU, with the same vectors.
- **Instructions.** The CPU-only 360-line pack retired 98.3 × 10¹² CPU instructions; the GPU pack
  retired 2.05 × 10¹².

### 9.2 Pack and verify, joules per fabric (one run each)

The GPU's other clients held under 1% of its time in every pack of this run.

| Run | Wall | CPU (task) | GPU (rail) | Total | Peak memory | CPU-only baseline (task) |
|---|---:|---:|---:|---:|---:|---|
| 360-line pack, search on the GPU | 56 s | 573 J | 1,284 J | **1,857 J** | 0.95 GB | 20,299 J, 3,689 s, 3.04 GB |
| 360-line pack, `auto` (search on the GPU; SSE and hashes on the CPU) | 56 s | 571 J | 1,283 J | **1,854 J** | 0.76 GB | as above |
| four-rung CMAF pack, search on the GPU, `--jobs 12` | 71 s | 525 J | 1,239 J | **1,763 J** | 1.62 GB | 19,541 J, 980 s at `--jobs 6`, 0.17 GB |
| floor-only pack, hashes on the GPU | 2.2 s | 4.4 J | 2.3 J | **6.7 J** | 0.65 GB | 3.7 J, 1.5 s, 0.04 GB |
| `wai-pack verify`, 360-line, CPU | 32 s | 118 J | 0 | 118 J | 0.58 GB | 31–33 s |
| `wai-pack verify`, 360-line, hashes on the GPU | 31 s | 120 J | none (GPU shared) | ≥ 120 J | 2.04 GB | |
| `wai-pack verify`, four-rung, CPU | 34 s | 119 J | 0 | 119 J | 1.55 GB | 35 s, 121 J |
| `wai-pack verify`, four-rung, hashes on the GPU | 32 s | 122 J | 0.2 J | 122 J | 3.44 GB | |

What the table shows:
- Moving the motion search to the GPU cut the 360-line pack's energy 9 to 11 times over the two
  runs made: 1,857 J here and 2,248 J in the sealed run of §9.5, against a 20,299 J baseline. The
  baseline is CPU task energy alone, and the GPU runs add the whole rail. The CPU work left
  (residuals, rANS, digests, I/O: 116 s of CPU against the baseline's 3,635 s) shows that the
  search was at least 96.8% of the baseline's CPU time.
- Moving the object hashes or SSE gained nothing: the floor-only pack, whose only movable step is
  hashing, cost more on the GPU.
- `wai-pack verify` is dominated by the integer base's decode, which is sequential rANS, so it
  stays on the CPU.

### 9.3 The measured cost table (`wai-pack fabric calibrate`, one run)

Each backend, two call sizes, 6 s each, fitted to `µJ per call + nJ per unit`:

| Step | CPU | GPU | Placement |
|---|---|---|---|
| `base-search` (unit: candidate-sample) | 0 + 20.2 nJ | 11,276 µJ + 1.47 nJ | GPU from about 600,000 units, a 32×24 frame at the policy's range; every frame of the film is 32 million |
| `sse` (unit: byte) | 0 + 0.85 nJ | 3,136 µJ + 0.76 nJ | CPU below about 35 MB a call; a piece of the film is 5 MB |
| `object-hash` (unit: byte) | 7 µJ + 1.33 nJ | 1,699 µJ + 1.43 nJ | CPU always: the GPU's BLAKE3 is no cheaper per byte either |

The GPU's per-call cost includes its idle and static draw over the call, since the rail is the whole
GPU's.

### 9.4 Decoding: CPU against the media engine (one run)

The decode service was counted; the GPU was shared in this run, and no decode used it.

| Decode of the whole floor | dav1d: wall, CPU | media engine: wall, CPU (ffmpeg and its service), engine |
|---|---|---|
| 1080 lines | 23.1 s, 198.7 J | 22.0 s, 17.4 J, + unmetered |
| 720 lines | 12.4 s, 111.0 J | 17.3 s, 11.3 J, + unmetered |
| 540 lines | 7.7 s, 73.3 J | 14.1 s, 8.5 J, + unmetered |
| 360 lines | 5.1 s, 53.8 J | 11.3 s, 15.9 J, + unmetered |
| proxy source (1080 lines decoded and scaled to 256×144) | 22.7 s, 304.1 J | 25.6 s, 350.5 J, + unmetered |

What the table shows:
- A floor decode on the media engine cost the CPU 8 to 11 times less at 540 to 1080 lines, and 3.4
  times less at 360, where the per-frame download dominates.
- The engine's own draw is not visible to an unprivileged reader on macOS 27 (§7.1), so these are
  lower bounds.
- For the proxy source the engine saved nothing: the Lanczos scale and the P010 repack on the CPU
  cost more than the decode saved. Placing `source-decode` on the media engine is worth it only
  where the scaling also moves, and moving it would change the source's bytes unless the scaler is
  exact.

### 9.5 A metered pack, sealed (one run)

- **Sealed.** The 360-line pack with `--fabric gpu --meter fabric`, run when the GPU's other
  clients had held under 0.5% of its time over the preceding 10 s, sealed 2,248.4 J with the fabric
  record in its measurement claim: the CPU's share 475.4 J (`task`) and the GPU's 1,773.0 J
  (`rail`, the pack its sole user); the steps `base-search`, `sse` and `object-hash` on the GPU and
  every other step on the CPU, with no search call handed back to the CPU.
- **Which meter.** That run used the meter's first revision, which sealed the figure as
  `ModelBased`, took the GPU's time at the two ends only, and multiplied the rail by the pack's
  share of GPU time (at least 99%). The meter now seals `OnChipCounter` with the whole rail and
  samples GPU time through the span (§7.1). The run has not been repeated under it, so its figures
  are the first revision's.
- **Verified.** `wai-pack verify` accepts the package, and `pack_verify.py` holds every check of it
  (3622 of 3622), the claim among them.
- **Refused.** The same metered four-rung pack, a minute later, found the GPU shared (39.9 s of its
  50.0 s) and was refused `pack-meter`, as was the calibration then.

### 9.6 Load, and what is indicative

- The host's one-minute load average was 14 to 50 over the runs of §9.2 and 25 to 90 over §9.4.
- The CPU figures are each process's own, by the kernel's attribution, so other load changes them
  only through contention.
- The GPU figures of §9.2 are from a run that held the GPU alone. In the run of §9.4 another process
  held about a third of the GPU's time; the GPU packs then had no GPU figure, and the metered packs
  were refused `pack-meter` ("the GPU was shared: this process had 45.2 s of its 65.0 s, under
  99%").
- Instruction counts and digests are exact.

## 10. Judged unsound

- **Any WAI step on the neural processor** (§5).
- **A hardware encoder for any WAI step:** floors are supplied, the packer refuses to encode one
  (`pack-floor-encode-unsupported`), and no hardware encoder's bytes are reproducible (§6.2).
- **The platform's hardware motion estimation in place of §5.5's search** (§3).
- **Dividing a shared GPU rail by GPU time** (§7.1).
- **Media-engine and neural-processor energy on macOS 27 without privileges:** no live counter
  (§7.1); a step run there makes a metered span refuse rather than guess.
- **A floor verdict from a hardware decoder** (§6.3).
- **ffmpeg's P010-to-8-bit conversion in a comparison** (§3).