wai.video.encode: AVC and HEVC through the encoders a device already has, in the CLI and the packager, each encode's maker in a signed record (ENC) #484

Closed
dcharlot wants to merge 0 commits from wai/encode-avc-hevc into main
Owner

AVC (ISO/IEC 14496-10) and HEVC (ISO/IEC 23008-2) encoding through the encoders a device already has, in the wai CLI and the packager, each encode recorded in a signed record that names the encoder, its version, the device and the driver. WAI writes and vendors no AVC or HEVC encoder.

What is in it

  1. The extension wai/extensions/platform-encode.md, registering the record wai.video.encode.
    • Interfaces, by role (§3). Every encoder is loaded at run time or is an operating-system framework; nothing is linked or shipped.

      • platform: the operating system's media framework, with its hardware media engine or its software encoder.
      • va: a GPU's encode engine through the video acceleration API.
      • gpu-driver: a GPU driver's own encode interface.
      • software: a software AVC library the user obtains separately and names at run time.
      • The phone and TV operating system's codec interface is a documented path (§3.2); no adapter for it is built.
    • The stream's shape (§4.3). A producer refuses to record, and a verifier refuses, anything else:

      • an Annex B byte stream with one access unit per picture;
      • an IDR access unit carrying its parameter sets at exactly the planned pictures, and no other random access point;
      • the same parameter set bytes at every IDR;
      • 8-bit 4:2:0, progressive, one layer, no reordering;
      • a sequence parameter set whose cropped size, profile, level and tier match the record.
    • Resampling. A rung of another size is resampled first by wai.resample.two-tap, a pure function in integers, so every host hands its encoder the same samples.

    • The record (§6, §7). It holds:

      • the interface, the encoder, its version, whether a hardware engine coded the pictures, the device and the driver, and for a library the SHA-256 of the file loaded;
      • the source by digest, the settings and the IDR plan;
      • the stream by digest, with its profile, level and tier;
      • an optional energy figure (§8).

      Verification runs in ten steps, each refusal with its reason. A verifier never re-encodes. No tier and no replicate claim covers the encoded pictures, since no encoder is a pure function of its input.

    • Energy (§8). jwp-receipts §2.4.5's members, plus the counter, the attribution (task, rail or bracket) and the fabrics read. A hardware encode's figure that counts no engine must be Partial.

    • An encode over a package's pieces (§9). Each piece becomes one floor object whose derived_from.renderer names the encoder and version (packaging §7.5). The device and driver are bound through its render settings. The record binds the package receipt, each piece's base and each object. The package itself is unchanged.

  2. wai-rs feature platform_encode (src/platform_encode/):
    • the four adapters (adapter_videotoolbox.rs, adapter_vaapi.rs, adapter_nvenc.rs, adapter_openh264.rs): the only files that name an interface's real symbols;
    • the Annex B reader, the record, the y4m reader and the resampler;
    • the energy counters, the corpus generator, and the platform decoder used for measurement.
  3. The CLI:
    • wai encode avc|hevc <in.y4m> -o out.wai [--encoder auto|platform|va|gpu-driver|software] [--size WxH] [--kbps n] [--idr-every n|--idr-plan f] [--es f] [--record f --key seed] [--meter task|rapl|gpu-board|rapl-and-gpu-board --rounds n];
    • wai encoders (what each interface offers here);
    • wai encode-record verify <record> [--stream es] [--source y4m].
  4. The packager:
    • wai-pack encode-rung --package <dir> --component <id> --source <y4m> --rung <id> --codec avc|hevc --kbps n --encoder <api> --key <seed> --out <dir> writes the stream, one floor object per piece, and the signed record;
    • wai-pack verify-rung checks them against the package.
  5. wai_encode_measure:
    • speed and energy over an odd number of rounds, after one warm-up round;
    • quality (PSNR per plane and 6:1:1, SSIM of luma), decoding with the platform decoder.
  6. The corpus wai/encode-conformance/:
    • Fixture streams. Made once through every interface on the fleet, pinned by SHA-256 and never regenerated.
    • Generated. Built byte for byte from the fixtures with a fixed-seed key: the records, accepted cases, and negative controls, each refused with its reason. Every record reason has at least one case.
    • rendition/. Encoded renditions of the base packaging golden, accepted and mutated.
    • encode_verify.py. The second implementation, written by a separate agent from the extension's text alone, sharing no code with wai-rs or wai-pack.
  7. Text and registration:
    • SPEC §5 paragraph and registry entry;
    • export regenerated;
    • packaging §7.5 cross-reference;
    • CHANGELOG entry and corpus README.
  8. CI:
    • wai-conformance.yml: the module tests, the corpus test and encode_verify.py on linux-arm64;
    • the same file, on macos: the media framework's hardware encoder end to end. On that runner the step fails if that test did not run and pass; a host without the engine fails it, not skips it;
    • wai-pack.yml: tests/encode_rung.rs, with a test-count floor.

Design decisions, and why

  • No encoder of our own. AVC/HEVC essential patents are licensed for royalties. The text states the position by mechanism (§2.2): the maker of the device supplies the platform's and the GPU driver's encoders; the user obtains the library from its distributor. The record says who made the bytes, never whether a use is licensed.
  • One shape for every encoder's output. A package's pieces need each piece to start at a self-contained IDR access unit, and access unit i to be frame i. So §4.3 is checked at production, before signing, and again by every verifier.
    • It caught a real case: the media framework's software AVC encoder inserts an IDR at picture 15 that no plan names, and is refused (encode-keyframe-mismatch). Its HEVC encoder, and every hardware path, hold.
  • The rendition sits beside the package, not inside it. On main the packer packs exactly one floor rung (AV1). Listing AVC/HEVC rungs in the stage index, catalogs and CMAF belongs on the several-floor packer of the open packaging train (#465), and would conflict with it here. §9 binds the rendition to a package by the package receipt's digest and each piece's base. The package's own verification, and its §11 re-pack equivalence, are untouched.
  • Header syntax under the video acceleration API (§2.3).
    • Both drivers reached emit no parameter sets unless the application supplies packed headers; one also emits a zeroed slice NAL header. So the adapter writes the VPS/SPS/PPS and the slice headers from the values it passes the driver, and the driver codes all slice data.
    • One driver's own rate control coded slices at a QP other than the header's: the stream parsed but decoded to the wrong pictures (12–19 dB). So where a driver offers constant QP, the adapter picks each picture's QP and writes it in both places.
    • This needs an owner decision (below).
  • Energy, as each host allows (§8.2).
    • On macOS: the per-task counter, with work billed to the process. It counts none of the media engine, for which an unprivileged process has no counter, so every hardware figure there is Partial and covers the CPU side only.
    • On Linux: energy-measurement §3's bracketed marginal, over an idle window just before each round.
    • The median of five rounds, after one unmetered warm-up, with the rounds' spread declared. A round with no marginal leaves the figure absent rather than estimated.
  • The second implementation drove the text. Its first pass raised 16 questions: check order, sizes, a clause that could never fire, and source geometry. Each is now settled in the text, in wai-rs and in the corpus. The cases it found unisolated were added.

Hardware each path ran on

interface device (by role) codecs driver / version, as the interface reported
platform the build host's media engine (arm64 macOS 27, 24+8 cores) AVC, HEVC (hardware); HEVC (its software encoder) framework bundle 3350.77.5.6
va an x86-64 APU's integrated GPU AVC High, HEVC Main (EncSlice) libva 1.23, the open graphics stack's VA driver 26.0.8
va an x86-64 laptop processor's integrated GPU AVC High, HEVC Main (EncSliceLP) libva 1.20, the vendor's media driver 24.1.0
gpu-driver a discrete laptop GPU AVC High, HEVC Main (preset P5) encode API 12.0 (driver supports 13.0), kernel module 595.99.02
software arm64 macOS, and both x86-64 Linux hosts AVC High library 2.7.0. The library made the same bytes on all three hosts; nothing relies on that.

Not reached:

  • The x86-64 laptop's integrated GPU through its vendor's own media SDK runtime. The dispatcher is installed but no GPU runtime is, so that GPU was reached through the video acceleration API instead.
  • The phone/TV codec interface. Documented only.
  • iOS and tvOS devices. The adapter compiles for the platform but was not run on a device.
  • Windows. No adapter.

Measured

Every run used the same source: 300 frames, 1920×1080 at 30 fps, from an openly licensed animated film, starting at 60 s (SHA-256 b38d4098…753f).

  • Rungs and plan. 1080p, 720p and 360p, with an IDR every 2 s.
  • Rounds. Five rounds after one warm-up. fps is the median round's.
  • Energy figures. All OnChipCounter, filtering Unknown, Partial. "u" is the declared relative uncertainty.
  • Quality. Measured against the frames the encoder was handed, decoded by the build host's platform decoder. VMAF (default model, every other frame) is at 1080p only.
  • Decoder check. Every stream was also decoded by an independent software decoder, and the two decoders' pictures were byte-identical for every stream (StandardDefined).

The build host (shared: load 35–39 during the runs). Counter task, attribution task, fabrics cpu. The media engine is not counted.

rung interface codec engine kb/s asked → made fps mJ/frame (u) PSNR-Y SSIM-Y VMAF
1080p platform AVC hw 6000 → 6082 642 0.74 (2.4%) 46.48 0.9941 92.16
1080p platform HEVC hw 4000 → 4146 603 0.61 (8.7%) 46.86 0.9929 92.17
1080p software AVC sw 6000 → 4753 156 20.72 (1.7%) 45.17 0.9897 90.65
1080p platform HEVC sw 4000 → 4058 10.7 1242 (1.1%) 47.31 0.9940 92.32
1080p platform AVC sw refused at §4.3: an IDR at picture 15 the plan does not name
720p platform AVC hw 3000 → 3402 1115 0.66 (18.6%) 45.56 0.9931
720p platform HEVC hw 2000 → 2161 967 0.33 (27.4%) 45.22 0.9909
720p software AVC sw 3000 → 2428 428 9.85 (1.4%) 43.65 0.9877
360p platform AVC hw 800 → 963 2176 0.52 (4.7%) 40.71 0.9878
360p platform HEVC hw 550 → 715 2563 0.50 (0.8%) 41.02 0.9857
360p software AVC sw 800 → 734 1622 2.73 (1.3%) 40.04 0.9815

The x86-64 APU host (idle: load 0.2–0.5). Counter rapl (the package zones), attribution bracket, fabrics stated as cpu only. Whether this part's package counter includes its integrated GPU is not documented here, so no more is claimed.

rung interface codec kb/s asked → made fps mJ/frame (u) PSNR-Y SSIM-Y VMAF
1080p va AVC 6000 → 6043 755 32.76 (10.8%) 48.22 0.9943 96.69
1080p va HEVC 4000 → 4054 761 32.16 (4.4%) 47.28 0.9934 96.02
720p va AVC 3000 → 3027 1538 18.55 (2.2%) 46.31 0.9923
720p va HEVC 2000 → 2032 1458 19.02 (4.6%) 45.65 0.9916
360p va AVC 800 → 812 3806 8.98 (1.4%) 40.67 0.9848
360p va HEVC 550 → 560 3554 8.61 (7.8%) 39.67 0.9822
1080p software AVC 6000 → 4753 251 82.80 (5.2%) 45.17 0.9897 90.65
720p software AVC 3000 → 2428 549 37.60 (5.5%) 43.65 0.9877

The x86-64 laptop with integrated and discrete GPUs. It is a CI runner: two CI containers ran throughout, at load 4.6–12.7. Its machine-wide counters were read honestly, and many bracket rounds were refused ("no marginal") as a result. The discrete GPU ran 900 frames per round (the clip three times) because its counter is coarse. Its quality comes from a separate 300-frame encode at the same settings, so its "made" rate is that encode's.

rung interface codec kb/s asked → made fps mJ/frame (u), counter PSNR-Y SSIM-Y VMAF
1080p va AVC 6000 → 6026 501 13.71 (94.1%), rapl 48.33 0.9939 97.00
1080p va HEVC 4000 → 4041 551 unmetered (no marginal) 47.15 0.9930 96.21
720p va AVC 3000 → 3030 1072 5.25 (33.0%), rapl 46.12 0.9917
720p va HEVC 2000 → 2020 1065 5.64 (21.3%), rapl 45.79 0.9915
360p va AVC 800 → 823 3923 unmetered 40.67 0.9844
360p va HEVC 550 → 562 4225 unmetered 39.71 0.9821
1080p gpu-driver AVC 6000 → 6201 245 unmetered 50.59 0.9961 93.73
1080p gpu-driver HEVC 4000 → 4825 337 unmetered 48.67 0.9945 92.96
720p gpu-driver AVC 3000 → 3137 517 19.34 (29.6%), rapl + board 48.56 0.9947
720p gpu-driver HEVC 2000 → 2264 675 unmetered 46.81 0.9928
360p gpu-driver AVC 800 → 819 1513 12.97 (67.3%), rapl + board 43.63 0.9898
360p gpu-driver HEVC 550 → 587 1478 15.23 (30.0%), rapl + board 41.53 0.9855
1080p software AVC 6000 → 4753 178 54.90 (43.3%), rapl 45.17 0.9897
  • The board counter. The discrete GPU's board counter alone gave 42.3 mJ/frame (34.8%) at 1080p AVC. On this host that driver's own power reading shows 588 W at idle on a part rated far lower, so its energy counter is not trusted here; the figure is reported, not relied on.
  • What the figures are. A figure is of the encoder, the host, its load and its counter at the time. None is compared with a figure taken under another counter or attribution.

Checks

Portable runs went to the x86-64 APU host (RB_HOST=…47 rb). The platform encoders and macOS runs went to the build host (rb). Results are at the head unless a row says otherwise.

check where result
cargo test --locked --no-default-features --features platform_encode --lib platform_encode:: APU host 15 passed
… --test platform_encode APU host / build host 4 passed / 5 passed (the hardware encode included)
cargo test --locked --no-default-features --lib codecs::; wai registry export --check APU host 53 passed; up to date
wai_encode_vectors verify ../encode-conformance APU host PASS: every fixture pinned, every generated file byte for byte, 62 cases with their outcomes
wai-pack: cargo test --release --locked; rendition-vectors check APU host all passed (encode_rung 3); PASS, 10 cases
cargo clippy --locked --no-default-features --features platform_encode --all-targets APU host (Linux adapters), build host (macOS adapter) no note in any new file on either
wai-pack: cargo clippy --locked --all-targets -- -D warnings APU host clean
cargo metadata --locked --offline (wai-rs, wai-pack) APU host ok: no dependency added, no lockfile changed
python3 -I encode-conformance/encode_verify.py encode-conformance APU host PASS: 62/62 and 10/10 cases; every record reason; 119 mutations, each with its outcome; 54 checks dropped in turn, each caught
every other Python verifier the two workflows run (with the pinned requirements-verify.txt venv) APU host every one exits 0. Not run here: the ones that need a built packager binary, Node, or a per-run seed (the *_diff.py and differential scripts, ad_presentation_verify.py, dred_verify.py, the quantum referee)
the adapters' own smoke runs (1080p and 720p, and 540p on the two VA hosts; IDR every 30, every 37, and only at 0) each GPU host every stream passed §4.3; decoded with no error, at the right frame count
check-tests-in-ci.py, check-workflow-concurrency.sh, check-spec-refs.py, check-wai-section-refs.py, check-no-vanity-metrics.sh, test-public-text-checks.sh, check-comparative-prose.sh, check-orphan-sources.py local all exit 0
check-public-names.sh over every changed file under wai/; commit messages against the patterns local clean; no hit
after merging origin/main (cd35b8ff9): wfsteps.py HEAD <merge-base> HEAD^1 origin/main; ruby -ryaml over every workflow local 0 problem(s); all parse
CI on this PR forge every run cancelled on opening, as asked

Falsifications

Everything was committed first. Each rule was broken alone, the guarding test was run, and the file was restored from git; git diff --quiet was clean after each.

rule broken guarding test result
an IDR at exactly the plan tests/platform_encode.rs fails: stream-avc-idr-missing refused for another reason
picture order increasing (no reordering) same fails: stream-avc-swapped accepted
every IDR carries every parameter set same not caught at first: the byte-equality rule refused the case too. Added stream-avc-no-pps (no PPS anywhere); now fails
the single-layer rule same fails: stream-hevc-layer accepted
one access unit per picture same fails: stream-avc-middle-dropped refused for another reason
trailing NAL units with no slice same fails: stream-avc-trailing-sei refused for another reason
a hardware figure counting no engine must be Partial same fails: energy-whole-without-engine accepted
the signature same fails: bad-signature accepted
the held source's geometry same fails: source-other-frames accepted
§12's table equals the code's reasons same (the_reasons_table_is_the_references) fails
the rendition's package receipt wai-pack tests/encode_rung.rs fails: receipt-other accepted
the rendition's object hash same fails: object-misnamed accepted

encode_verify.py drops each of its own checks in turn: every one is caught, all but four by the committed corpus. Those four (a source side past 16384; a non-zero reorder field; an AVC VUI range that differs from the rung's; a stage-index span that differs from the record's) are caught only by its mutations. The first three need SPS bit rewriting; the last needs a mutated package.

Not done

  • AVC/HEVC rungs inside a package: stage-index entries, catalogs, and CMAF with avcC/hvcC sample entries. These wait for the several-floor packer (#465). §9's rendition sits beside the package until then.
  • What the encode settings cover. 10-bit, 4:2:2, B-pictures and reordering, HDR signalling, and multiple slices per picture on the VA path are outside this revision's settings.
  • Adapters and devices.
    • The phone/TV codec interface: documented, not built.
    • iOS and tvOS: not run on a device.
    • Windows: no adapter.
    • The laptop's integrated GPU through its vendor SDK runtime: not installed, so it was reached through VA-API.
  • CI on Linux GPU hardware. No CI job runs the VA or GPU-driver adapters; they were run by hand. The linux-hw runner's host has both GPUs.
  • Energy:
    • no counter on macOS covers the media engine;
    • on the shared Linux runner most bracket rounds were refused under CI load;
    • the discrete GPU's board counter is not trusted on that host;
    • wai-pack encode-rung writes no energy figure: the CLI's --meter has no counterpart there yet.
  • The VA adapter's QP controller is a small model-based one, used only because a driver's own rate control disagrees with the slice header. Its rate held within 2.1% and 4.2% of target on the two drivers.

Needs an owner decision

  1. The runtime-loaded AVC library's terms. The distributor covers the royalties for its binary only under conditions:

    • the binary is downloaded separately to the end user's device;
    • the user can switch it on and off;
    • the software using it displays a fixed attribution that names the distributor, where the user enables it, and reproduces the licence text in its licensing information;
    • and the coverage is for uses without remuneration.

    The public-text rule forbids that attribution, so the CLI does not print it. Decide whether to allow that one string as an allow-name exception, or to treat the software path as bring-your-own-licence or test-only. Commercial content providers need their own AVC licence either way.

  2. Header syntax and QP under the video acceleration API (§2.3). The adapter writes parameter sets and slice headers, and picks each picture's QP; the driver codes every slice's data. Confirm this is "the platform's encoder" under the no-encoder-of-our-own rule.

  3. Shipping platform_encode in release builds of wai. It adds no link dependency, but turns on the software path's terms above.

🤖 Generated with Claude Code


Closed: Landed on main through merge train E (#499, merge bf39ec9a); verified by content: this PR's head is in main

AVC (ISO/IEC 14496-10) and HEVC (ISO/IEC 23008-2) encoding through the encoders a device already has, in the `wai` CLI and the packager, each encode recorded in a signed record that names the encoder, its version, the device and the driver. WAI writes and vendors no AVC or HEVC encoder. ## What is in it 1. **The extension** `wai/extensions/platform-encode.md`, registering the record `wai.video.encode`. - **Interfaces, by role** (§3). Every encoder is loaded at run time or is an operating-system framework; nothing is linked or shipped. - `platform`: the operating system's media framework, with its hardware media engine or its software encoder. - `va`: a GPU's encode engine through the video acceleration API. - `gpu-driver`: a GPU driver's own encode interface. - `software`: a software AVC library the user obtains separately and names at run time. - The phone and TV operating system's codec interface is a documented path (§3.2); no adapter for it is built. - **The stream's shape** (§4.3). A producer refuses to record, and a verifier refuses, anything else: - an Annex B byte stream with one access unit per picture; - an IDR access unit carrying its parameter sets at exactly the planned pictures, and no other random access point; - the same parameter set bytes at every IDR; - 8-bit 4:2:0, progressive, one layer, no reordering; - a sequence parameter set whose cropped size, profile, level and tier match the record. - **Resampling.** A rung of another size is resampled first by `wai.resample.two-tap`, a pure function in integers, so every host hands its encoder the same samples. - **The record** (§6, §7). It holds: - the interface, the encoder, its version, whether a hardware engine coded the pictures, the device and the driver, and for a library the SHA-256 of the file loaded; - the source by digest, the settings and the IDR plan; - the stream by digest, with its profile, level and tier; - an optional energy figure (§8). Verification runs in ten steps, each refusal with its reason. A verifier never re-encodes. No tier and no `replicate` claim covers the encoded pictures, since no encoder is a pure function of its input. - **Energy** (§8). jwp-receipts §2.4.5's members, plus the counter, the attribution (`task`, `rail` or `bracket`) and the fabrics read. A hardware encode's figure that counts no engine must be `Partial`. - **An encode over a package's pieces** (§9). Each piece becomes one floor object whose `derived_from.renderer` names the encoder and version (packaging §7.5). The device and driver are bound through its render settings. The record binds the package receipt, each piece's base and each object. The package itself is unchanged. 2. **`wai-rs` feature `platform_encode`** (`src/platform_encode/`): - the four adapters (`adapter_videotoolbox.rs`, `adapter_vaapi.rs`, `adapter_nvenc.rs`, `adapter_openh264.rs`): the only files that name an interface's real symbols; - the Annex B reader, the record, the y4m reader and the resampler; - the energy counters, the corpus generator, and the platform decoder used for measurement. 3. **The CLI:** - `wai encode avc|hevc <in.y4m> -o out.wai [--encoder auto|platform|va|gpu-driver|software] [--size WxH] [--kbps n] [--idr-every n|--idr-plan f] [--es f] [--record f --key seed] [--meter task|rapl|gpu-board|rapl-and-gpu-board --rounds n]`; - `wai encoders` (what each interface offers here); - `wai encode-record verify <record> [--stream es] [--source y4m]`. 4. **The packager:** - `wai-pack encode-rung --package <dir> --component <id> --source <y4m> --rung <id> --codec avc|hevc --kbps n --encoder <api> --key <seed> --out <dir>` writes the stream, one floor object per piece, and the signed record; - `wai-pack verify-rung` checks them against the package. 5. **`wai_encode_measure`**: - speed and energy over an odd number of rounds, after one warm-up round; - quality (PSNR per plane and 6:1:1, SSIM of luma), decoding with the platform decoder. 6. **The corpus** `wai/encode-conformance/`: - **Fixture streams.** Made once through every interface on the fleet, pinned by SHA-256 and never regenerated. - **Generated.** Built byte for byte from the fixtures with a fixed-seed key: the records, accepted cases, and negative controls, each refused with its reason. Every record reason has at least one case. - **`rendition/`.** Encoded renditions of the base packaging golden, accepted and mutated. - **`encode_verify.py`.** The second implementation, written by a separate agent from the extension's text alone, sharing no code with `wai-rs` or `wai-pack`. 7. **Text and registration:** - SPEC §5 paragraph and registry entry; - export regenerated; - packaging §7.5 cross-reference; - CHANGELOG entry and corpus README. 8. **CI:** - `wai-conformance.yml`: the module tests, the corpus test and `encode_verify.py` on `linux-arm64`; - the same file, on `macos`: the media framework's hardware encoder end to end. On that runner the step fails if that test did not run and pass; a host without the engine fails it, not skips it; - `wai-pack.yml`: `tests/encode_rung.rs`, with a test-count floor. ## Design decisions, and why - **No encoder of our own.** AVC/HEVC essential patents are licensed for royalties. The text states the position by mechanism (§2.2): the maker of the device supplies the platform's and the GPU driver's encoders; the user obtains the library from its distributor. The record says who made the bytes, never whether a use is licensed. - **One shape for every encoder's output.** A package's pieces need each piece to start at a self-contained IDR access unit, and access unit *i* to be frame *i*. So §4.3 is checked at production, before signing, and again by every verifier. - It caught a real case: the media framework's **software** AVC encoder inserts an IDR at picture 15 that no plan names, and is refused (`encode-keyframe-mismatch`). Its HEVC encoder, and every hardware path, hold. - **The rendition sits beside the package, not inside it.** On main the packer packs exactly one floor rung (AV1). Listing AVC/HEVC rungs in the stage index, catalogs and CMAF belongs on the several-floor packer of the open packaging train (#465), and would conflict with it here. §9 binds the rendition to a package by the package receipt's digest and each piece's base. The package's own verification, and its §11 re-pack equivalence, are untouched. - **Header syntax under the video acceleration API** (§2.3). - Both drivers reached emit no parameter sets unless the application supplies packed headers; one also emits a zeroed slice NAL header. So the adapter writes the VPS/SPS/PPS and the slice headers from the values it passes the driver, and the driver codes all slice data. - One driver's own rate control coded slices at a QP other than the header's: the stream parsed but decoded to the wrong pictures (12–19 dB). So where a driver offers constant QP, the adapter picks each picture's QP and writes it in both places. - This needs an owner decision (below). - **Energy, as each host allows** (§8.2). - On macOS: the per-task counter, with work billed to the process. It counts none of the media engine, for which an unprivileged process has no counter, so every hardware figure there is `Partial` and covers the CPU side only. - On Linux: energy-measurement §3's bracketed marginal, over an idle window just before each round. - The median of five rounds, after one unmetered warm-up, with the rounds' spread declared. A round with no marginal leaves the figure absent rather than estimated. - **The second implementation drove the text.** Its first pass raised 16 questions: check order, sizes, a clause that could never fire, and source geometry. Each is now settled in the text, in `wai-rs` and in the corpus. The cases it found unisolated were added. ## Hardware each path ran on | interface | device (by role) | codecs | driver / version, as the interface reported | |---|---|---|---| | `platform` | the build host's media engine (arm64 macOS 27, 24+8 cores) | AVC, HEVC (hardware); HEVC (its software encoder) | framework bundle 3350.77.5.6 | | `va` | an x86-64 APU's integrated GPU | AVC High, HEVC Main (`EncSlice`) | libva 1.23, the open graphics stack's VA driver 26.0.8 | | `va` | an x86-64 laptop processor's integrated GPU | AVC High, HEVC Main (`EncSliceLP`) | libva 1.20, the vendor's media driver 24.1.0 | | `gpu-driver` | a discrete laptop GPU | AVC High, HEVC Main (preset P5) | encode API 12.0 (driver supports 13.0), kernel module 595.99.02 | | `software` | arm64 macOS, and both x86-64 Linux hosts | AVC High | library 2.7.0. The library made the same bytes on all three hosts; nothing relies on that. | Not reached: - **The x86-64 laptop's integrated GPU through its vendor's own media SDK runtime.** The dispatcher is installed but no GPU runtime is, so that GPU was reached through the video acceleration API instead. - **The phone/TV codec interface.** Documented only. - **iOS and tvOS devices.** The adapter compiles for the platform but was not run on a device. - **Windows.** No adapter. ## Measured Every run used the same source: 300 frames, 1920×1080 at 30 fps, from an openly licensed animated film, starting at 60 s (SHA-256 `b38d4098…753f`). - **Rungs and plan.** 1080p, 720p and 360p, with an IDR every 2 s. - **Rounds.** Five rounds after one warm-up. fps is the median round's. - **Energy figures.** All `OnChipCounter`, filtering `Unknown`, `Partial`. "u" is the declared relative uncertainty. - **Quality.** Measured against the frames the encoder was handed, decoded by the build host's platform decoder. VMAF (default model, every other frame) is at 1080p only. - **Decoder check.** Every stream was also decoded by an independent software decoder, and the two decoders' pictures were **byte-identical for every stream** (`StandardDefined`). **The build host** (shared: load 35–39 during the runs). Counter `task`, attribution `task`, fabrics `cpu`. The media engine is not counted. | rung | interface | codec | engine | kb/s asked → made | fps | mJ/frame (u) | PSNR-Y | SSIM-Y | VMAF | |---|---|---|---|---:|---:|---:|---:|---:|---:| | 1080p | platform | AVC | hw | 6000 → 6082 | 642 | 0.74 (2.4%) | 46.48 | 0.9941 | 92.16 | | 1080p | platform | HEVC | hw | 4000 → 4146 | 603 | 0.61 (8.7%) | 46.86 | 0.9929 | 92.17 | | 1080p | software | AVC | sw | 6000 → 4753 | 156 | 20.72 (1.7%) | 45.17 | 0.9897 | 90.65 | | 1080p | platform | HEVC | sw | 4000 → 4058 | 10.7 | 1242 (1.1%) | 47.31 | 0.9940 | 92.32 | | 1080p | platform | AVC | sw | refused at §4.3: an IDR at picture 15 the plan does not name | | | | | | | 720p | platform | AVC | hw | 3000 → 3402 | 1115 | 0.66 (18.6%) | 45.56 | 0.9931 | | | 720p | platform | HEVC | hw | 2000 → 2161 | 967 | 0.33 (27.4%) | 45.22 | 0.9909 | | | 720p | software | AVC | sw | 3000 → 2428 | 428 | 9.85 (1.4%) | 43.65 | 0.9877 | | | 360p | platform | AVC | hw | 800 → 963 | 2176 | 0.52 (4.7%) | 40.71 | 0.9878 | | | 360p | platform | HEVC | hw | 550 → 715 | 2563 | 0.50 (0.8%) | 41.02 | 0.9857 | | | 360p | software | AVC | sw | 800 → 734 | 1622 | 2.73 (1.3%) | 40.04 | 0.9815 | | **The x86-64 APU host** (idle: load 0.2–0.5). Counter `rapl` (the package zones), attribution `bracket`, fabrics stated as `cpu` only. Whether this part's package counter includes its integrated GPU is not documented here, so no more is claimed. | rung | interface | codec | kb/s asked → made | fps | mJ/frame (u) | PSNR-Y | SSIM-Y | VMAF | |---|---|---|---:|---:|---:|---:|---:|---:| | 1080p | va | AVC | 6000 → 6043 | 755 | 32.76 (10.8%) | 48.22 | 0.9943 | 96.69 | | 1080p | va | HEVC | 4000 → 4054 | 761 | 32.16 (4.4%) | 47.28 | 0.9934 | 96.02 | | 720p | va | AVC | 3000 → 3027 | 1538 | 18.55 (2.2%) | 46.31 | 0.9923 | | | 720p | va | HEVC | 2000 → 2032 | 1458 | 19.02 (4.6%) | 45.65 | 0.9916 | | | 360p | va | AVC | 800 → 812 | 3806 | 8.98 (1.4%) | 40.67 | 0.9848 | | | 360p | va | HEVC | 550 → 560 | 3554 | 8.61 (7.8%) | 39.67 | 0.9822 | | | 1080p | software | AVC | 6000 → 4753 | 251 | 82.80 (5.2%) | 45.17 | 0.9897 | 90.65 | | 720p | software | AVC | 3000 → 2428 | 549 | 37.60 (5.5%) | 43.65 | 0.9877 | | **The x86-64 laptop with integrated and discrete GPUs.** It is a CI runner: two CI containers ran throughout, at load 4.6–12.7. Its machine-wide counters were read honestly, and many bracket rounds were refused ("no marginal") as a result. The discrete GPU ran 900 frames per round (the clip three times) because its counter is coarse. Its quality comes from a separate 300-frame encode at the same settings, so its "made" rate is that encode's. | rung | interface | codec | kb/s asked → made | fps | mJ/frame (u), counter | PSNR-Y | SSIM-Y | VMAF | |---|---|---|---:|---:|---|---:|---:|---:| | 1080p | va | AVC | 6000 → 6026 | 501 | 13.71 (94.1%), rapl | 48.33 | 0.9939 | 97.00 | | 1080p | va | HEVC | 4000 → 4041 | 551 | unmetered (no marginal) | 47.15 | 0.9930 | 96.21 | | 720p | va | AVC | 3000 → 3030 | 1072 | 5.25 (33.0%), rapl | 46.12 | 0.9917 | | | 720p | va | HEVC | 2000 → 2020 | 1065 | 5.64 (21.3%), rapl | 45.79 | 0.9915 | | | 360p | va | AVC | 800 → 823 | 3923 | unmetered | 40.67 | 0.9844 | | | 360p | va | HEVC | 550 → 562 | 4225 | unmetered | 39.71 | 0.9821 | | | 1080p | gpu-driver | AVC | 6000 → 6201 | 245 | unmetered | 50.59 | 0.9961 | 93.73 | | 1080p | gpu-driver | HEVC | 4000 → 4825 | 337 | unmetered | 48.67 | 0.9945 | 92.96 | | 720p | gpu-driver | AVC | 3000 → 3137 | 517 | 19.34 (29.6%), rapl + board | 48.56 | 0.9947 | | | 720p | gpu-driver | HEVC | 2000 → 2264 | 675 | unmetered | 46.81 | 0.9928 | | | 360p | gpu-driver | AVC | 800 → 819 | 1513 | 12.97 (67.3%), rapl + board | 43.63 | 0.9898 | | | 360p | gpu-driver | HEVC | 550 → 587 | 1478 | 15.23 (30.0%), rapl + board | 41.53 | 0.9855 | | | 1080p | software | AVC | 6000 → 4753 | 178 | 54.90 (43.3%), rapl | 45.17 | 0.9897 | | - **The board counter.** The discrete GPU's board counter alone gave 42.3 mJ/frame (34.8%) at 1080p AVC. On this host that driver's own power reading shows 588 W at idle on a part rated far lower, so its energy counter is not trusted here; the figure is reported, not relied on. - **What the figures are.** A figure is of the encoder, the host, its load and its counter at the time. None is compared with a figure taken under another counter or attribution. ## Checks Portable runs went to the x86-64 APU host (`RB_HOST=…47 rb`). The platform encoders and macOS runs went to the build host (`rb`). Results are at the head unless a row says otherwise. | check | where | result | |---|---|---| | `cargo test --locked --no-default-features --features platform_encode --lib platform_encode::` | APU host | 15 passed | | `… --test platform_encode` | APU host / build host | 4 passed / 5 passed (the hardware encode included) | | `cargo test --locked --no-default-features --lib codecs::`; `wai registry export --check` | APU host | 53 passed; up to date | | `wai_encode_vectors verify ../encode-conformance` | APU host | PASS: every fixture pinned, every generated file byte for byte, 62 cases with their outcomes | | `wai-pack`: `cargo test --release --locked`; `rendition-vectors check` | APU host | all passed (`encode_rung` 3); PASS, 10 cases | | `cargo clippy --locked --no-default-features --features platform_encode --all-targets` | APU host (Linux adapters), build host (macOS adapter) | no note in any new file on either | | `wai-pack`: `cargo clippy --locked --all-targets -- -D warnings` | APU host | clean | | `cargo metadata --locked --offline` (wai-rs, wai-pack) | APU host | ok: no dependency added, no lockfile changed | | `python3 -I encode-conformance/encode_verify.py encode-conformance` | APU host | PASS: 62/62 and 10/10 cases; every record reason; 119 mutations, each with its outcome; 54 checks dropped in turn, each caught | | every other Python verifier the two workflows run (with the pinned `requirements-verify.txt` venv) | APU host | every one exits 0. Not run here: the ones that need a built packager binary, Node, or a per-run seed (the `*_diff.py` and differential scripts, `ad_presentation_verify.py`, `dred_verify.py`, the quantum referee) | | the adapters' own smoke runs (1080p and 720p, and 540p on the two VA hosts; IDR every 30, every 37, and only at 0) | each GPU host | every stream passed §4.3; decoded with no error, at the right frame count | | `check-tests-in-ci.py`, `check-workflow-concurrency.sh`, `check-spec-refs.py`, `check-wai-section-refs.py`, `check-no-vanity-metrics.sh`, `test-public-text-checks.sh`, `check-comparative-prose.sh`, `check-orphan-sources.py` | local | all exit 0 | | `check-public-names.sh` over every changed file under `wai/`; commit messages against the patterns | local | clean; no hit | | after merging `origin/main` (cd35b8ff9): `wfsteps.py HEAD <merge-base> HEAD^1 origin/main`; `ruby -ryaml` over every workflow | local | `0 problem(s)`; all parse | | CI on this PR | forge | every run cancelled on opening, as asked | ## Falsifications Everything was committed first. Each rule was broken alone, the guarding test was run, and the file was restored from git; `git diff --quiet` was clean after each. | rule broken | guarding test | result | |---|---|---| | an IDR at exactly the plan | `tests/platform_encode.rs` | fails: `stream-avc-idr-missing` refused for another reason | | picture order increasing (no reordering) | same | fails: `stream-avc-swapped` accepted | | every IDR carries every parameter set | same | **not caught at first**: the byte-equality rule refused the case too. Added `stream-avc-no-pps` (no PPS anywhere); now fails | | the single-layer rule | same | fails: `stream-hevc-layer` accepted | | one access unit per picture | same | fails: `stream-avc-middle-dropped` refused for another reason | | trailing NAL units with no slice | same | fails: `stream-avc-trailing-sei` refused for another reason | | a hardware figure counting no engine must be `Partial` | same | fails: `energy-whole-without-engine` accepted | | the signature | same | fails: `bad-signature` accepted | | the held source's geometry | same | fails: `source-other-frames` accepted | | §12's table equals the code's reasons | same (`the_reasons_table_is_the_references`) | fails | | the rendition's package receipt | `wai-pack tests/encode_rung.rs` | fails: `receipt-other` accepted | | the rendition's object hash | same | fails: `object-misnamed` accepted | `encode_verify.py` drops each of its own checks in turn: every one is caught, all but four by the committed corpus. Those four (a source side past 16384; a non-zero reorder field; an AVC VUI range that differs from the rung's; a stage-index span that differs from the record's) are caught only by its mutations. The first three need SPS bit rewriting; the last needs a mutated package. ## Not done - **AVC/HEVC rungs inside a package**: stage-index entries, catalogs, and CMAF with `avcC`/`hvcC` sample entries. These wait for the several-floor packer (#465). §9's rendition sits beside the package until then. - **What the encode settings cover.** 10-bit, 4:2:2, B-pictures and reordering, HDR signalling, and multiple slices per picture on the VA path are outside this revision's settings. - **Adapters and devices.** - The phone/TV codec interface: documented, not built. - iOS and tvOS: not run on a device. - Windows: no adapter. - The laptop's integrated GPU through its vendor SDK runtime: not installed, so it was reached through VA-API. - **CI on Linux GPU hardware.** No CI job runs the VA or GPU-driver adapters; they were run by hand. The `linux-hw` runner's host has both GPUs. - **Energy:** - no counter on macOS covers the media engine; - on the shared Linux runner most bracket rounds were refused under CI load; - the discrete GPU's board counter is not trusted on that host; - `wai-pack encode-rung` writes no energy figure: the CLI's `--meter` has no counterpart there yet. - **The VA adapter's QP controller** is a small model-based one, used only because a driver's own rate control disagrees with the slice header. Its rate held within 2.1% and 4.2% of target on the two drivers. ## Needs an owner decision 1. **The runtime-loaded AVC library's terms.** The distributor covers the royalties for its binary only under conditions: - the binary is downloaded separately to the end user's device; - the user can switch it on and off; - the software using it displays a fixed attribution that names the distributor, where the user enables it, and reproduces the licence text in its licensing information; - and the coverage is for uses without remuneration. The public-text rule forbids that attribution, so the CLI does not print it. Decide whether to allow that one string as an `allow-name` exception, or to treat the software path as bring-your-own-licence or test-only. Commercial content providers need their own AVC licence either way. 2. **Header syntax and QP under the video acceleration API** (§2.3). The adapter writes parameter sets and slice headers, and picks each picture's QP; the driver codes every slice's data. Confirm this is "the platform's encoder" under the no-encoder-of-our-own rule. 3. **Shipping `platform_encode` in release builds of `wai`.** It adds no link dependency, but turns on the software path's terms above. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- **Closed:** Landed on main through merge train E (#499, merge bf39ec9a); verified by content: this PR's head is in main
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
encode corpus: a stream with no picture parameter set, which only the every-parameter-set rule refuses
Some checks failed
WAI HTTP delivery / HTTP corpus · falsify patches 0/3 (pull_request) Has been cancelled
WAI HTTP delivery / HTTP corpus · falsify patches 1/3 (pull_request) Has been cancelled
WAI HTTP delivery / HTTP corpus · falsify patches 2/3 (pull_request) Has been cancelled
WAI HTTP delivery / HTTP corpus · every patch and mutation in exactly one shard (pull_request) Has been cancelled
WAI HTTP delivery / HTTP corpus · verified both ways, and the differential (pull_request) Has been cancelled
WAI HTTP delivery / cache rules · controls and labelled caches (pull_request) Has been cancelled
WAI energy receipts / does this change touch it (pull_request) Has been cancelled
WAI energy receipts / energy receipts · linux (pull_request) Has been cancelled
WAI energy receipts / energy receipts · linux-arm64 (pull_request) Has been cancelled
WAI energy receipts / energy receipts · macos (pull_request) Has been cancelled
WAI conformance / does this change touch it (pull_request) Has been cancelled
WAI conformance / cargo-class corpora · linux (pull_request) Has been cancelled
WAI conformance / cargo-class corpora · linux-arm64 (pull_request) Has been cancelled
WAI conformance / cargo-class corpora · macos (pull_request) Has been cancelled
WAI conformance / quantum core and corpus under wasm32-wasip2 (wasmtime) (pull_request) Has been cancelled
WAI conformance / v2 vocabulary · manifests validate as stated (pull_request) Has been cancelled
WAI conformance / independent verifiers · claims, JWP, key discovery and integer payloads (Python) (pull_request) Has been cancelled
WAI conformance / delivery telemetry · CMCD and CMSD against an independent library (pull_request) Has been cancelled
WAI conformance / cross-arch byte-equality · linux (pull_request) Has been cancelled
WAI conformance / cross-arch byte-equality · linux-arm64 (pull_request) Has been cancelled
WAI conformance / cross-arch byte-equality · macos (pull_request) Has been cancelled
WAI conformance / codec wrappers · fuzz smoke, text decode limits, C entry points, provenance, the semantic path (pull_request) Has been cancelled
WAI conformance / C2PA manifest emission · signed, read back, validated (pull_request) Has been cancelled
standards workspaces build + test / which workspaces this change touches (pull_request) Has been cancelled
standards workspaces build + test / macOS energy backends (pull_request) Has been cancelled
repository checks / lockfiles, advisories, tests in CI, workflow hygiene, spec and section references, public text (pull_request) Has been cancelled
joule-moq-web browser client / joule-moq-web · does this change touch it (pull_request) Has been cancelled
joule-moq-web browser client / joule-moq-web · browser end-to-end (macos) (pull_request) Has been cancelled
joule-moq-room room service / joule-moq-room · does this change touch it (pull_request) Has been cancelled
joule-moq-room room service / joule-moq-room, every target (pull_request) Has been cancelled
b41b566be1
Co-Authored-By: Claude Opus 5.5 <[email protected]>
dcharlot closed this pull request 2026-10-26 20:59:25 -04:00
Some checks failed
wai-web wasm32 / does this change touch it (pull_request) Has been cancelled
wai-web wasm32 / wai-web wasm32 build · corpus, menu and writers (pull_request) Has been cancelled
wai-web reach builds / wai-web reach · does this change touch it (pull_request) Has been cancelled
wai-web reach builds / wai-web reach builds · the package of record, every output as recorded (pull_request) Has been cancelled
wai-web reach builds / wai-web reach builds · this tree's wai-web, every rung on its engine line (pull_request) Has been cancelled
wai-quantum studio and desktop / does this change touch it (pull_request) Has been cancelled
wai-quantum studio and desktop / studio · browser (pull_request) Has been cancelled
wai-quantum studio and desktop / desktop app · macos (pull_request) Has been cancelled
wai-pack / wai-pack · does this change touch it (pull_request) Has been cancelled
wai-pack / wai-pack · linux (pull_request) Has been cancelled
wai-pack / wai-pack · linux-arm64 (pull_request) Has been cancelled
wai-pack / wai-pack · macos (pull_request) Has been cancelled
wai-pack / wai-pack · independent verifier (Python) (pull_request) Has been cancelled
wai-pack / wai-pack · the floor through a media prober (pull_request) Has been cancelled
wai-pack / wai-pack · dependency licences (pull_request) Has been cancelled
wai-ops-gpu integer operators on GPUs / does this change touch it (pull_request) Has been cancelled
wai-ops-gpu integer operators on GPUs / builds · no adapter is not a pass (linux) (pull_request) Has been cancelled
wai-ops-gpu integer operators on GPUs / Metal GPU and browser WebGPU (macos) (pull_request) Has been cancelled
WAI HTTP delivery / WAI HTTP · does this change touch it (pull_request) Has been cancelled
WAI HTTP delivery / HTTP corpus · falsify mutations 0/3 (pull_request) Has been cancelled
WAI HTTP delivery / HTTP corpus · falsify mutations 1/3 (pull_request) Has been cancelled
WAI HTTP delivery / HTTP corpus · falsify mutations 2/3 (pull_request) Has been cancelled
WAI HTTP delivery / HTTP corpus · falsify patches 0/3 (pull_request) Has been cancelled
WAI HTTP delivery / HTTP corpus · falsify patches 1/3 (pull_request) Has been cancelled
WAI HTTP delivery / HTTP corpus · falsify patches 2/3 (pull_request) Has been cancelled
WAI HTTP delivery / HTTP corpus · every patch and mutation in exactly one shard (pull_request) Has been cancelled
WAI HTTP delivery / HTTP corpus · verified both ways, and the differential (pull_request) Has been cancelled
WAI HTTP delivery / cache rules · controls and labelled caches (pull_request) Has been cancelled
WAI energy receipts / does this change touch it (pull_request) Has been cancelled
WAI energy receipts / energy receipts · linux (pull_request) Has been cancelled
WAI energy receipts / energy receipts · linux-arm64 (pull_request) Has been cancelled
WAI energy receipts / energy receipts · macos (pull_request) Has been cancelled
WAI conformance / does this change touch it (pull_request) Has been cancelled
WAI conformance / cargo-class corpora · linux (pull_request) Has been cancelled
WAI conformance / cargo-class corpora · linux-arm64 (pull_request) Has been cancelled
WAI conformance / cargo-class corpora · macos (pull_request) Has been cancelled
WAI conformance / quantum core and corpus under wasm32-wasip2 (wasmtime) (pull_request) Has been cancelled
WAI conformance / v2 vocabulary · manifests validate as stated (pull_request) Has been cancelled
WAI conformance / independent verifiers · claims, JWP, key discovery and integer payloads (Python) (pull_request) Has been cancelled
WAI conformance / delivery telemetry · CMCD and CMSD against an independent library (pull_request) Has been cancelled
WAI conformance / cross-arch byte-equality · linux (pull_request) Has been cancelled
WAI conformance / cross-arch byte-equality · linux-arm64 (pull_request) Has been cancelled
WAI conformance / cross-arch byte-equality · macos (pull_request) Has been cancelled
WAI conformance / codec wrappers · fuzz smoke, text decode limits, C entry points, provenance, the semantic path (pull_request) Has been cancelled
WAI conformance / C2PA manifest emission · signed, read back, validated (pull_request) Has been cancelled
standards workspaces build + test / which workspaces this change touches (pull_request) Has been cancelled
standards workspaces build + test / macOS energy backends (pull_request) Has been cancelled
repository checks / lockfiles, advisories, tests in CI, workflow hygiene, spec and section references, public text (pull_request) Has been cancelled
joule-moq-web browser client / joule-moq-web · does this change touch it (pull_request) Has been cancelled
joule-moq-web browser client / joule-moq-web · browser end-to-end (macos) (pull_request) Has been cancelled
joule-moq-room room service / joule-moq-room · does this change touch it (pull_request) Has been cancelled
joule-moq-room room service / joule-moq-room, every target (pull_request) Has been cancelled

Pull request closed

Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Transaction-Science/open-standards!484
No description provided.