b518ffcf32
The record leg's builds failed the menu workload: the package of record is
built from 248846f1, and the tree's registry now has a candidate that
package does not know, #446's wai.image.ieee1857_11, so the package's
isCandidate disagreed with the registry. A package cannot emit or dispatch a
capability it knows nothing of; the workload now compares candidates for the
capabilities the package registers and lists the rest as `newer`. The tree's
build registers every capability of the registry it is exported from, so on
the tree leg `newer` is empty and nothing is passed by.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
231 lines
15 KiB
Markdown
231 lines
15 KiB
Markdown
# wai-web reach builds
|
||
|
||
The shipped wai-web package (`wai_web.js` and `wai_web_bg.wasm`, the wasm-bindgen web
|
||
target) runs where an engine has WebAssembly with sign-extension, non-trapping
|
||
float-to-int conversion and bulk memory, where a 64-bit integer crosses the JS boundary as
|
||
a `BigInt`, and where ES modules load. The reach builds run the same code on engines
|
||
without those features, and give the same outputs byte for byte. Nothing in the Rust
|
||
changes: every reach build is derived from the shipped package itself.
|
||
|
||
| rung | files | the engine needs | the CI proxies it runs on |
|
||
|---|---|---|---|
|
||
| shipped | `../wai_web.js`, `../wai_web_bg.wasm` (the package) | WebAssembly with sign-extension, non-trapping float-to-int and bulk memory; i64 as `BigInt`; ES modules | a current Node |
|
||
| **L1** | `wai_web_bg.l1.wasm` + `wai_web.reach.js` | the same WebAssembly features; no `BigInt` at the boundary | `node:12`, `node:14` |
|
||
| **L2** | `wai_web_bg.l2.<profile>.wasm` + `wai_web.reach.js` | MVP WebAssembly | `node:8.0.0` (receipt), `node:8`, `node:10` |
|
||
| **J** | `wai_web.j.<profile>.js` | ES5 and typed arrays; no WebAssembly | `node:0.12`, `node:8.0.0` (full), and a current Node stripped to ES5 |
|
||
|
||
- **L1** is the shipped binary with its JS interface legalised (`wasm-opt
|
||
--legalize-js-interface`): an i64 parameter becomes a pair of i32 halves, and an i64
|
||
result returns its low half and passes the high half through `env.setTempRet0`. Nothing
|
||
else changes, so the code runs at the shipped build's speed.
|
||
- **L2** is the shipped binary lowered to the MVP (`--signext-lowering`,
|
||
`--llvm-nontrapping-fptoint-lowering`, `--llvm-memory-copy-fill-lowering`) and then
|
||
legalised. Lowered without legalisation it would still fail every export with an i64 in
|
||
its signature, so it is never shipped alone.
|
||
- **J** is the MVP-lowered module translated to JavaScript by `wasm2js`, in one ES5
|
||
script per profile: the runtime, the i64 table, the translation and the glue.
|
||
|
||
`wai_web.reach.js` and every J script carry the same ES5 parts:
|
||
|
||
- **The runtime** (`src/wai-reach-runtime.js`) installs what an old engine lacks and the
|
||
glue or the translation uses (`Math.imul` / `fround` / `clz32` / `trunc`,
|
||
`TypedArray#slice`, `TextEncoder` / `TextDecoder`, `Array#fill`, a small `Map`, a
|
||
`Symbol.iterator` stand-in, and a few more), each only where it is missing. It also joins
|
||
and splits i64 values at the boundary, from the table the build extracts from the binary.
|
||
With `BigInt` a 64-bit value is a `BigInt`, as the shipped build gives it. Without
|
||
`BigInt` it is a `Number`, exact below 2^53.
|
||
- **The glue** is the shipped `wai_web.js`, changed in a fixed, counted set of places
|
||
(`tools/cli.mjs`, `glue`; `manifest.json` records the counts) and compiled to ES5:
|
||
- its WebAssembly loading is replaced by two hooks the loader calls;
|
||
- a cached memory view is rebuilt whenever `wasm.memory.buffer` is a different buffer.
|
||
The shipped glue rebuilt one only when its buffer was detached, which a WebAssembly
|
||
memory does when it grows and the translation's memory does not;
|
||
- `Symbol.dispose` is read only where `Symbol` exists;
|
||
- the 64-bit helpers take the `Number` path without `BigInt`;
|
||
- copies out of wasm memory no longer use `TypedArray#slice`, which early engines
|
||
implement as a view.
|
||
|
||
## Profiles
|
||
|
||
L2 and J come in profiles. `profiles.json` lists the exports each one keeps, and
|
||
`wasm-metadce` drops what they cannot reach:
|
||
|
||
| profile | what it keeps |
|
||
|---|---|
|
||
| `full` | everything |
|
||
| `verify_decode` | `verify`, plus the integer decoders (WIV1, WIH1, WIA1, WIS1, int_synth, the JSON integer forms) and the staged-delivery sink |
|
||
| `verify` | `receipt`, plus multi-rendition envelopes, pins, the capability registry, stage identities, session claims and the output tree |
|
||
| `receipt` | envelope parsing, asset provenance (content hash, Ed25519 signature, Merkle chain) and the hashes |
|
||
|
||
A page that calls a function outside its profile gets a `TypeError`. The shipped build and
|
||
L1 are always the full module.
|
||
|
||
## The loader
|
||
|
||
`wai-load.js` is ES5. It chooses a rung from feature probes alone, never from the user
|
||
agent, and its detection never throws:
|
||
|
||
1. no typed arrays: none, and the reason says to serve the classical floor renditions;
|
||
2. no WebAssembly, or a WebAssembly that refuses a minimal MVP module: **J**;
|
||
3. an early WebAssembly engine line (WebAssembly and `Error.captureStackTrace` present, no
|
||
RegExp `dotAll` flag; the feature `noDotAll`): **J**, or the L2 receipt build when the
|
||
page asks for the receipt profile. The full MVP module has crashed an engine of this
|
||
line on arm64;
|
||
4. a probe module for sign-extension, non-trapping float-to-int or bulk memory (with a
|
||
data-count section) fails `WebAssembly.validate`: **L2**;
|
||
5. a one-function module returning its i64 argument does not take and give a `BigInt`:
|
||
**L1**;
|
||
6. otherwise: **shipped**.
|
||
|
||
```html
|
||
<script src="/wai/pkg/reach/wai-load.js"></script>
|
||
<script>
|
||
WaiLoad.load({ profile: 'verify' }, function (err, r) {
|
||
if (err) { /* r.reason says why; serve the classical floor */ return; }
|
||
// r.api is wai-web's API; r.rung is 'shipped' | 'L1' | 'L2' | 'J'; r.reason says why.
|
||
var p = new r.api.AssetProvenance(envelopeBytes, receiptJson);
|
||
});
|
||
</script>
|
||
```
|
||
|
||
| option | meaning |
|
||
|---|---|
|
||
| `base` | the folder of the reach files, ending in `/`; by default the folder `wai-load.js` came from |
|
||
| `profile` | `full` (default), `verify_decode`, `verify` or `receipt`; used by L2 and J |
|
||
| `force` | `shipped`, `L1`, `L2` or `J`: load that rung whatever the probes say (testing). The global `WAI_REACH_FORCE` does the same |
|
||
| `shipped` | `{ glue, wasm }` URLs of the shipped package; by default `base + '../wai_web.js'` and `base + '../wai_web_bg.wasm'` |
|
||
| `host` | `{ loadScript, fetchBytes, importModule, streams }`, to load from somewhere other than the page. With `streams` (a browser's default), the shipped glue gets the wasm URL and compiles it while it downloads |
|
||
|
||
`WaiLoad.detect(opts)` returns `{ rung, profile, reason, features }` without loading
|
||
anything. After a load, `WaiLoad.state` holds what the callback received.
|
||
|
||
## Building
|
||
|
||
```
|
||
bash wai/wai-web/reach/build.sh <package-dir> [<dist-dir>]
|
||
```
|
||
|
||
`<package-dir>` holds `wai_web.js` and `wai_web_bg.wasm`. The output goes to
|
||
`<dist-dir>` (default `reach/dist/`, which git ignores) with a `manifest.json`: every
|
||
file's SHA-256 and its raw, gzip and brotli sizes, the input's SHA-256, the tool pins, and
|
||
the counts of the glue changes.
|
||
|
||
The tools are pinned:
|
||
|
||
- binaryen at release `version_123`, by the SHA-256 of each platform's archive, checked
|
||
before use. The platforms are macOS arm64 and x86_64, and Linux x86_64 and aarch64;
|
||
- TypeScript, esbuild, acorn and binaryen's WebAssembly build of the same release by
|
||
`package-lock.json`. `npm ci` checks their integrity hashes. J's `wasm2js` is that
|
||
WebAssembly build, run on Node: the x86_64 Linux archive's native `wasm2js` allocates the
|
||
translation's locals differently from the macOS archive's, while the WebAssembly build
|
||
gives the same output on every host (and the macOS archive's output). Every other step
|
||
gives the same bytes from every archive.
|
||
|
||
The build refuses:
|
||
|
||
- an input that uses any feature beyond the shipped build's;
|
||
- a reduced module that exports anything other than its profile;
|
||
- an L2 module whose i64 boundary differs from the shipped binary's;
|
||
- a glue whose layout it does not recognise;
|
||
- a script that does not parse as ECMAScript 5.
|
||
|
||
`manifest.json` in this folder is the build of the shipped package of record. Its `input`
|
||
says which package: the SHA-256 of both files, the commit they were built from, and how.
|
||
Given that package, `build.sh` refuses any output whose SHA-256 differs from the
|
||
manifest's; with `WAI_REACH_REQUIRE_RECORD=1` it also refuses any other package.
|
||
`WAI_REACH_RECORD=1` writes a new manifest.
|
||
|
||
## Conformance
|
||
|
||
Every rung must give the shipped build's outputs. The runners below check this:
|
||
|
||
- **`test/conformance.js`** (ES5, Node 0.12 onward) loads a rung through `wai-load.js` and
|
||
runs `test/workloads.js` over the repository's corpora. Each workload is checked against
|
||
an independent reference (a corpus's hashes, a sidecar's SHA-256, the receipt's signed
|
||
fields, the registry), and its output must equal the shipped build's byte for byte. The
|
||
workloads, by profile:
|
||
- receipt: the hashes; envelope parsing; asset provenance over the vector in
|
||
`test/vectors/provenance/` (its README says how it is made), including a tampered
|
||
copy and a forged joule total;
|
||
- verify: the capability menu against the registry; the staged corpus's content hashes;
|
||
multi-rendition envelopes; the output tree of the staged-measurement goldens;
|
||
- verify_decode: every committed WIV1, WIH1, WIA1 and WIS1 payload against its sidecar;
|
||
the JSON integer forms; the committed int_synth vectors; all of `staged-conformance/`
|
||
replayed through `StageSink`;
|
||
- full: i64 parameters and results on both sides of 2^32.
|
||
|
||
Under J, `WebAssembly` is deleted before anything loads. `--strip es5` also deletes
|
||
`BigInt`, `Map`, `Symbol`, `Promise` and the other post-ES5 builtins, and makes
|
||
`TypedArray#slice` return a view.
|
||
- **`test/run-modern.mjs`** runs on a current Node:
|
||
- the loader must choose the shipped build, and that run is the reference;
|
||
- every rung and profile is forced and must equal it;
|
||
- an engine stripped to ES5 must be given J, and the same outputs;
|
||
- the demo verify scripts (`demo/verify_*.mjs`: integer payloads, int_synth, the staged
|
||
corpus with its guards and fuzz, the menu, the writers, the staged measurement) run
|
||
against shipped, L1, L2 and J through a generated adapter module. The measurement
|
||
script's written files must equal the shipped build's. They are this tree's scripts,
|
||
or, with `--demo-commit <commit>`, that commit's scripts over that commit's corpora
|
||
(its `wai/`, exported with `git archive`). A script this tree runs that the commit
|
||
lacks is reported as `SKIP`.
|
||
|
||
```
|
||
node wai/wai-web/reach/test/run-modern.mjs --dist <dist> --shipped <package-dir> [--demo-commit <commit>]
|
||
node wai/wai-web/reach/test/conformance.js --dist <dist> --wai wai --rung J --profile receipt --reference <reference.json>
|
||
```
|
||
|
||
CI (`.github/workflows/wai-web-reach.yml`, on the `linux` runner) runs a leg per package:
|
||
|
||
- **The package of record.** `record/` keeps the two files the site serves under
|
||
`/wai/pkg/`, built from the commit `manifest.json` names (`input.commit`; `input.source`
|
||
says how). Both files must have the SHA-256 the manifest records, and `build.sh` then
|
||
runs with `WAI_REACH_REQUIRE_RECORD=1`, so every output must have the manifest's SHA-256
|
||
and size: these are the bytes the site serves under `/wai/pkg/reach/`. The package is
|
||
kept rather than rebuilt in CI: the same commit and recipe give a different wasm on a
|
||
Linux x86_64 host than on the macOS arm64 host that built it (the glue is the same), so
|
||
the commit's build is checked on that host (the site's `build-wasm-pkg.sh recorded wai`).
|
||
The demo verify scripts on this leg are that commit's, over that commit's corpora
|
||
(`run-modern.mjs --demo-commit`): a script a later commit changes checks code the
|
||
package does not hold. The reach workloads and the engine images are this tree's. The
|
||
menu workload holds the package to the registry for the capabilities the package
|
||
registers; one the registry gained since is listed (`newer`), not held against it.
|
||
- **The tree.** wai-web is built from the pull request's own tree, so a change to wai-web
|
||
meets the reach builds before the package of record moves to it. These outputs are not
|
||
the recorded ones. The demo verify scripts are this tree's, so a check a pull request
|
||
adds runs here against the code it checks.
|
||
|
||
On each leg:
|
||
|
||
1. `run-modern.mjs` runs on the outputs;
|
||
2. `conformance.js` runs in one Node image per engine line. On each line the loader must
|
||
choose that line's rung by itself, and every output must equal the shipped build's
|
||
from step 1.
|
||
|
||
### What each proxy stands for
|
||
|
||
The images are proxies. Each runs one JavaScript engine line on x86_64 Linux, with the
|
||
engine features below; real devices ship engines of the same lines. **No real old device
|
||
has been run.** Real devices also differ in ways no proxy reproduces: 32-bit ARM code
|
||
generation, engines with the JIT switched off, small memories, and engine families other
|
||
than the proxied one.
|
||
|
||
| proxy | engine features | rung the loader must choose | device classes it stands for |
|
||
|---|---|---|---|
|
||
| `node:0.12` | ES5 and typed arrays; no WebAssembly | J | TV sets and boxes whose web engine, or system web view, is frozen at an engine with no WebAssembly, smart TVs of model years 2014–2019 among them; phones whose system web view predates WebAssembly; broadcast-application receivers (HbbTV 2.0.x, ATSC 3.0 A/344). Console browsers that run with the JIT off, and engine families other than the proxied one, are J devices that this proxy does not cover |
|
||
| `node:8.0.0` | MVP WebAssembly; no RegExp `dotAll` flag | J, or L2 for the receipt profile | TV sets and boxes whose system web view is frozen at an engine with WebAssembly but no RegExp `dotAll` flag. The full MVP module has crashed an engine of this line on arm64; this x86_64 proxy does not crash, so it checks the choice and the outputs, not the crash |
|
||
| `node:8` | MVP WebAssembly and the `dotAll` flag; no `BigInt` | L2 | TV sets and boxes whose system web view is frozen at MVP WebAssembly with no `BigInt`, and TV platforms of model years 2019–2020 |
|
||
| `node:10` | MVP WebAssembly and `BigInt`; no sign-extension or bulk memory | L2 | TV sets and boxes whose system web view has `BigInt` but not the post-MVP features the shipped build uses |
|
||
| `node:12` | the shipped build's WebAssembly features; i64 does not cross the JS boundary as `BigInt` | L1 | TV platforms of model year 2021 |
|
||
| `node:14` | the same | L1 | TV platforms and streaming sticks whose engine has the shipped build's WebAssembly features but passes no i64 as `BigInt`, and phones' system web views of the same line |
|
||
| a current Node | every feature, with i64 as `BigInt` | shipped | current engines. Engine families other than the proxied one are not proxied here |
|
||
|
||
## Limits
|
||
|
||
- J is slower than WebAssembly, and far slower on engines without a JIT. The `receipt`
|
||
and `verify` profiles are the ones for small or slow devices.
|
||
- Without `BigInt`, a 64-bit value is a `Number`, exact below 2^53. `IntPayloadCost.macs`
|
||
(a 128-bit value) and the functions that return a `BigInt64Array` still need `BigInt`.
|
||
- A trap in the WebAssembly builds is an `Error('abort')` in J. The translation does not
|
||
check memory accesses out of bounds, which trap in WebAssembly. The Rust does not make
|
||
such accesses, and the conformance runs compare every output.
|
||
- Reduced profiles throw a `TypeError` for any function outside them.
|