open-standards/wai/extensions/avatars.md

WAI Extension: Avatars (Digital Humans)

Status: Draft. Phase 4 of the cargo roadmap. Motion as instructions: ship a signed stream of joint transforms — a skeleton, not a video of a body — and the sink skins a mesh by the deterministic pose. An avatar is a skeletal world, so conformance is pose-equivalence, a special case of replay-equivalence. Reference impl: the world feature of wai-rs (sim wai.world.sim.skeletal0); corpus: avatar-conformance/; live sink: wai.transaction.science/avatar.

§6 to §13 register wai.avatar.face: a talking head made from a signed keypoint track and an identity prior, the same bytes on every sink, so that a receipt proves the face a viewer saw. A participant commits once to the prior, a reference picture and its keypoints named by SHA-256, and drives it with segments of keypoints, both signed by one key: that signature records who drove which face. A sink warps the reference picture to the driving keypoints and refines the warp with an integer network pinned by SHA-256, one graph of the integer operator set; CPU, wasm32 and GPU give the same frames. Each segment carries a synthesis origin claim of class synthesized, and a player that presents it is held to synthesis's label rules. Reference impl: the avatar_face module of wai-rs (feature avatar_face); corpus: live-conformance/avatar-face/, whose second implementation is live-conformance/avatar_face_verify.py; the tool that trains and measures generators is tools/wai_avatar_face_train.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

1. Scope and model

A wai.avatar.pose object is a skeletal world: the same WWLD container, op log, canonical ordering, causality, sealing, state hash, and receipt chain as interactive-worlds (see interactive-worlds.md), with the simulation step reduced to the identity (as with state-feeds). The bones are nodes; an animation / expression op stream sets joint transforms; nothing is integrated. A telepresence stream is a live world of one avatar — the op log that drives a multiplayer world drives a body.

In scope: the bone model, the registered skeletal sim (wai.world.sim.skeletal0) and its mutation kinds, the statement that conformance is worlds' replay-equivalence over the pose, and the binding to a wai.mesh.glb for skinning.

Out of scope: everything interactive-worlds already defines (normative here unchanged); the skinning algorithm and forward-kinematics composition (sink presentation — a pure function of the hashed pose + the hierarchy); a transport; blend-shape / morph-target semantics beyond joint transforms (a later registration may add them).

Relationship to the core spec: a registration under interactive-worlds, not a new container. media takes the value "avatar"; the authoritative capability is wai.avatar.pose, carried in a WWLD whose contract names sim = wai.world.sim.skeletal0. The avatar's mesh is a content-addressed wai.mesh.glb asset (its armature + skin weights), bound to the rig like any other presentation asset.

2. The bone

A node is a bone: a parent reference plus a local TRS transform. Canonical field order (and therefore hash order) is exactly:

field	type	meaning
parent	[u8; 16]	parent bone's node id, or all-zero for a root
translation	[Fx; 3]	local translation (Q32.32)
rotation	[Fx; 4]	local rotation quaternion (x, y, z, w)
scale	[Fx; 3]	local scale

The canonical node serialization is parent(16) | translation | rotation | scale = 16 + 10×i64 = 96 bytes. The default bone is the identity (zero translation, identity quaternion, unit scale).

The floor stays trig-free: a rotation is carried as a quaternion, and the quaternion → rotation-matrix the sink uses for skinning is + − × only. Axis-angle authoring (which needs sin/cos) happens once, off-line, in whatever produces the op bytes — the canonical artifact is the Fx quaternion, and every sink replays it exactly.

3. The skeletal sim — wai.world.sim.skeletal0

Registers these mutation kinds; the step is the identity. All are LWW (a transform set rides the canonical total order — last canonical writer wins; there is no commutative transform set, so no SUM kind).

kind	payload	effect
0x0001 CREATE	parent[16] | translation[3] | rotation[4] | scale[3]	insert a bone; target MUST equal mint_node_id(actor, lamport)
0x0002 DELETE	(empty)	remove the bone
0x0010 SET_ROTATION	rotation[4]	set a joint's local rotation (the animation driver)
0x0011 SET_TRANSLATION	translation[3]	set a bone's local translation
0x0012 SET_SCALE	scale[3]	set a bone's local scale

An op targeting an absent bone is a deterministic no-op (interactive-worlds §6.4). parent need not name a present bone — a missing parent is treated as a root at FK time; the hash depends only on the stored values, so existence is irrelevant to conformance.

4. Conformance — pose-equivalence

Given the same avatar container and the same signed op set, a conforming sink MUST compute the identical BLAKE3 state hash at every checkpoint — the per-bone local transforms, in node-id order — on every machine, no tolerance parameter.

This is interactive-worlds' criterion verbatim, applied to the bone state. The forward-kinematics composition into world space and the skinning of the mesh produce identical results on any two sinks that agree on the pose (a pure function of the hashed locals + the hierarchy), so they are legitimately left to presentation and excluded from the hash — exactly as wai.mesh.glb geometry is.

5. Receipt (JWP profile)

The interactive-worlds session receipt (jwp-receipts.md) applies unchanged: world identity over contract + tick-0 graph; the group's objects are the op batches, Merkle-bound; one Ed25519 seal (verified as SPEC §7 Signatures requires) over world hash + root + checkpoints + exact signed work + carried joules_micro (with its acquisition class as an optional signed label, energy-measurement §2.8); parent_receipt_hash chains a performance's seals into a timeline.

6. Faces from keypoints: wai.avatar.face

A participant sends a face as instructions: a stream of keypoints (the eyes, the lids, the pupils, the nose, the mouth, the chin, the brows, the cheeks), a few bytes a frame, rather than pictures. Every sink makes the pictures, the same bytes on every sink, from three things:

• the identity prior (§7.3): a reference picture of the face and its keypoints, one file named by its SHA-256, to which the participant commits once in a signed identity record (§8.2);
• the generator (§7.2): an integer network in one file, pinned by its SHA-256 as SPEC §3.1 pins any parameter set;
• the keypoint track: segments of the payload WFK1 (§7.4), each one object, each signed by the participant in a track record (§8.3).

A sink's driver warps the reference picture to each frame's keypoints and draws a map of each keypoint (§7.5); the generator's network, one WHG1 graph of wai-ops/1 (integer-operators), refines the warp into the frame (§7.6). Every operation is integer, so a frame is a function of the three files' bytes and nothing else, and it is the same on a CPU, under wasm32 and on a GPU (§7.8). Since the frames are generated, every segment carries a synthesis origin claim of class synthesized (§8.4), and a player that presents them is held to synthesis's label rules (§8.5). A receipt over a segment then proves the face a viewer saw: anyone who holds the three files recomputes it.

The identity record and the track records are signed by one key. That signature is provenance: it records which participant drove which face. It is not a grant of permission, and this extension states no rule about permission: the service a distributor runs is its own matter. Where a lawful basis for processing a face is recorded, the record is a lawful-basis claim of jwp-receipts §2.4.6.

In scope: the integer profile of the generator and its three files (§7.1 to §7.4); the driver and the graph (§7.5, §7.6); the frames and the tier they hold (§7.7, §7.8); the measurement of the registered generator (§7.9); the records and the synthesis claims over the frames (§8); verification (§9); the mappings of §11.

Out of scope: how a sender finds the keypoints in a camera's picture (a detector at the sender, which may be floating point: its output is the keypoints the track carries, and nothing a sink computes depends on how they were found); how a generator is trained (§7.9 says how the registered one was); a transport, which WAI defines none of (moq-streaming-format states it in its preamble); an identity prior in the form of a latent, which this revision does not register; a watermark in the frames (synthesis §7), which this revision does not apply; and faces in three dimensions, which the pose capability of §1 to §5 serves.

Relationship to §1 to §5: a pose stream drives a mesh that a sink skins, and the skinning is presentation, outside the state hash; a keypoint track drives a generator whose frames are the hashed output. Relationship to the core spec: wai.avatar.face is a payload capability of SPEC §5, registry class GenerativeDeterministic (SPEC §7): its decode generates, and its conformance, face-equivalence (§7.7), is a case of generation-equivalence (determinism-tiers §2). Its parameter set is one file, the generator; the identity prior is named by the payload's own digest, as wai.generate.spectral's model is, and no pin covers it (SPEC §3.1). A decoder module of the WA²ISM™ profile (waism §5) is given both as its set, the file sequence of the generator, named generator.wfg, and the identity prior, named identity.wfi; it refuses a generator whose SHA-256 is not the segment's generator (generator_mismatch), and a prior whose SHA-256 is not its identity (identity_mismatch).

7. The face generator

7.1 The integer profile

Every value of §7.5 and §7.6 is an integer, and no formula wraps or saturates unless it says so. The generator's network follows one profile:

• Weights are integers at the scale 2^12, each in a signed 16-bit field: a float weight w is the integer round(w · 2^12). Biases are integers at the accumulator's scale, 2^27, each in a signed 32-bit field.
• Activations are integers at the scale 2^15 (Q15). Every hidden activation lies in 0..32767: a clamp after each hidden convolution.
• One rounding per operator. A convolution forms its sum exactly, adds its bias, and rounds once, R_half_up of integer-operators §2 with the shift 12, back to Q15. Every other operator that rounds rounds once, half up.
• Nonlinearities are clamps, or a lut16 table (integer-operators §5.6) whose every entry is a stated integer computation (§7.6), so that a second implementation computes the table and takes no float library on trust.

How the integers were obtained is not part of the decode. Informative: the reference tool rounds each float32 weight times 2^12, and each bias times 2^27, half to even; a power-of-two scaling of a float32 is exact, so the integers follow from the float's bits alone.

7.2 The generator: WFG1

One file. Every multi-byte field is little-endian.

off	size	field	rule
0	4	magic WFG1
4	1	version	1
5	1	K	1 to 32: the keypoints of a face
6	2	width	4 to 512, a multiple of 4
8	2	height	4 to 512, a multiple of 4
10	2	c1	1 to 64
12	2	c2	1 to 64
14	2	c3	1 to 64
16	2	gauss	at least 1: g of §7.5
18	4	bg	at least 1: B of §7.5
22	…	the ten convolutions	in the order below; nothing follows the last

Each convolution is 3 × 3 with zero padding 1 and one group, and is its weights, signed 16-bit integers in the order [cout][cin][3][3], then its biases, cout signed 32-bit integers. Their input channels, output channels and strides are, in order:

#	cin	cout	stride
1	6 + K	c1	1
2	c1	c2	2
3	c2	c3	2
4	c3	c3	1
5	c3	c3	1
6	c3	4 · c2	1
7	2 · c2	c2	1
8	c2	4 · c1	1
9	2 · c1	c1	1
10	c1	4	1

A reader checks, in this order, and the first that fails names the refusal: the magic (bad_magic); the version, present and 1 (version); the header's other fields present (truncated) and within their rules (field_range); each convolution's arrays present (truncated); nothing after the last (trailing). A file that fails is not a generator.

The generator is pinned as SPEC §3.1 pins a parameter set of one file: sha256 is the SHA-256 of the file, with no form. Nothing but the digest names a generator.

7.3 The identity prior: WFI1

One file: a reference picture and where its keypoints are.

off	size	field	rule
0	4	magic WFI1
4	1	version	1
5	1	K	1 to 32
6	2	width	4 to 512, a multiple of 4
8	2	height	4 to 512, a multiple of 4
10	4 · K	the keypoints	each x then y, signed 16-bit, in sixteenths of a pixel (§7.5)
…	width · height · 3	the picture	RGB8: rows top to bottom, pixels left to right, R, G, B bytes

Nothing follows the picture. A reader checks in §7.2's order: bad_magic, version, truncated for a header cut short, field_range, truncated for the keypoints and the picture, trailing. The prior's digest is the SHA-256 of the file; nothing else names it.

7.4 The keypoint segment: WFK1

A segment holds the keypoints of one or more consecutive frames of one face.

off	size	field	rule
0	4	magic WFK1
4	1	version	1
5	1	K	1 to 32
6	1	step	1 to 16: a coordinate is a multiple of step sixteenths of a pixel
7	1	order	0 to 15: the order of the deltas' code
8	32	generator	the SHA-256 of the generator (§7.2)
40	32	identity	the SHA-256 of the identity prior (§7.3)
72	4	first	the index of the segment's first frame in the track
76	2	n	1 to 1024: its frames
78	2	rate_num	at least 1
80	2	rate_den	at least 1: the frame rate is rate_num / rate_den frames a second
82	2	reserved	0
84	4 · K	frame 0	its 2 · K coordinates, x then y for each keypoint, each a signed 16-bit value v; the coordinate is v · step
84 + 4 · K	…	the deltas	frames 1 to n − 1

The deltas. For each later frame, for each of its 2 · K coordinates in frame 0's order, one code: the value becomes the value before it plus the code's delta. A code is Exp-Golomb of order order, most significant bit first: z zero bits, a one bit, then z bits, which with the one form m (so m lies in 2^z..2^(z+1) − 1); then order bits r; the code is u = (m − 1) · 2^order + r. The delta is u / 2 for an even u and −(u + 1) / 2 for an odd one. The codes are packed with no gap; after the last, zero bits pad to a byte boundary, and nothing follows. Each coordinate is the value times step, in sixteenths of a pixel.

A reader checks, in this order: the magic (bad_magic); the version (version); the rest of the header present (truncated) and within its rules (field_range); frame 0 present (truncated), each of its coordinates in −32768..32767 (field_range); then each code in order: its bits present (truncated), at most 31 zeros before its one (field_range), the coordinate it gives in −32768..32767 (field_range); then fewer than eight bits of padding (trailing), each 0 (noncanonical).

Choosing order. Any order reads; an encoder chooses one per segment. Informative: the reference writer chooses the order that makes the segment shortest, the lowest where two tie.

The envelope. A segment is the payload of a WAI1 envelope (SPEC §2) whose capability is wai.avatar.face, whose intent is create, and whose conditioning.prior pins the generator the payload names: sha256 is the payload's generator (§9.2 step 3, generator_mismatch). The identity prior is the payload's identity, which no pin covers. The track's segments are objects of one track (a MoQ track, a sequence of envelopes); a segment of one frame is a frame sent on its own.

7.5 The driver

For a frame of driving keypoints Dₖ = (Dxₖ, Dyₖ), k < K, over an identity prior whose keypoints are Sₖ and whose picture is R of width × height (w × h), the driver computes, for each pixel (x, y), x < w, y < h, in sixteenths of a pixel (a pixel's centre is at (16x + 8, 16y + 8)):

1. Weights. For each k: dx = 16x + 8 − Dxₖ, dy = 16y + 8 − Dyₖ; eₖ = ⌊((dx² + dy²) · g + 128) / 256⌋ with g the generator's gauss; and wₖ = exp_neg_q16(−eₖ) (integer-operators §2), in 0..65553.
2. Displacement. N = B + Σ wₖ with B the generator's bg; Δx = rdiv(Σ wₖ · (Sxₖ − Dxₖ), N) and Δy likewise, with rdiv(n, d) = sgn(n) · ⌊(2|n| + d) / (2d)⌋, integer-operators E12's rounding, half away from zero.
3. The sample. U = 16x + Δx and V = 16y + Δy; ix = ⌊U / 16⌋, fx = U − 16 · ix, iy = ⌊V / 16⌋, fy = V − 16 · iy. With p(Y, X) = R[clamp(Y, 0, h − 1)][clamp(X, 0, w − 1)], each channel of the warped picture at (x, y) is ⌊((16 − fx)(16 − fy) · p(iy, ix) + fx(16 − fy) · p(iy, ix + 1) + (16 − fx)fy · p(iy + 1, ix) + fx · fy · p(iy + 1, ix + 1) + 128) / 256⌋.
4. The maps. For each k, the keypoint map Mₖ at (x, y) is ⌊wₖ / 256⌋, in 0..256.

At rest, where each Dₖ is Sₖ, the displacement is 0 and the warped picture is the reference picture. Width: every value is below 2^41 in magnitude; 64-bit integers hold them.

7.6 The graph: the lowering face-int/1

The generator's network is one WHG1 graph (integer-operators §10), which the lowering face-int/1 writes from the file by integer-operators §13.0's canonical rules. Its three data inputs, sizes set per decode: tensor 0, the warped picture [3, h, w], and tensor 1, the reference picture [3, h, w], each check [0, 255]; tensor 2, the maps [K, h, w], check [0, 256]. Then its operators, each output taking the next tensor index, every tensor derived with no exponent, each convolution of §7.2 written as conv with its cout, kernel 3 × 3, its stride on both sides, zero padding 1 on each side, one group, its weights at two bytes, and an rq of no multiplier, one shift 12, a pre bias per channel at four bytes and the rounding half_up; and relu written clamp(0, 32767):

1. rescale of tensor 0 with one multiplier 256 (four bytes), shift 0, a pre bias of −128 on each channel (four bytes), floor: the warped picture centred, t₃; the same of tensor 1: t₄; rescale of tensor 2 with one multiplier 128, shift 0, no bias, floor: t₅;
2. concat(t₃, t₄, t₅); convolution 1; relu: e₁;
3. convolution 2; relu: e₂; convolution 3; relu: e₃;
4. convolution 4; relu; convolution 5; add of it and e₃; relu: r;
5. convolution 6 of r; pixel_shuffle(2); relu; concat with e₂; convolution 7; relu: d₂;
6. convolution 8 of d₂; pixel_shuffle(2); relu; concat with e₁; convolution 9; relu: d₁;
7. convolution 10 of d₁: o, of four channels;
8. split(0, 3) of o; clamp(−32768, 32767): the generated picture G;
9. split(3, 1) of o; clamp(−32767, 32767); lut16 with the table T below and no exponents: the mask m;
10. mul(t₃, m) with an rq of no bias, shift 15, half_up; rescale of m with one multiplier −1, shift 0, a post bias of 32768, floor: 32768 − m; mul(G, that) with shift 15, half_up; add of the two;
11. rescale with no multiplier, shift 8, a post bias of 128 on each channel, half_up; then clamp(0, 255), the graph's one output, the frame [3, h, w].

The mask's table. For v in 0..32767, S(v) = ⌊(2^31 + ⌊D / 2⌋) / D⌋ with D = 65536 + min(exp_neg_q16(−16v), 65536): the logistic function of v / 4096, at Q15, from integer-operators §2's exponential, whose value at 0, 65553, is taken no higher than 2^16. T[v + 32768] = S(v) for v ≥ 0, 32768 − S(−v) for −32767 ≤ v < 0, and T[0] = 0. So T[32768] = 16384, the table is symmetric about it, and every entry lies in 0..32767.

So the frame is the warped picture blended with the generated one by the mask: m · t₃ + (32768 − m) · G, each product rounded once, brought back to samples. A generator that reads always meets acc64 (integer-operators §8): no value it forms passes 2^63. The registered generator does not meet acc32 (a 16-bit activation times a 16-bit weight over hundreds of terms), and every operator of the graph is in wai-ops/1-i32 (integer-operators §9.1), so a fabric of that subset runs it at acc64 (integer-operators §7.3).

7.7 Frames and face-equivalence

A segment's frames are, for each frame of the segment in order, the driver over the frame's keypoints (§7.5) and then the graph (§7.6); a frame is RGB8 of width × height, rows top to bottom. The generator, the identity prior and the segment MUST agree on K, and the generator and the prior on width and height; where they do not there are no frames (shape_mismatch, §9.2).

Face-equivalence. Given the same generator, identity prior and segment, a conforming sink MUST compute the same frames, byte for byte, on every machine. A segment's decode in canonical form, the buffer synthesis's present entry and a review name by digest (synthesis §2, §9), is its frames' RGB8 buffers one after another. The tier a sink's path holds over it is generation-equivalence (synthesis §9.2; determinism-tiers §2), with the rule of determinism-tiers §2 What a tier covers: a step after the generation that alters the frames leaves the tier.

7.8 Fabrics

The registered generator's frames were computed on these fabrics, and gave the same bytes:

• CPU: the integer operator set's interpreter, macOS arm64 (the corpus generator and its tests; §12);
• wasm32: the corpus regenerated byte for byte by wai_avatar_face_vectors verify built for wasm32-wasip2 and run under wasmtime, on macOS arm64;
• a WA²ISM module: every frame vector, as a segment of one frame, through the operator-set module held at the integer level (waism; wai/wai-int-module), its scalar and its SIMD128 build, on wasmtime (Cranelift and Winch), Wasmi, Node and Chromium, on x86_64 and arm64;
• GPU: every graph on the device through wai-ops/1-i32 in WGSL (determinism-tiers §5), at acc64, every graph also run on the interpreter and compared, on Metal, the integrated GPU of an arm64 desktop (tests/avatar_face_gpu.rs);
• the corpus's second implementation, live-conformance/avatar_face_verify.py, in Python's integers.

Nothing is claimed for a fabric not listed.

7.9 Measurement

The registered generator, the corpus's generator.wfg (SHA-256 2316f0c37f29c121e18627a836c50b959f0332f3320a338ec1dd814cf4f6dbc6), has K 16: in order, the left and the right eye, the left and the right upper lid, the left and the right pupil, the tip of the nose, the left and the right corner of the mouth, the upper and the lower lip, the chin, the left and the right brow, and the left and the right cheek. Its frames are 64 × 64, its widths c1, c2, c3 are 24, 48 and 64, its gauss is 1024 (a Gaussian of about 5.7 pixels) and its bg 6554 (0.1 at Q16).

tools/wai_avatar_face_train.py trained it on faces it draws itself, from seeded parameter sets (a shaded head, hair, eyes with lids and pupils, brows, a nose, a mouth with lips and teeth, a background), each picture with the keypoints its parameters place on it exactly; no picture from outside this repository was used. It drew 6000 faces, each at nine poses and expressions, and trained on pairs of one face: a reference picture warped by §7.5 to another of its pictures' keypoints, the target that picture. It trained 60000 steps in floating point, then 12000 with the integer arithmetic emulated in the loop (weights on the 2^−12 grid, activations on the 2^−15 grid rounded half up, the mask's table), on the mean absolute error of the frame and of its horizontal and vertical differences, plus a quarter of the generated picture's own error. Its report is the corpus's training.json.

Why a generator trained here. Before anything was downloaded, the open keypoint-driven face-reenactment generators were surveyed with the licences of their code and their weights:

generator family	licence of the weights	used
implicit three-dimensional keypoints over a feature volume, with a warping module and a decoder	the base networks permissive (MIT); the bundled face-detection models non-commercial	no: its receiving side needs operators wai-ops/1 does not define, three-dimensional convolution, sampling at positions the data gives, and spatial normalisation
a three-dimensional latent volume driven by a motion code	the repository's (Apache-2.0); its archive of dependencies carries other models under their own terms	no: the same operators
first-order motion, and thin-plate-spline motion, from keypoints	none stated for the weights (the code is MIT)	no: no licence is granted for the weights
a latent-motion animator; a depth-aware generator; a reimplementation of three-dimensional keypoint synthesis	non-commercial (CC BY-NC 4.0)	no
an articulated-motion animator	no licence granted	no
an audio-driven talking-head toolkit whose renderer is keypoint-driven	none stated for the weights (the code is Apache-2.0)	no

So no generator whose weights carry a permissive licence could be lowered to wai-ops/1 in this revision, and the registered one was trained here.

Quality. Measured over 12 faces held out of training, each driven for 100 frames at 25 frames a second (smooth head motion, blinks, a speech-like mouth, glances), each frame generated by the integer generator (the arithmetic of §7.5 and §7.6, which gives the corpus's frames) and by the floating-point network it was quantised from (the weights of the floating-point phase, in float32), against each other and against the face the renderer draws at those keypoints. PSNR is over RGB8; SSIM uses an 11 × 11 Gaussian window of σ 1.5, per channel, averaged:

compared	PSNR mean (dB)	PSNR min (dB)	SSIM mean	SSIM min
integer, float	48.54	39.42	0.9988	0.9953
integer, rendered	28.24	21.27	0.9238	0.8558
float, rendered	28.22	21.11	0.923	0.8546
warp alone, rendered	22.89	13.99	0.7635	0.3499

The last row is §7.5's warp alone, before the network.

Keypoint error. A floating-point keypoint detector, trained on the training faces, read the sixteen keypoints of each picture; the table gives the root mean square of their distances from the keypoints that drove the frame, in pixels of the 64 × 64 frame, over all sixteen and over the mouth's four (its corners and its lips). The first row is the detector's own error:

pictures read	all (px)	mouth (px)
the rendered faces	0.111	0.092
the float network's	0.569	0.547
the integer generator's	0.575	0.542

Bitrate. The tracks coded by §7.4 in segments of 25 frames (one second), the order chosen per segment, counting the payloads' bytes and no envelope's; "rendered" are the keypoints the renderer placed, "detected" those the detector read from the rendered frames, as a sender reads them from its camera's pictures; step 4 is a quarter of a pixel and step 1 a sixteenth:

keypoints	kbit/s
rendered, step 4	2.942
rendered, step 1	4.386
detected, step 4	2.96
detected, step 1	4.413

These are figures for one generated face set at 64 × 64. Nothing is claimed for captured faces, another size or another generator.

8. Records and synthesis

8.1 Encodings and signatures

Every record of this section is a JSON object under the encodings of synthesis §2: byte strings in lowercase hexadecimal, unsigned integers of at most 2^53 − 1 in plain decimal, no other number, no boolean, date-times as jwp-receipts §2.4.3 defines them, no member named twice, at most 32 levels of nesting, exactly the members a section lists, and a kind naming the record; one that does not is record_malformed. Every record carries signer_pubkey (32 bytes, an Ed25519 key of RFC 8032) and sig (64 bytes):

sig       = Ed25519(signer_key, "wai:avatar\x01" || JCS(record without "sig"))
link_hash = BLAKE3("wai:avatar-link\x01" || JCS(record))

JCS is RFC 8785. The signature is verified strictly (SPEC §7 Signatures): a key that fails step 1 or 2 is bad_key, and a signature that fails a later step bad_signature. A record is named by its link hash.

8.2 The identity record

A participant commits to an identity prior once, kind "wai-face-identity":

member	type	meaning
kind	string	"wai-face-identity"
identity	32 B hex	the SHA-256 of the identity prior (§7.3)
issued_at	date-time	when it signed it
signer_pubkey, sig		§8.1: the participant's key

8.3 The track record

The participant signs one track record for each segment, kind "wai-face-track":

member	type	meaning
kind	string	"wai-face-track"
object	32 B hex	the content hash (jwp-receipts §2.1) of the segment's envelope
identity	32 B hex	the link hash of the identity record of the face it drives
seq	integer	its position among that face's track records, from 0
prev	32 B hex or null	the link hash of that face's track record before it; null exactly for seq 0
frames	array or null	the SHA-256 of each frame the segment generates (§7.7), in order, 1 to 1024; null where the participant states none
issued_at	date-time	when it signed it
signer_pubkey, sig		§8.1: the identity record's key

A track record MUST be signed by the key that signs the identity record it names (driver_key_mismatch), and the segment's payload MUST name that record's prior (identity_mismatch). So one key, one face: a key cannot drive a face another key committed to, and a face cannot be driven by a segment that names another prior. A face's track records chain by seq and prev (track_chain_broken). Where frames is given, the participant states the frames every sink makes, and a verifier that holds the three files checks them (frame_mismatch).

8.4 Origin claims

Every segment carries a synthesis origin claim (synthesis §3.1), signed by the participant or by a party that generates its frames, with:

• subject the segment object, {"type": "object", "content_hash"};
• class synthesized and step generate; operation wai.avatar.face;
• models naming the generator's SHA-256, of type trained for a trained generator;
• record the link hash of the segment's track record;
• media video, and oversight as the generating party states it (informative: the corpus states prompt_guided, the keypoints of a person guiding the generation);
• source, as synthesis §5 computes it.

The segment's envelope generates (synthesis §3.3: its capability's class is GenerativeDeterministic), so a claim of any step but generate over it is class_understated, and a claim over a conversion of the frames keeps the class (synthesis §3.4). A segment with no claim is claim_missing; a claim whose class, step, model or record is not the above is claim_mismatch (§9.2 step 5). The claim covers every frame of the segment: its subject is the object whose decode the frames are (§7.7). A frame a party shows or holds apart from its segment is named in a claim of its own by the digest of its samples (synthesis §3.1), of class synthesized, with the segment as its input.

8.5 Presentation and labels

A player that presents a segment writes synthesis's present entry (synthesis §10.3): the segment as subject, its claim as origin, class synthesized, media video, tier generation-equivalence, decoder wai.avatar.face, and as output the digest of the frames it presented, face-equivalence's canonical buffer (§7.7). A verifier that recomputes the frames holds that buffer, so a player that presents other pictures under the claim, the warped pictures for instance, states an output the segment does not decode to (present_output_irreproducible, synthesis §9.2). Under a label policy (synthesis §10.1) whose classes include synthesized, the label is due at first exposure, after each interruption and at the policy's intervals, and a log that does not show it is label_not_shown (synthesis §10.4). A review of a segment (synthesis §9) names the canonical buffer as its reconstruction at generation-equivalence, and so holds for every viewer whose player generated the frames exactly.

9. Verification

9.1 Inputs

A verifier is given, each in a list: identity records; track records; origin claims (synthesis §3.1); reviews (synthesis §9.1); a label policy or none; a player's log or none (synthesis §10); the envelopes it holds, each named by its content hash; the generators and the identity priors it holds, each named by its SHA-256; and a policy of synthesis §11.1's members and one more:

member	values	default	meaning
generation	accept-unrecomputed, require-proof	accept-unrecomputed	require-proof: §9.2 step 3.10 applies (generation_not_proved)

9.2 Steps

The verifier runs these steps and stops at the first refusal, which it reports with the reason and where: a record by its list and index (["identity", i], ["track", i]), or synthesis's own place (["synthesis", …]).

1. Reading. Each identity record, then each track record, reads (§8.1 to §8.3) and its signature verifies: record_malformed, bad_key, bad_signature.

2. Identity records, in order: where the verifier holds a prior by identity, it reads (§7.3) (identity_invalid).

3. Track records, in order, each by these checks in order:

   1. Its identity record is one the verifier holds (identity_unknown).
   2. Its key is that record's (driver_key_mismatch).
   3. It is that face's first track record with seq 0 and prev null, or follows the face's track record before it in the list with seq one higher and prev its link hash (track_chain_broken).
   4. Its object is an envelope the verifier holds (object_unknown).
   5. The envelope is a WAI1 whose capability is wai.avatar.face and whose payload reads as a segment (§7.4) (payload_invalid).
   6. The envelope's conditioning.prior.sha256 is the payload's generator (generator_mismatch).
   7. The payload's identity is the identity record's (identity_mismatch).
   8. Where the verifier holds the generator, by the payload's generator, it reads (§7.2) and loads under acc64 (generator_invalid).
   9. With the generator and the prior held: the generator, the prior and the segment agree (§7.7) (shape_mismatch); the frames are generated, and where frames is given, each frame's SHA-256 is its entry (frame_mismatch).
   10. Under generation require-proof, a segment whose frames were not generated in step 9: generation_not_proved.

   A segment whose frames were generated is proved; its canonical buffer (§7.7) is the object's decoded samples for step 4. Otherwise it is not recomputed.

4. Synthesis. Synthesis §11.2 over the origin claims, the reviews, the label policy, the player's log, the envelopes, and the decoded samples of step 3, with the policy's members: a refusal there is this verification's, at ["synthesis", …].

5. Each segment's claim. For each track record in order: a claim whose subject is its object (claim_missing); the first such claim is of class synthesized, step generate, names the payload's generator among its models, and has the track record's link hash as record (claim_mismatch).

9.3 What an accepted case states

• for each identity record, in order: its link hash, its identity, and read where the verifier held the prior, not-held otherwise;
• for each track record, in order: its link hash, its object, its frames' count, proved or not-recomputed, and for a proved segment the SHA-256 of its canonical buffer, null otherwise;
• synthesis's statements: for each origin claim, in order, its link hash and class; each label due and the shown entry that met it; each review presented; the EU icon the label policy shows, where it shows one (synthesis §11.3).

10. Reasons

The codes a verification refuses with, each with where it is raised:

reason	raised at
record_malformed	§8.1, §9.2 step 1
bad_key	§8.1
bad_signature	§8.1
identity_invalid	§7.3, §9.2 step 2
identity_unknown	§9.2 step 3.1
driver_key_mismatch	§8.3, §9.2 step 3.2
track_chain_broken	§8.3, §9.2 step 3.3
object_unknown	§9.2 step 3.4
payload_invalid	§7.4, §9.2 step 3.5
generator_mismatch	§7.4, §9.2 step 3.6
identity_mismatch	§8.3, §9.2 step 3.7
generator_invalid	§7.2, §9.2 step 3.8
shape_mismatch	§7.7, §9.2 step 3.9
frame_mismatch	§8.3, §9.2 step 3.9
generation_not_proved	§9.1, §9.2 step 3.10
claim_missing	§8.4, §9.2 step 5
claim_mismatch	§8.4, §9.2 step 5

A verifier also raises every code synthesis raises (synthesis §12), at ["synthesis", …]: class_understated, present_output_irreproducible and label_not_shown among them.

The codes a file is refused with as it is read (§7.2 to §7.4), which §9.2 reports as payload_invalid, identity_invalid or generator_invalid:

code	raised at
bad_magic	§7.2, §7.3, §7.4
version	§7.2, §7.3, §7.4
field_range	§7.2, §7.3, §7.4
truncated	§7.2, §7.3, §7.4
trailing	§7.2, §7.3, §7.4
noncanonical	§7.4

11. Mappings to other documents

Each mapping below states what WAI's elements are in the other document's terms, read from the text named, and marks what that text leaves open. None changes this extension.

11.1 ISO/IEC 23090-39, the Avatar Representation Format

ISO/IEC 23090-39 was at the Draft International Standard stage when this revision was written, and its text is sold, not openly readable; it was not read. What is mapped here is what two openly readable documents publish about it: 3GPP TR 26.813 V20.0.0 §6.3.4, which summarises its data model and its animation stream, and draft-ietf-avtcore-rtp-avatar-02 §4, which summarises its animation units. Every field width, code value and URN namespace of ISO/IEC 23090-39 is therefore open here, and the rows mark where the mapping depends on one.

WAI	the Avatar Representation Format, as those documents describe it	mapping	left open by the text read
the identity prior (§7.3)	a base avatar: an ARF document in a container (an ISOBMFF or a ZIP container), its components held as data items	the prior is one data item of the container, named by its SHA-256	how a two-dimensional picture and its keypoints are declared as a component
the generator (§7.2)	the preamble's proprietaryAnimations: an animation scheme named by a URN, with data items that carry its model or configuration, "for example ML-based reconstruction models"	the generator is such a data item, pinned by its SHA-256; the scheme is wai.avatar.face under the lowering face-int/1	the URN namespace; how a data item is typed
the identity record (§8.2)	the preamble's authenticationFeatures: encrypted face and voice feature vectors and a URI to a public key, which bind the avatar to its owner	WAI binds the prior to a key by a signature over the prior's digest, and carries no feature vector; the record's key is the key such a URI would name	the vectors' form; the encryption
a frame of a segment (§7.4)	a landmark animation unit, AAU_LANDMARK: a landmark set, flags for velocity and confidence, a dimensionality (two or three), a count, and for each landmark its index and coordinates	one frame is one unit: the face's K keypoints as a two-dimensional landmark set, indices 0 to K − 1, no velocity, no confidence, each coordinate c sixteenths of a pixel of the generator's frame written as c / 16 pixels. Each such value is a dyadic number below 2^11 in magnitude with at most four fractional bits, which a binary floating-point coordinate of at least 16 significand bits holds exactly. Read back, a coordinate in pixels becomes sixteenths, rounded half to even, and then a multiple of the segment's step	the coordinates' type and width; how a landmark set is defined in the container
a segment's timing (§7.4)	each unit's 32-bit timestamp in ticks; the configuration unit's timescale, ticks a second, a 32-bit float	with a timescale of rate_num ticks a second (an integer below 2^24, which a 32-bit float holds exactly), frame i of a segment has the timestamp (first + i) · rate_den	the timestamp's wrap; how a profile string is chosen
the frames (§7.7)	none: the animation stream drives a base avatar that a receiver renders by its own means	the frames are not carried; every sink generates them, the same bytes	—

So a WAI segment converts to landmark units and back without loss where the unit's coordinate type holds the values above; which type it is, is the open question. The track record, the claim and face-equivalence rest on the segment's envelope, whose content hash a track record names: a party that rebuilds a segment from units must rebuild the same bytes (step, order, first and the rate are fields a unit does not carry), so WAI's provenance travels with the segment, not with the units.

11.2 draft-ietf-avtcore-rtp-avatar-02

The draft carries animation units in RTP, one unit per packet, a fragment of one, or several in an aggregation packet, under a two-byte payload header: D (dependency), UT (the unit type; 4 is a landmark unit), L (a level of detail) and AvID (an avatar's identifier in the session), and signals the media type application/ampg with the optional parameters version, frameworks, avatar-ids and avatar-lods. WAI defines no transport (the preamble of moq-streaming-format); this is how its elements sit in that carriage:

the draft	WAI
a single-unit packet of UT 4 (§5.4.2)	one frame of a segment as a landmark unit (§11.1)
a multi-time aggregation packet, UT 14 (§5.4.4)	the frames of one segment, each its unit, with its timestamp offset from the first
D	0: each landmark unit holds its frame's keypoints in full; the deltas of §7.4 stay inside a segment's payload, never across units
L	one level of detail: a generator generates at one size, width × height
AvID and avatar-ids	the face: the value an avatar-ids entry maps an AvID to is application dependent (§7.2 of the draft), and a WAI session maps it to the identity record's link hash, which names the prior
frameworks	a URN for the face's keypoint scheme; WAI registers none in this revision
the RTP timestamp, at the clock rate the SDP gives and the configuration unit's timescale	§11.1's timestamp, with rate_num ticks a second
§11's security considerations, which leave integrity and authenticity to the application	the track records and the claims, which sign the segment object, not the packets; carried with RTP they go beside the units, as the segment envelopes do

A receiver that holds only the units has the keypoints and can generate the frames, the same bytes, given the generator and the prior; it cannot check a track record or a claim, which name a segment's envelope.

11.3 3GPP TR 26.813 V20.0.0

The technical report is openly readable, and was read in V20.0.0.

the report	WAI
§6.5.3: a two-dimensional avatar's base is a neural network and a base image; a robust model animates characters other than those it was trained on, a personalized model one character	the generator is a model and the identity prior the base image; the registered generator is a robust model in that sense (§7.9: its measured faces were held out of its training); a personalized model is a generator trained for one face, pinned by its digest the same way
§6.5.3: the model is stored in an exchangeable format with metadata (identifier, input and output size, framework and version)	the generator is one file of integers, pinned by SHA-256, and its network is one graph of wai-ops/1 that every receiver runs to the same bytes (§7.6, §7.8)
§6.5.4: inference is driven by the user's video, audio or text, and its output is encoded with a video codec and streamed	the driving input is keypoints the sender extracts from its video; the inference moves to every receiver, and what is streamed is the keypoint track (§7.9 measures its bitrate)
§8.2.1: animation samples in a data channel's message envelope, whose message types are defined for blend-shape and joint samples; "for other animation samples, a dedicated message type needs to be defined"	WAI defines no message type; a segment envelope is the unit such a message would carry
§8.2.2: RTP carriage by draft-ietf-avtcore-rtp-avatar	§11.2
§8.3.4.2: an Avatar Representation Assertion, a credential that binds an avatar representation, by a hash over the container's graph, to a user's key pair, issued by a trusted issuer	the identity record binds the prior, by its SHA-256, to the participant's key, signed by that key; an issuer's assertion of who holds the key is outside WAI, where jwp-receipts §7's key discovery serves
§9.2.2: receivers extract facial features from the user's video and compare them with features stored in the base avatar, to detect impersonation	a receiver checks that the frames are the generation of the committed prior under the participant's key (§9.2); nothing is extracted or compared
§9A.1: the evaluation framework's reproducibility rests on fixed content and deterministic rendering conditions	face-equivalence makes the rendered frames themselves reproducible: a test bed's output is a digest every implementation reproduces
§9A.3.1 and §9A.3.2.1: PSNR and SSIM against a reference, and the Lip Vertex Error, the RMS error of the mouth's landmarks	§7.9 reports PSNR and SSIM, and the RMS error of a keypoint detector's reading over all keypoints and over the mouth's four
§10.2, recommendation 3.b: study compression of animation streams	§7.4's coding, and §7.9's measured bitrate

12. Conformance

live-conformance/avatar-face/ holds the corpus. Its inputs, each pinned by SHA-256 in the generator and in vectors.json, are written by tools/wai_avatar_face_train.py (§7.9): generator.wfg, the registered generator; the identity priors identities/face1.wfi and identities/face2.wfi, faces the tool generated and held out of training; their keypoint tracks tracks/face1.json and tracks/face2.json, in sixteenths of a pixel; and training.json, the tool's report. wai_avatar_face_vectors gen writes the rest from fixed seeds: seeded.wfg, a generator of three keypoints at 16 × 16 drawn from SHA-256 of a label and trained on nothing; seeded.wfi, a picture that is an integer function of a label; and vectors.json, which holds:

• the mask's table of §7.6: its SHA-256, each entry a 16-bit little-endian signed integer, T[0] first, and its entries at some values;
• forms: files of §7.2 to §7.4 and how each reads, its fields or its refusal, every code of §10's second table at least once;
• frames: for the seeded generator and identity at three sets of keypoints, and for the registered generator over each frame of both tracks, the SHA-256 of the warped picture, of the maps (each value a 16-bit little-endian unsigned integer, in [K][h][w] order) and of the frame, and for the seeded ones the warped picture and the frame in full;
• cases of §9.1, each with its outcome, and, accepted, everything §9.3 states; refused, its reason and its place. Accepted: one participant's two segments, proved, presented with the label at first exposure and within the interval; the same with nothing held, not recomputed; two participants each driving her own face; a track record that states no frames. Refused: each code of §10's first table, and synthesis's class_understated (a segment claimed as a capture), present_output_irreproducible (a player that presents the warped pictures) and label_not_shown.

The segments the cases carry round each keypoint of the tracks to a quarter of a pixel (step 4). A conforming implementation gives every vector what the corpus states. wai_avatar_face_vectors verify and the test tests/avatar_face.rs check that every file but the inputs regenerates byte for byte, that wai-rs gives every vector what it states, that §10 lists exactly the reference's codes, and that §7.9's figures are training.json's. live-conformance/avatar_face_verify.py is the second implementation, written from this text: it shares no code with wai-rs; it reads the files and runs §7.5 and §7.6 in Python's integers, operator by operator; it verifies the claims, the log and the label policy with the synthesis corpus's second implementation (synthesis_verify.py), every signature with WAI's strict rule (claim-conformance/_ed25519.py) and every link hash with BLAKE3 written from its specification (claim-conformance/_blake3.py); it reads §10's codes from this text; and it mutates copies of the corpus and requires every copy to be refused.

13. Security and privacy considerations

• A frame is a function of the generator, the prior and the keypoints, and nothing else. A proved segment shows that the frames are what those files give, not that the person in front of the sender's camera looked so: the keypoints are the sender's statement of where the features were, and the generator adds what it learned.
• The identity record and the track records bind a face to a key. They show that the holder of that key drove the face, not who the holder is: binding a key to a person is jwp-receipts §7's key discovery, or a credential outside WAI. A party that obtains an identity prior can commit to it under a key of its own; what the records then show is that this other key drove the face, which is what a verifier checks against the key it expects. A verifier that knows the participant's key refuses a track signed by another (driver_key_mismatch).
• An identity prior is a picture of a face. It is fetched by its digest from whoever serves it, and anyone who holds it can generate the face from any keypoints. A service that does not want a prior to be held outside a session keeps it in the session's confidentiality (confidentiality); the digest in a record names the picture and lets a party that holds the picture recognise it.
• A keypoint track describes a person's expressions and movements. Where processing them needs a lawful basis, the service records it as a lawful-basis claim (jwp-receipts §2.4.6).
• Every frame is synthesized, and is labelled as such (§8.4, §8.5). A party that strips the claim and the label from the frames is caught only where a verifier receives them through the segment, whose envelope generates (synthesis §3.3); this revision applies no watermark to the frames (synthesis §7).

Appendix

An avatar is the highest-bandwidth capture reduced to its instructions: a body in motion is a rig plus a stream of joint angles, and the standard carries the rig and the angles, not the pixels. Pose-equivalence makes the motion auditable — two parties to a call hold the same signed skeleton — while the appearance stays the sink's, skinned from a shared mesh. Sight (worlds), sound (mixdown), touch (sample), live state (feed), and now a moving body (pose) are one floor, one verification.