open-standards/wai/extensions/opus-dred.md
Transaction Science cc253b7353 wai: register wai.audio.opus.dred and .dred.int; extension opus-dred, corpus, second implementation, CI
The deep redundancy of draft-ietf-mlcodec-opus-dred-07 registered as two
companion-only enhancement layers over wai.audio.opus.packet: the draft's
decoding within its tolerances, and the integer profile dred-int/1. The
extension text (registration, finding DRED, the profile, a stream under
loss, the measurement, receipts and the synthesis lattice, refusals), SPEC
§5 and §7, the registry export, the research registry's basis and licence
note, the corpus live-conformance/dred/ with its fetch script, its second
implementation dred_verify.py, CI steps, the CHANGELOG, and the proposal
to the working group, drafted and not filed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-22 04:34:29 -04:00

37 KiB
Raw Permalink Blame History

WAI Extension: Opus Deep Redundancy, Decoded Exactly

Status: Draft. Registers two companion-only capabilities for the deep redundancy (DRED) of draft-ietf-mlcodec-opus-dred-07: a compact neural latent of the recent speech, carried in an Opus packet's padding, from which a receiver rebuilds the acoustic features of packets that were lost. wai.audio.opus.dred is that decoding as the draft defines it, held to the draft's own conformance test (its §4.1), which admits any decoder whose features lie within its thresholds of the reference. wai.audio.opus.dred.int is the same decoding through an integer profile, dred-int/1: the draft's own decoder network and its own weights file, pinned by digest, lowered to wai-ops/1 graphs (integer-operators) whose output is the same integers on every machine and on every fabric whose operator implementation conforms. With it, a stream under loss has one reconstruction, named by a digest that a verifier recomputes from the packets, the loss and the weights, and a player can sign that digest in the log of synthesis §10.3. This extension also says where the working group's other documents stand in WAI's registry and in synthesis's lattice (§2.3, §7.1), and measures the profile against the floating-point network (§6). No new transport. The reference implementation is the dred_exact module of wai-rs (feature dred_exact); the corpus is live-conformance/dred/, and its second implementation is live-conformance/dred_verify.py. The integer profile as a proposal to the working group is drafted in proposals/dred-integer-profile.md, which is not filed. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Texts read for this revision, each pinned to the version named: draft-ietf-mlcodec-opus-dred-07 (4 August 2026), with the files its §7 references by SHA-256 ([DRED-Weights], [DRED-Test-Vectors], [DRED-Compare], [DRED-Ref-Impl]); draft-ietf-mlcodec-opus-extension-06; draft-ietf-mlcodec-opus-speech-coding-enhancement-04; draft-ietf-mlcodec-opus-scalable-quality-extension-02; draft-ietf-mlcodec-test-battery-00; RFC 6716 §3, §4.1 and §5.1; RFC 7587.

1. Scope

The draft's decoder has two parts (its §4): a feature decoder, a recurrent network that turns the latents into 20 acoustic features per 10 ms, which the draft makes normative and tests within tolerances; and a vocoder that turns features into samples, which the draft leaves to the implementation (its §2.2.4, §4.2). This extension's profile is of the first part.

Conformance targets.

  • A sink that advertises wai.audio.opus.dred.int finds DRED in a packet (§3), decodes it through dred-int/1 (§4), and gives exactly the features §4 gives.
  • A verifier of a reconstruction recomputes a stream's reconstruction from its packets, its loss pattern and the weights, and checks a record of it (§5) and a player's statement of it (§7.2).

Out of scope.

  • The vocoder. A sink renders features to samples with any vocoder the draft admits; those samples are its own, and §7.2 says what a player states of them.
  • The decoding of an Opus packet's own frames (RFC 6716 §4), and concealment of a frame no DRED rebuilds. §7.1 classes both.
  • Encoding. The draft's encoder is not normative (its §2.2.1); the corpus's packets are written by a generator that quantizes as §9 describes.

2. Registration

2.1 Two strings, one payload

string kind role replicate class parameter set
wai.audio.opus.dred companion enhancement layer over wai.audio.opus.packet StandardTolerance: features within the draft's §4.1 thresholds of the reference the draft's, fixed by the draft
wai.audio.opus.dred.int companion enhancement layer over wai.audio.opus.packet NeuralIntegerExact: the features of §4, exactly one file, the draft's weights file (§4.1)

Both carry the same payload. They differ in what a sink that advertises them promises, which is what a sender chooses between: a sender that needs any rebuilt speech names the first; a sender whose reconstruction is to be recomputed, signed or presented at decode-equivalence names the second, with a pin on the weights file (SPEC §3.1) that declares tier decode-equivalence. Two strings are needed because a capability's class bounds the tier a pin may claim (SPEC §3.1, Tier): a decode admitted within tolerances cannot back replicate, and the draft's decoding is admitted within tolerances.

2.2 Why a companion, not a refinement

A refinement-only capability is applied to the output of the stage below it, bound to that output's producer by content hash (staged-delivery): its payload corrects what the base decoded. DRED corrects nothing the base decoded: it is a second, coarser encoding of the same speech, sent ahead so that a receiver can rebuild what a loss took, and it is read only where the base's own frames are missing. Its payload has no meaning without the encode it was made with: its offsets index that encode's frames (draft §3), and its first rebuilt samples continue that encode's last decoded ones (draft §2.2.4). That is the model of an enhancement layer (SPEC §5): a companion-only capability, decoded by its own standard over one base encode, bound to exactly one selectable rendition of a base it lists, and the base of no second layer. So both strings are registered as enhancement layers. Their one base is wai.audio.opus.packet, one Opus packet per object, because there the envelope itself aligns the DRED payload with the packet it belongs to; this revision pins no alignment of a separate DRED stream with an Ogg Opus stream (wai.audio.opus), as SPEC §5 pins none for a separately carried LCEVC stream.

DRED is carried one of two ways, as an enhancement layer is (SPEC §5, Enhancement layers):

  • Embedded. The DRED extension rides in the base packet's padding (draft §3), under extension ID 32, or, until the IANA action the draft asks for, under the experimental ID 126 with the two-byte prefix 'D', 12 (draft §3, §5). The payload is the base's bytes, under the base's capability; a sink without DRED discards the padding (draft-ietf-mlcodec-opus-extension-06 §2). No DRED component exists.
  • Separate. A WAI2 companion of the base rendition (SPEC §7.1), role "companion", companion_of wai.audio.opus.packet. Its payload is one DRED extension payload as draft §3 defines it (the bytes after the extension's ID byte and any length, with no experimental prefix), for frame 0 of the base packet in the same envelope (a frame offset of 0, §3.3).

2.3 The working group's other documents

  • draft-ietf-mlcodec-opus-extension-06 defines the padding's extension format that §3.2 reads. It carries no payload of its own and registers no capability here.
  • draft-ietf-mlcodec-opus-speech-coding-enhancement-04 sets requirements for a decoder-side enhancement of the Silk layer's output, and for a blind extension of its bandwidth above 8 kHz. It changes no byte of a packet, so it is no WAI capability: it is a property of the sink's Opus decoder, and a stream decoded with it is still wai.audio.opus or wai.audio.opus.packet, decoded within RFC 6716 §6's tolerances by a decoder that also passes that draft's objective tests. §7.1 classes what it does to an object.
  • draft-ietf-mlcodec-opus-scalable-quality-extension-02 adds refinement bits for the CELT layer, and bands above 20 kHz, in the same padding. It is not a neural decoder, and the refinement rides embedded in the base packets, decoded by the base's decoder; this revision registers no string for it.
  • draft-ietf-mlcodec-test-battery-00 describes test material for these extensions. WAI reads it as informative.

2.4 Licence class

Both strings are classed patent-pool, the conservative class REGISTRY.md gives a capability whose royalty status is not established. One IPR disclosure on the individual draft that became draft-ietf-mlcodec-opus-dred, the IETF's disclosure 6063, states an intent to license royalty-free if the technology is included in a standard the IETF adopts; until that happens the status is not established. A later revision MAY move both strings to royalty-free once it is.

3. Finding DRED and its integers

3.1 The packet

A sink reads an Opus packet's framing as RFC 6716 §3.2 defines it, and holds it to §3.4's rules R1–R7. A packet that breaks one is not a valid Opus packet: nothing in it is read, and its DRED is refused (packet_malformed). From a valid packet it takes the number of frames, each frame's duration in units of 2.5 ms (RFC 6716 §3.1: 4, 8, 16 or 24 for configurations 0–11 by configuration modulo 4; 4 or 8 for 12–15 by configuration modulo 2; 1, 2, 4 or 8 for 16–31 by configuration modulo 4), and the padding of a code-3 packet.

3.2 The extensions

The padding's extensions are read as draft-ietf-mlcodec-opus-extension-06 §2 defines them, including the frame separator (ID 1) and the repeat mechanism (ID 2, §2.3), in bitstream order, each instance with its frame index. Iteration stops at the first extension whose length, or whose frame index, would read past the padding or the packet's last frame (§2.7); the instances before it stand.

The packet's DRED payload is the first instance, in that order, that is either:

  • ID 32 with at least one byte of data: the data; or
  • ID 126 whose data is longer than two bytes and starts with 'D' (0x44), 12: the data after those two bytes.

Its frame offset is its instance's frame index times the frame duration, in units of 2.5 ms. A packet with no such instance carries no DRED. (The reference implementation the draft names in its Appendix B finds DRED this way, under ID 126.)

3.3 The payload

The payload is decoded with the range decoder of RFC 6716 §4.1, as draft §3 and §3.1 define it. This section fixes the details the draft's prose leaves open, as the draft's reference decoder does; where the two could be read apart, the reference decoder's reading is this profile's.

  1. The header: q0 = uint(16), dQ = uint(8), X = uint(2); if X = 1, an extended offset E = uint(256), else E = 0; an indicator I = uint(32). uint(n) is ec_dec_uint of RFC 6716 §4.1.5 for n ≤ 256, one symbol of n equally likely values. encoded_offset = 32·E + I.
  2. qmax: 15, unless q0 < 14 and dQ ≠ 0. Then, with n = 14 − q0, a symbol is decoded over a total of 2n (ec_decode): a value s ≥ n gives qmax = q0 + (s − n) + 1 and is consumed as [s, s + 1); a value s < n gives 15 and is consumed as [0, n).
  3. The offset, dred_offset = 16 − encoded_offset + frame_offset, in units of 2.5 ms: the newest feature vector the payload rebuilds ends this long after the start of the frame that carries the extension.
  4. The initial state: 50 coefficients k = 0 … 49, at quantizer q0, with the state tables (draft Tables 6, 8 and 9). Then latent vectors l = 0, 1, …, newest first, each of 25 coefficients at quantizer q_l = min(qmax, q0 + ⌊(D[dQ]·l + 8) / 16⌋) with D = [0, 2, 3, 4, 6, 8, 12, 16], the slope table of draft §3 in sixteenths, with the latent tables (Tables 2, 4 and 5). Before each latent vector, if 8·len − tell ≤ 7, where len is the payload's length in bytes and tell is ec_tell (RFC 6716 §4.1.6.1), no more are decoded.
  5. A coefficient with decay r and zero probability p0 (the tables' entries for its coefficient and quantizer): if r = 0 or p0 = 255, it is 0 and nothing is decoded. Otherwise, with P = 128·p0 and S = 32768 − P, a symbol is decoded by ec_dec_icdf16 (with ftb = 15) against the table [S, ⌊S/2⌋, 0]: 0 is a zero coefficient, 1 positive, 2 negative. A non-zero coefficient's magnitude is then 1 + v_1 + v_2 + …, each v_j decoded by ec_dec_icdf16 against icdf[0] = max(7, 128·r), icdf[i] = max(7 − i, ⌊icdf[i − 1]·128·r / 2^15⌋) for i = 1 … 6, icdf[7] = 0, while the symbol just decoded is 7. When the magnitude being summed passes 2^16, the payload is refused (dred_latent_range): the draft's escape rule bounds nothing, and past a payload's last byte the decoder reads zero bytes (RFC 6716 §4.1.2), so a crafted payload could otherwise keep a decoder reading.

The draft's test bit files carry payloads with no header (each record gives q0, with no slope, and a count of latents); §6 reads them so.

4. The integer profile dred-int/1

4.1 The parameter set

The parameter set of wai.audio.opus.dred.int is one file: the decoder weights file draft §2.2.2.1 and §7.1 name ([DRED-Weights]), whose SHA-256 is 394f0ecf19926cebd678019dc94a54c88ae078340c16325069707f286cdf28a6. A sink MUST check the file against that digest before it reads it, and MUST refuse any other file (weights_digest_mismatch): the draft makes another set of weights non-conforming (its §4.1, "an alternate DNN model is not allowed"), and §4.4 makes these weights' integers a function of this file's bytes. A pin over the set names that digest.

The file is read as the draft's Appendix A defines it. The profile reads, for each layer of the table below, the arrays dec_<layer>_weights_float and dec_<layer>_bias, and for a block-sparse layer dec_<layer>_weights_idx; it reads no other array. A dense matrix is stored input-major (weights[in·outputs + out], A.3.1). A block-sparse one (A.3.2) is a run of groups of 8 outputs, each a count of blocks and then each block's first input (a multiple of 4), and its weights are blocks of 8 outputs by 4 inputs stored input-major inside the block (w[in·8 + out]); every position no block covers is 0.

layer inputs outputs sparse
dense1 26 96 no
hidden_init 50 128 no
gru_init 128 320 yes
gru1_input … gru5_input 96, 192, 288, 384, 480 192 yes
gru1_recurrent … gru5_recurrent 64 192 no
glu1 … glu5 64 64 no
conv_dense1 … conv_dense5 160, 256, 352, 448, 544 32 yes
conv1 … conv5 64 32 no
output 576 80 yes

The weights are not in this repository. The corpus names the file by digest, and its fetch script (live-conformance/dred/fetch.sh) fetches it from the location the draft gives and checks the digest.

4.2 The tables

The decoder reads draft Tables 2, 4, 5, 6, 8 and 9 (scale, decay and P(0), for the latents and for the state). Serialized row by row (coefficient k from 0, the sixteen quantizers in order, one byte each), in the order latent scale, latent decay, latent P(0), state scale, state decay, state P(0), their SHA-256 is 03f7d6b22e2aca5f5a28525a1311e1113d94b917fbc74a5493b04bbed5c97ded. An implementation that transcribes them checks its transcription against this digest.

4.3 The inputs

Every graph input is an integer at 2^16 (Q16). A coefficient x of the initial state, with state scale s at q0, and a coefficient of latent l, with latent scale s at q_l, stand for x·256 / max(1, s) (draft §3.1), which the input holds as ⌊(2·x·2^24 + s′) / (2·s′)⌋ with s′ = max(1, s): the quotient rounded to the nearest integer, a tie upward. A latent's 26th input is q_l/8 − 1 (draft §2.2.2), held as (q_l − 8)·2^13.

4.4 Weights to integers

Every weight becomes an integer at 2^24, and every bias an integer at the scale of the accumulator it joins: 2^40 for dense1 and hidden_init, whose inputs are at 2^16, and 2^39 for every other layer, whose inputs are activations at 2^15. Each is computed from the float32 value's bits alone, with no floating-point arithmetic:

  • a float32 with sign σ, exponent field e and fraction field m is (−1)^σ·M·2^E, with M = m, E = −149 when e = 0, and M = m + 2^23, E = e − 150 otherwise;
  • at scale 2^f, its integer is (−1)^σ·R_half_even(M, −(E + f)) when E + f < 0, and (−1)^σ·M·2^(E + f) otherwise, with R_half_even of integer-operators §2 (0 for a shift of 63 or more).

A value that is not finite (e = 255), a weight whose integer does not fit i32, a missing array, an array of the wrong length, a block that starts at an input that is not a multiple of 4 or past the layer's inputs, a group that names one block twice, or a record the file format refuses, makes the set fail to lower (model_mismatch, integer-operators §10.5). The pinned file does none of these.

4.5 Activations

Activations are integers at 2^15 (Q15), in [−32767, 32767]. tanh and the sigmoid are lut16 tables (integer-operators §5.6), read at 2^12 and 2^11. Their entries are this integer computation, not the functions' exact values:

  • C_0 = 4609434768280048980 (⌊e^(−1/2048)·2^62⌋), and C_(k+1) = ⌊C_k² / 2^62⌋;
  • for 0 ≤ u < 2^15, X(u) starts at 2^62, and for each bit k of u from bit 0 to bit 14 that is set, in that order, X ← ⌊X·C_k / 2^62⌋;
  • tanh: entry v + 32768, for v in −32767 … 32767, is sgn(v)·min(32767, ⌊(2a + b) / (2b)⌋) with a = 2^15·(2^62 − X(|v|)) and b = 2^62 + X(|v|);
  • sigmoid: entry v + 32768 is min(32767, ⌊(2a + b) / (2b)⌋) with b = 2^62 + X(|v|), and a = 2^77 for v ≥ 0, a = 2^15·X(|v|) for v < 0;
  • entry 0 (v = −32768) of each is 0, as lut16 requires. No certified input reaches it.

Every intermediate fits 128 bits. Serialized as their 65,536 entries in order, each a 16-bit two's-complement little-endian integer, the tables' SHA-256 are aea887ef206a10ea37bd31ef310a75c2ff05ca17df7edbbc15916969bccb4c22 (tanh) and 1fe2d7c6b3255127f612576093f2f50e9e13d88da546ad4c04de264e2bbc8c36 (sigmoid).

4.6 The graphs

dred-int/1 lowers the set to two WHG1 graphs (integer-operators §10), init and step, in which every tensor is [C, 1, 1]. Below, dense(x; L, s) is a conv with kh = kw = 1 and groups = 1 over x, with layer L's integer weights (w_bytes 4), and an RQ with no multiplier, one shift s, L's integer bias placed pre (b_bytes 8) and round half_up; T_tanh and T_sig are lut16 with the tables of §4.5; C(·) marks an output tensor whose policy is clamp [−32767, 32767] (every table input, and every GRU state); mul(a, b) is mul with an RQ of shift 15, no bias, half_up; [a, b] is concat; every other tensor's policy is derived.

init. Input 0 is the initial state [50], policy clamp [−(2^31 − 1), 2^31 − 1].

  1. a = T_tanh(C(dense(IS; hidden_init, 28))) [128];
  2. s = T_tanh(C(dense(a; gru_init, 27))) [320];
  3. outputs: the five GRU states h_k = split(s, 64·(k − 1), 64), k = 1 … 5.

step. Inputs: the latent Z [26], policy clamp [−(2^31 − 1), 2^31 − 1]; the GRU states h_1 … h_5 [64]; the convolution memories c_1 … c_5 [32]; each of the last ten with policy clamp [−32767, 32767]. Then:

  1. t = T_tanh(C(dense(Z; dense1, 28))) [96].
  2. For k = 1 … 5, with x = t, h = h_k, I = gru<k>_input and U = gru<k>_recurrent:
    • the gates: g = T_sig(C(conv([x, h]))), [128], where the conv is dense with shift 28 whose output o < 128 has the weights of row o of I followed by those of row o of U, and the bias b_I[o] + b_U[o] (each converted by §4.4 on its own); z = split(g, 0, 64), r = split(g, 64, 64);
    • the candidate: p = dense(x; rows 128–191 of I, 19) + mul(r, dense(h; rows 128–191 of U, 19)) (add), at 2^20; then hc = T_tanh(C(rescale(p))), the rescale with one shift 8, no bias, half_up;
    • the state: h′ = C(hc + mul(z, h + rescale⁻(hc))), where rescale⁻ is rescale with one multiplier −1 (m_bytes 4), shift 0, no bias, floor, and each + is add. This is draft §2.2.2's z·h + (1 − z)·hc, written so that each product has one rounding; with 0 ≤ z < 2^15 its value lies between hc and h, so the clamp changes nothing;
    • the GLU: u = mul(h′, T_sig(C(dense(h′; glu<k>, 28))));
    • t ← [t, u];
    • the causal convolution: c = T_tanh(C(dense(t; conv_dense<k>, 27))) [32] and v = T_tanh(C(dense([c_k, c]; conv<k>, 27))) [32];
    • t ← [t, v]; the new h_k is h′ and the new c_k is c.
  3. x = dense(t; output, 23) [80], at 2^16, policy clamp [−2^31, 2^31 − 1].

Outputs, in order: x, the five new GRU states, the five new memories.

Tensors are numbered as integer-operators §13.0 numbers a lowered graph (inputs first, then each operator's output in operator order), every input declares max_h = max_w = 1, every exponent is none, and the operators are written in the order above (for the GRU: concat, the gates' conv, lut16, the two splits, the candidate's two convs, mul, add, rescale, lut16, then rescale⁻, add, mul, add). For the pinned set, the graphs' identities (the SHA-256 of their WHG1 bytes) are 6527304e48bbd6c972fb41cb24ab5f387a3adee6f88347dfc8f10f47060e5c9d (init) and 4c76f2e5f6b2657b985aa171f7872fc18ae48b9419447e69cb3e52a3225b34d2 (step).

4.7 The driver

A payload is decoded as a sequence of graph runs:

  1. init on the state input (§4.3); its outputs are the GRU states. The memories are 0.
  2. For latent l = 0, 1, …, newest first: step on the latent's input (§4.3), the GRU states and the memories; its first output x gives four feature vectors, and the others are the states and memories of the next latent.

The rows of the payload are those vectors, newest first: row 4l + j is x[20j], …, x[20j + 19] of latent l (draft §2.2.2: the four vectors of a latent are in reverse chronological order), and each row is 10 ms older than the row before it. Row 0 ends at the payload's offset (§3.3 step 3). Each value is the feature at 2^16, an i32.

Decoding the first n latents gives the first 4n rows, the same integers whatever follows: the network runs newest first, so a row never depends on an older latent. A sink MAY stop after the rows it needs.

4.8 Profile and fabrics

Every operator of both graphs is in wai-ops/1-i32 (integer-operators §9.1), and both graphs' certificates meet acc64 and not acc32: a 16-bit activation times a 32-bit weight, summed over up to 576 inputs, passes 32 bits. The registry's ops member of wai.audio.opus.dred.int is {"set": "wai-ops/1", "profile": "acc64", "subset": "full", "lowering": "dred-int/1"}; the payload's range decoder is RFC 6716's and not a stage operator of integer-operators §5E, so the member has no stage. On a GPU, the reference implementation's WGSL kernels (wai-ops-gpu) run both graphs at acc64, and wai-rs/tests/dred_exact_gpu.rs holds their outputs to the CPU's graph by graph.

5. A stream under loss

5.1 The loss pattern

A loss pattern is a seed of 32 bytes and two thresholds, enter and stay, each an integer in 0 … 2^16. Packet n (from 0) draws u_n, the first two bytes, big-endian, of SHA-256("wai:dred-loss\x01" ‖ seed ‖ n as a big-endian u32). It is lost when u_n < enter and the packet before it was received (or n = 0), and when u_n < stay and the packet before it was lost.

5.2 Slots

A stream is a run of packets of one frame each, all of duration d (in units of 2.5 ms), so packet n starts at n·d. Each lost packet's frame is two 10 ms slots, half 0 (the earlier) and half 1, and half h of frame m ends at E = m·d + 4·(h + 1). A maximal run of lost packets is recovered by the packet after it, when that packet was received and carries DRED whose payload is not refused (§3); then, with A = n·d + dred_offset the end of its row 0 (n the recovering packet), half h of frame m takes row r = ⌊(A − E) / 4⌋ of it when A − E ≥ 0 and r is less than four times the payload's latent count. Otherwise the slot is not covered: no DRED rebuilds it.

5.3 The reconstruction buffer

The reconstruction of a stream is its slots in order (frame ascending, then half), as bytes: for each, the frame (u32), the half (u8) and 1 if it is covered or 0 if not (u8); for a covered one then the recovering packet (u32), the row (u32) and the row's 20 features (i32 each); every integer little-endian. Its digest is SHA-256 over those bytes. The buffer of one run is the same over that run's slots alone.

5.4 The record

A record of a reconstruction states:

  • weights_sha256: the weights file the decode used;
  • lost: the indices of the packets the stream lost, ascending;
  • slots: each slot as [frame, half, packet, row], or [frame, half, null, null] for one not covered;
  • output: the reconstruction's digest, hexadecimal.

5.5 Verification

A verifier is given the stream (each received packet as the object a live track carries it in: a WAI1 envelope of wai.audio.opus.packet, with nothing where a packet was lost), its loss pattern, the record, and the weights file. It checks, in order, and refuses at the first check that fails:

  1. the record's weights_sha256 is the digest of §4.1 (weights_digest_mismatch);
  2. the packets the loss pattern loses are the record's lost, and are exactly the packets the stream lacks (loss_pattern_mismatch);
  3. the slots §5.2 gives are the record's slots (reconstruction_mismatch);
  4. the digest of the reconstruction §4 gives is the record's output (reconstruction_mismatch).

A reconstruction that passes is the one every conforming sink gives: the profile admits no other.

6. Measurement

What the profile changes is the arithmetic: the same network, the same weights, at fixed point. wai-rs/tests/dred_exact_measure.rs measures how far that moves the features, on the draft's own test vectors ([DRED-Test-Vectors]): all eight DRED bit files, 20 records each, every record's latents decoded through the profile, and through the network in floating point (f64, from the same float32 weights, with tanh and the sigmoid computed exactly to f64 precision); and every packet of the eight packet files that carries DRED, its DRED decoded both ways. Each pair is compared by the draft's feature comparison (its §4.1 and its tool [DRED-Compare]): per feature, the mean squared difference over the frames; total the square root of their sum, max of their largest, the pitch feature left out of both; pitch the mean absolute pitch difference over the frames the reference voices. The draft's DRED decoding test passes at total ≤ 0.5, max ≤ 0.15 and pitch ≤ 0.02. Two figures are added: the largest absolute difference of any feature but pitch, and a band log-spectral distance. The 18 cepstral features are the orthonormal DCT-II of the log10 band energies (draft §2.1.3), so the inverse DCT of their difference is the difference of the log band energies, and ten times its root mean square over the bands is a distance in decibels for each frame; its mean over the frames and its largest are given.

The reference features are the test vectors' own. "8-bit emulation" is the floating-point network computing each layer the weights file also holds 8-bit weights for as the reference implementation does when it reads them: each input rounded to ⌊0.5 + 127·x⌋, the 8-bit weights, each output scaled by its scale.

compared frames total max pitch largest difference band LSD mean (dB) band LSD max (dB)
integer, reference 16000 0.1314 0.0767 0.00492 0.4139 0.2945 1.0034
float, reference 16000 0.1314 0.0767 0.00492 0.4151 0.2944 1.0060
8-bit emulation, reference 16000 0.0831 0.0458 0.00255 0.2542 0.1844 0.6929
integer, float 16000 0.0021 0.0013 0.00011 0.0084 0.0048 0.0210
integer, float (packets) 80452 0.0022 0.0013 0.00004 0.0085 0.0049 0.0208

What the table shows:

  • The profile passes the draft's test against the draft's reference features, on each of the eight vectors and over all of them, with total, max and pitch well inside the thresholds.
  • The profile is close to the network. Against the floating-point network on the same latents, its features differ by total 0.0021, at most 0.0084 in any feature, 0.005 dB of band log-spectral distance on average: about one sixtieth of the distance between the network and the draft's reference features.
  • The reference features are not the network's. The floating-point network and the profile lie the same distance from them, and the 8-bit emulation lies closer, but not at them. So the draft's test vectors carry the arithmetic of the build that wrote them.

The last point was also checked once, informatively, on one arm64 host, by building the draft's reference implementation ([DRED-Ref-Impl]) twice: in its default configuration, which reads the 8-bit weights, and with its float weights, which it computes in float32 with a rational approximation of tanh. Over the eight vectors, the default build's features lie at total 0.090–0.096 from the test vectors and the float build's at 0.131–0.136; the profile and the floating-point network each lie at 0.015–0.018 from the float build, and at 0.129–0.135 from the default build. The two builds differ from each other by about 0.13 (on the three vectors compared directly), and neither gives the test vectors' features. Two builds of the draft's own decoder on one machine, and the build that wrote the test vectors, give three different sets of features from the same latents; under the integer profile there is one.

7. Receipts and the synthesis lattice

7.1 Classes

In synthesis's lattice (§3.2):

  • A reconstruction from DRED is a convert. The features are what the packets carried: a second encoding of the same speech, which a decoder with a pinned parameter set rebuilds (synthesis §3.2: "a decoder's reconstruction (neural or not)"). The vocoder that renders them continues that reconstruction; it generates no content the packets did not describe. So an object rebuilt from DRED keeps its input's class, reconstructed from a capture.
  • Concealment beyond DRED is a generate. Where no DRED covers a slot (§5.2), a sink that fills it with speech predicted by a model (packet loss concealment, which draft §2 names as what DRED's synthesis may continue from) produces content the packets never carried, by a trained model: that is generate, class synthesized, with the model in models. It is not assistive: synthesis's assist edits an input without substantially altering it, and a concealed slot has no input to edit.
  • Speech coding enhancement is an assist. A decoder-side enhancement of decoded speech (draft-ietf-mlcodec-opus-speech-coding-enhancement-04) edits one input without substantially altering it, and that draft's tests bound how far it may move the decoder's own output (its §3 and §4); a blind bandwidth extension adds content above 8 kHz the packets did not carry, as an upscale adds detail, which synthesis also classes assist (§3.4). Neither is a pinned derivation of synthesis §4, so their evidence states tier attest.

7.2 The presentation statement

A player that presents a reconstruction under the profile signs it in its log (synthesis §10.3): for each run of lost packets it presents, a present entry whose

  • subject is the content hash (jwp-receipts §2.1) of the recovering packet's object;
  • tier is decode-equivalence, decoder is wai.audio.opus.dred.int and variant is dred-int/1;
  • output is the digest of that run's reconstruction buffer (§5.3), origin and class as synthesis §10.3 has them (null where the player applies no claim).

A verifier checks the log as synthesis §10.3 does (record_malformed, bad_key, bad_signature, player_chain_broken), then each present entry: its decoder, variant and tier are those above, its subject is the content hash of the object that recovers a run of the stream, and its output is the digest of that run's buffer that §4 gives. Otherwise the entry states an output the object does not decode to (present_output_irreproducible, synthesis §9.2). The features are the statement's subject because they are what the profile fixes; a player that also states the digest of its vocoder's samples states its own output, which no other sink reproduces unless it runs the same vocoder, exactly.

8. Refusals

The codes this extension adds, each with where it is raised:

reason raised at
packet_malformed §3.1
dred_latent_range §3.3 step 5
weights_digest_mismatch §4.1, §5.5 step 1
loss_pattern_mismatch §5.5 step 2
reconstruction_mismatch §5.5 steps 3 and 4

A set that fails to lower or load is refused model_mismatch (integer-operators §10.5). A verifier of a presentation statement also raises synthesis's codes for a log (§7.2).

9. Conformance

live-conformance/dred/ holds the corpus, written from fixed seeds by wai_dred_vectors gen with the weights file of §4.1, which it names and does not hold: vectors.json, its README, and fetch.sh. It has:

  • The profile's identities: the tables' digest (§4.2), the constant and the two tables' digests (§4.5), and the two graphs' identities (§4.6).
  • Payload vectors, each a packet and what a decoder states of it: for a packet with DRED, its header, frame offset, offset, latent count, the digest of its integers, the digest of every row and row 0 itself; for one without, that it has none; for one refused, the code. The digest of a payload's integers is SHA-256 over q0, dQ, X, E, I and qmax, one byte each; the latent count (u32); the state's 50 indices (i32 each); then, for each latent, its quantizer (one byte) and its 25 indices (i32 each); little-endian. The digest of the rows is SHA-256 over every row in order, each feature an i32, little-endian. They cover the experimental ID with its prefix and ID 32; a slope with a qmax below 15 and an extended offset; a DRED extension on a packet's second frame after a frame separator; an experimental extension of another version, a truncated extension and plain padding, which carry none; two malformed packets; and a payload whose magnitude passes the cap.
  • Streams, each a run of packets as WAI1 objects with the lost ones absent, and a loss pattern, and cases over them, each a record, a player's log, and the outcome: the reconstruction as the profile gives it, with a log of its recoveries, for two streams (one with a slope and an offset that leaves the newest frames of every run uncovered); and one refused case for each check of §5.5 and §7.2: a record naming another weights file; a record listing a received packet as lost; a record naming a row one older; a digest with one bit changed; the digest of a floating-point decode of the same latents, rounded to 2^16; a log presenting that decode at decode-equivalence; and a log entry whose signature is broken.

Every packet of the corpus is written by its generator: its DRED indices are drawn from the draft's own distributions, by decoding a seeded byte stream with §3.3's coefficient rule at q0, and requantized to each chunk's quantizer (x·s_q / s_q0, rounded half up, where q codes the coefficient). The primary frames are empty (length 0, RFC 6716 §3.2.1). No recorded speech is in the corpus.

Feature equivalence. For each payload vector and each case, a conforming implementation reaches the outcome the vector states, and for an accepted one, the same digests, rows and statements, byte for byte.

On every fabric. wai-rs/tests/dred_exact_gpu.rs decodes the corpus's payloads with every graph on each GPU adapter found and holds the output to the CPU's.

10. Security and privacy considerations

  • Bounded work. A payload's work is bounded by its length and by the magnitude cap (§3.3): without the cap, the escape rule could read without end past the payload's last byte. A sink SHOULD also bound the latents it decodes to the span it needs (§4.7), as the draft lets it (its §3: "decoders MAY ignore any redundancy data beyond a certain amount").
  • No floating point in the decode. The profile's decode forms no float, so the denormal slow-downs draft §6 warns of do not arise in it.
  • Speech before joining. DRED can carry speech from before a receiver joined (draft §6). A reconstruction record names the packets and slots it rebuilt; a verifier given a room's history learns nothing from a record that its packets did not already carry, but a relay that forwards DRED forwards that speech, and the draft's ext32-dred-duration limits apply.
  • What a statement proves. A presentation statement over a reconstruction's digest proves which features the player's decode fixed, from packets the verifier holds. It does not prove that the packets are authentic or that the samples a listener heard are those features rendered; the object's own receipts (jwp-receipts) and the vocoder's exactness are separate matters.