open-standards/wai/extensions/neural-fields.md

WAI Extension: Integer Neural Fields

Status: Draft. Registers wai.field.int_grid, a computed capability: a volumetric scene held as a neural field, a multi-resolution integer feature grid and a small integer network, carried together in one payload, WIG1 (§3). The sink computes the content itself. It decodes the field to its values at any point of its domain, a density and a colour (§5), and renders it at any view by integer ray marching (§6); the payload names the view it is presented at (§4). Every value is an integer and every rounding is stated: a level's table is read through a direct index or a spatial hash with a stated wrap, the interpolation weights are fixed point, the network's layers round half up and clamp, and the ray march's samples, its stop and its compositing are pinned. So the values at a point and the render at a view are the same bytes on every machine (field-equivalence, §9), and a digest of a render can be signed and checked where it is presented. A payload carries the coarsest levels of its grid and declares the rest; each further level is a refinement, wai.field.int_refine (WIL1, §7), delivered as its own object under staged delivery in the level form, and a level the sink does not hold reads as zero, so the coarse field is presented first and every stage is presentable. The reference implementation is the int_field module of wai-rs (feature field_int); the encoder is tools/wai_int_field_export.py; the corpus is volumetric-conformance/fields/, and its second implementation is volumetric-conformance/field_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

1. Scope

A field maps a point of the unit cube [0, 1]³, its domain, to a density and a colour. A wai.field.int_grid payload holds:

• L grid levels, from coarse to fine. Level l divides each axis of the domain into R_l cells, so it has (R_l + 1)³ vertices, and holds a table of entries, each F integer features. A level whose vertices fit its table gives each vertex its own entry; a finer one shares entries among vertices through a spatial hash (§5.2);
• a network of N integer layers that reads the L · F features interpolated at a point and gives four values: the density and the three colour channels (§5.4);
• a view (§4): a camera, the frame's size, the samples a ray takes and a background colour.

A payload carries the coarsest K levels. Each of the others is a payload of its own, a WIL1 of wai.field.int_refine (§7.2), which refines the field by one level.

A sink decodes the payload once and then evaluates the field: at points (§5), and as a frame at the payload's view or at any other view (§6). The field is the content: the payload carries no pixels, and every frame is computed by the sink. A field decodes against no parameter set (SPEC §3.1): everything its decode reads is in its payload.

The arithmetic is that of integer-operators §2, and each layer of the network is that extension's conv (§5.2 there) with a 1 × 1 kernel, followed by its requantisation rq (§5.1 there) and relu or a clamp (§5.5 there), with the settings §5.4 below states. The camera is splat-exact §5.3's, from a pose of that extension's §2 members. The payload's framing, check order and refusal codes are integer-payloads §2's and §8's.

Out of scope. How the field was made (§10 describes one encoder, informatively); a colour that depends on the direction a point is seen from; a field whose domain is not the unit cube (a sink places the cube in its scene by its own transform, outside the decode); and how a sink presents a frame after it is computed.

2. Integer functions

⌊a / b⌋, R_half_up(p, s), isqrt, clamp and exp_neg_q16 are integer-operators §2's; rdiv(a, b) = ⌊(2a + b) / 2b⌋ for b > 0 is splat-exact §5.1's. a >> s is ⌊a / 2^s⌋. · and × are exact products, of integers or, between vectors, the dot and cross products. No formula wraps or saturates unless it says so. A value "at 2^k" is the integer v that stands for v / 2^k; a position in the domain is at 2^16, so the domain is [0, 2^16]³.

3. The field payload WIG1

3.1 Layout

Every multi-byte field is little-endian.

off	size	field	rule
0	4	magic	WIG1
4	1	version	1
5	1	L, the levels the field declares	1–16
6	1	K, the levels the payload carries	1–L
7	1	F, the features of an entry	1–8
8	1	T, the log2 of a level's table size	4–19
9	1	ffrac, the fraction bits of a feature and of an activation	8–16
10	1	wfrac, the fraction bits of a weight	0–15
11	1	N, the network's layers	1–8
12	4	reserved	0
16	2·L	R_0 … R_(L−1), each level's cells an axis, u16	each at least 1; strictly increasing
16 + 2·L	2·(N + 1)	the widths w_0 … w_N, u16	each 1–256; w_0 = L · F; w_N = 4
	56	the view (§4)	§4
		the network: for each layer j from 0 to N − 1, its weights, w_(j+1) rows of w_j i16 at 2^wfrac, row by row; then its biases, w_(j+1) i32 at 2^(ffrac + wfrac)
		the levels carried: for each level l from 0 to K − 1, its table, E_l entries of F i16 features at 2^ffrac, an entry's features together, entry by entry

A level's table has E_l = min((R_l + 1)³, 2^T) entries. The levels K to L − 1 are declared and not carried: each is a WIL1 of its own (§7).

3.2 Rules

The rules of the header's fields are the table's, read in this order: L, K, F, T, ffrac, wfrac, N; then each side R_l in order, at least 1 and greater than the one before; then the widths: each in 1–256, then w_0 = L · F (the network reads every level's features, so a field whose network reads another number of levels is refused), then w_N = 4; then the view (§4). A field that breaks one is refused (field_range). The weights, biases and features take any value of their type.

3.3 Caps and declared cost

The network's multiply-accumulates a point, Σ_j w_j · w_(j+1), are at most 2^16; a payload past it is refused (too_large). A level's table holds at most 2^19 · 8 features by the rules of §3.2.

The header declares the work of a render at a view of W × H pixels and S samples a ray: at most W · H · S evaluations of the field, each 8 · L · F features read and Σ_j w_j · w_(j+1) multiply-accumulates, and 3 · W · H bytes out. A sink MAY apply a limit to it and refuse a payload past the limit before it renders (cost_over_limit, which routes as integer-payloads §8 says).

4. The view

A view is 56 bytes:

off	size	field	rule
0	12	eye, three i32 at 2^16	each magnitude at most 2^24
12	12	target, three i32 at 2^16	each magnitude at most 2^24
24	12	up, three i32 at 2^16	each magnitude at most 2^24
36	2	focal, the focal length in pixels, u16	1–4096
38	2	W, the frame's width in pixels, u16	1–1024
40	2	H, the frame's height in pixels, u16	1–1024
42	2	S, the samples a ray takes, u16	1–1024
44	4	t_near, i32 at 2^16	0–2^22
48	4	t_far, i32 at 2^16	greater than t_near; at most 2^22
52	3	bg, the background, RGB8	any
55	1	reserved	0

eye, target, up, focal, W and H are a pose of splat-exact §2, in the field's domain. The rules are read in the table's order (eye, target and up component by component), and then the pose MUST NOT be degenerate: n_f and n_r of §6.1 are not 0. A view that breaks one is refused (field_range).

The payload's view is the one the field is presented at. A sink MAY render the field at any other view that meets these rules (an interactive one, say); it is computed by §6 as the payload's is, so every conforming sink computes the same frame there too.

5. Values at a point

5.1 The domain

A point p = (p_x, p_y, p_z) is at 2^16. A point with a coordinate outside [0, 2^16] is empty: its values are (0, 0, 0, 0), and nothing below is computed for it. The faces of the domain are inside it.

5.2 The table index

At level l, with R = R_l, vertex (x, y, z) (each in 0..=R) reads one entry of the level's table:

• if (R + 1)³ ≤ 2^T, entry x + (R + 1) · (y + (R + 1) · z): every vertex has its own;

• otherwise entry h mod 2^T, where

  h = (x · 1 mod 2^32)  XOR  (y · 2654435761 mod 2^32)  XOR  (z · 805459861 mod 2^32)
  

  each product taken modulo 2^32 before the exclusive-or (the wrap): vertices share entries, and an encoder fits the shared entries to every vertex that reads them.

5.3 Interpolation

For each level l the sink holds (§7), with R = R_l, and for each axis a of x, y, z:

s_a = p_a · R                         the position in cells, at 2^16
c_a = min(⌊s_a / 2^16⌋, R − 1)         the cell
w_a = s_a − 2^16 · c_a                 the fraction, in [0, 2^16]

The cell's eight corners are (c_x + d_x, c_y + d_y, c_z + d_z) for d in {0, 1}³, and corner d's weight is

ω_d = Π_a (d_a = 1 ? w_a : 2^16 − w_a)          at 2^48

The weights are non-negative and sum to 2^48 exactly. Feature c of the level at p is

x_(l·F + c) = R_half_up(Σ_d ω_d · G_l[e_d][c], 48)          at 2^ffrac

where e_d is the entry corner d reads (§5.2) and G_l[e][c] feature c of entry e. A level the sink does not hold gives F zeros. The network's input is the L · F values x_0 … x_(L·F − 1), level by level.

5.4 The network

Layer j reads a_j (a_0 is the input of §5.3) and gives, for each output o:

acc = B_j[o] + Σ_i W_j[o][i] · a_j[i]
y   = clamp(R_half_up(acc, wfrac), −2^31, 2^31 − 1)
a_(j+1)[o] = max(y, 0)      for every layer but the last
a_N[o]     = y              for the last

So every activation is at 2^ffrac, and the only rounding a layer makes is R_half_up by wfrac bits: a tie rounds toward +∞, whatever the sign. In the terms of integer-operators: each layer is conv with a 1 × 1 kernel over [w_j, 1, 1]; its rq has no multiplier, one shift wfrac, a pre bias B_j and the rounding half_up; its output tensor has the policy clamp to [−2^31, 2^31 − 1]; and each layer but the last is followed by relu.

5.5 The values

With y = a_N and sh = 16 − ffrac, the values at p are, each at 2^16:

density  σ = min(max(y_0, 0) · 2^sh, 2^24)
colour   c_k = clamp(y_k, 0, 2^ffrac) · 2^sh            k = 1, 2, 3: red, green, blue

A density is an absorption per unit length of the domain: one at 2^16 absorbs a fraction 1 − e^(−1) of light over the domain's side. A colour is in [0, 2^16].

5.6 Widths

Every weight ω_d is at most 2^48 and every feature fits i16, and the weights sum to 2^48, so every partial sum of §5.3 lies in [−2^63, (2^15 − 1) · 2^48] and fits i64 in any order. In §5.4 an input fits i32, a weight i16 and a bias i32, with at most 256 terms, so every partial sum fits in 55 bits plus a sign. An implementation forms every value of §5 in 64 bits and none of them wraps.

6. The render

The render at a view is W × H pixels of RGB8, rows top to bottom and pixels left to right, three bytes a pixel and no padding: the canonical buffer of reconstruction-binding §2.

6.1 The camera

From eye E, target and up U, as splat-exact §5.3 forms it:

F  = target − E          n_f = isqrt(F·F)          f_k = rdiv(F_k · 2^14, n_f)      the forward row at 2^14
Rv = F × U               n_r = isqrt(Rv·Rv)        r_k = rdiv(Rv_k · 2^14, n_r)     the right row at 2^14
u_k = R_half_up((r × f)_k, 14)                                                     the up row at 2^14

6.2 A pixel's ray

The pixel in column i and row j has its centre at (256·i + 128, 256·j + 128) at 2^8, as in splat-exact §5.4. Its ray leaves E in the direction

dx = 256·i + 128 − 128·W            dy = 128·H − 256·j − 128
D_k = 256 · focal · f_k + dx · r_k + dy · u_k
n_d = isqrt(D·D)
dir_k = rdiv(D_k · 2^16, n_d)                                  at 2^16, of length 2^16 to rounding

A pixel with n_d = 0 takes no sample and shows the background (§6.4).

6.3 The samples

The march from t_near to t_far is cut into S pieces with the edges

e_k = t_near + ⌊k · (t_far − t_near) / S⌋          k = 0 … S

Sample k (for k from 0 to S − 1, in order) has the length δ_k = e_(k+1) − e_k, so the lengths sum to t_far − t_near exactly, and its point is at m_k = e_k + ⌊δ_k / 2⌋ along the ray:

p_c = E_c + R_half_up(dir_c · m_k, 16)          c = x, y, z

A sample of length 0 changes nothing.

6.4 Compositing

Each pixel starts with transmittance T = 2^16 and accumulators A_r = A_g = A_b = 0, and takes its samples in order. Before each, if T < 7 (0.0001 at 2^16), the ray is done and takes no further sample. Otherwise, with (σ, c_r, c_g, c_b) the values at the sample's point (§5):

τ = R_half_up(σ · δ_k, 16)                          the optical depth, at 2^16
α = 2^16 − min(2^16, exp_neg_q16(−τ))               the fraction absorbed, at 2^16
c = R_half_up(α · T, 16)
T = T − c
A_k = A_k + R_half_up(c_k · c, 16)                  k = r, g, b

exp_neg_q16(0) is 65553, past 2^16, so a sample of no density absorbs nothing. c never exceeds T, so T stays in [0, 2^16], and A_k ≤ 2^16 − T. The pixel's bytes are

min(255, R_half_up(255 · A_k + T · bg_k, 16))          k = r, g, b

6.5 Widths

|E_c| is at most 2^24, so F and U fit 26 bits, Rv fits 52 and Rv·Rv fits 106: the camera is formed in 128-bit arithmetic, once a view. D fits 37 bits and D·D 76, so a ray's direction is formed in 128 bits too. Everything a sample forms fits in 64 bits: |dir_c · m_k| is below 2^39, σ · δ_k at most 2^46, and α · T, c_k · c and 255 · A_k + T · bg_k below 2^33.

7. Levels as refinements

7.1 A level not held

A payload carries levels 0 to K − 1 and declares levels K to L − 1. A sink holds the levels the payload carries and the levels it has applied (§7.3), always a prefix 0 to h − 1 of the field's levels. A level it does not hold gives zero features at every point (§5.3), so the field is defined, and presentable, whatever prefix it holds. The network reads every level, so an encoder that sends a coarse prefix fits the coarse levels to be presentable alone and each finer level to what the levels below it leave.

7.2 The level payload WIL1

The payload of wai.field.int_refine: one level of a field.

off	size	field	rule
0	4	magic	WIL1
4	1	version	1
5	1	l, the level it carries	1–15
6	1	F, the features of an entry	1–8
7	1	T, the log2 of the level's table size	4–19
8	2	R, the level's cells an axis, u16	at least 1
10	2	reserved	0
12		the level's table: min((R + 1)³, 2^T) entries of F i16 features at 2^ffrac, an entry's features together, entry by entry

Level 0 is in every WIG1 (K ≥ 1), so a WIL1 carries level 1 or later. Its rules are read in the table's order (field_range).

7.3 Applying a level

A level applies to a field only as its next level: with the field holding levels 0 to h − 1 (h < L), the level's l is h, its F and T are the field's, and its R is R_h, the side the field declares for level h. Otherwise it is refused (geometry_mismatch) and the field is as it was: a level never applies out of order, twice, past L, or in another shape than the one the field declared. Applied, it is the field's level h, and the field holds levels 0 to h.

7.4 Levels as layers

wai.field.int_refine is refinement-only (SPEC §5, Refinement capabilities), of the level form of staged-delivery §6.2, with the one admitted base wai.field.int_grid (SPEC §5, Base admission). A sink that holds a field base processes each level layer by staged-delivery §4.1:

• the base's output has one unit, the whole field, so a layer names no span, and its prev is one producer;
• a field whose payload carries K levels is at stage 0; layer k brings it to stage k, and carries level K + k − 1;
• step 6's parse is §7.2's (payload-malformed, with the payload's code), and a level is one unit;
• step 9 is §7.3: a level that is not the field's next level fails it (geometry-mismatch);
• the next stage is the field with that level applied.

A unit's output, for the stage hash of staged-delivery §5, is the render at the field's own view (§6), and its stage hash is that frame's SHA-256. The base step is decode-equivalence (wai.field.int_grid is StandardDefined), and so is each layer step: a stage's values and frames are exact given the levels it holds. Every layer boundary is a truncation point. A level reads no parameter set: everything it reads is in its payload and in the field below.

8. Order of the checks, and refusal codes

A parser of WIG1 checks, in this order, and refuses with the first that fails (integer-payloads §2):

1. the magic: WIG1, or else format_capability_mismatch if it is WIL1, or bad_magic;
2. the version byte: present (stream_length), and 1 (version);
3. the fixed header, 16 bytes, present; then the header through the view, 74 + 2·L + 2·N bytes, present (stream_length);
4. the reserved fields: the header's, at 12, and the view's last byte (reserved_nonzero);
5. the header's fields in the order of §3.2, the view's in the order of §4 (field_range);
6. the cap of §3.3 (too_large);
7. the network, layer by layer, its weights then its biases, then the levels carried, level by level, each present (stream_length);
8. nothing after the last level (trailing).

A parser of WIL1 checks, in this order:

1. the magic: WIL1, or else format_capability_mismatch if it is WIG1, or bad_magic;
2. the version byte: present (stream_length), and 1 (version);
3. the header, 12 bytes, present (stream_length);
4. the reserved field at 10 (reserved_nonzero);
5. the header's fields in the order of §7.2 (field_range);
6. the table present (stream_length);
7. nothing after it (trailing).

Applying a level then checks §7.3 (geometry_mismatch).

The codes are integer-payloads §8's:

code	when
bad_magic	the payload does not start with WIG1 or WIL1
format_capability_mismatch	a WIL1 where a WIG1 is read, or a WIG1 where a WIL1 is
version	a version this revision does not define
stream_length	the payload ends before a field or section it declares
reserved_nonzero	a reserved field that is not zero
field_range	a field of the header or the view outside its rule, or a degenerate pose
too_large	a network past the cap of §3.3
trailing	bytes after the last level, or after a WIL1's table
geometry_mismatch	a level that is not the next level of the field it is applied to (§7.3)

A refusal is final, as integer-payloads §8 says of a payload's refusal. A sink that does not implement this extension does not advertise wai.field.int_grid and routes it as any capability it does not decode (SPEC §4); a level layer it receives is not applied (staged-delivery §4.1). An emitter checks the framing before it writes the envelope, with the parser a sink runs (container::check_int_payload_framing).

9. Digests and field-equivalence

• The values digest of a list of points is the SHA-256 (FIPS 180-4) of their values, each a little-endian i64, the points in order and a point's four values in order (density, red, green, blue).
• The frame digest of a render is the SHA-256 of its bytes (§6). The frame digest at the field's own view is its stage hash (§7.4).

Field-equivalence. Two conforming sinks that hold the same levels of a field compute the same values at every point, so the same values digest of any list of points, and the same frame at any view, so the same frame digest. The criterion has no tolerance. wai.field.int_grid is registered with the class StandardDefined (SPEC §7): this text fixes every integer of its decode, and the payload carries its network, so no parameter set or pin is involved. Its fidelity tier is lossy: the features and weights are quantized from whatever the encoder made.

10. The encoder (informative)

10.1 The JSON form

tools/wai_int_field_export.py reads a field in a JSON form, "form": "wai-int-field/1", with ffrac, wfrac, log2_table (T), carried (K, so the tool writes a WIL1 for each later level); levels, each a side and a table of E_l entries of F floats; network, each layer's weights (rows) and biases as floats; and view, its positions and t_near, t_far as floats in units of the domain's side, its other members as integers. The JSON form is a converter's input, never a payload.

10.2 Quantization

A feature is ⌊v · 2^ffrac + 0.5⌋, clamped to i16; a weight ⌊v · 2^wfrac + 0.5⌋, clamped to i16; a bias ⌊v · 2^(ffrac + wfrac) + 0.5⌋, clamped to i32; a position ⌊v · 2^16 + 0.5⌋.

The tool builds the corpus's scenes itself, with no download: a field it fits to a procedural scene (a ball and a box of absorbing medium, coloured by position) by stochastic gradient descent on each level's table in turn, coarse to fine, through the level's own index and hash and against what the quantized levels below it leave; a seeded random field, the network's general case; and a field of saturating values. Its float steps are additions, subtractions, multiplications, divisions and comparisons, so it writes the same bytes on every machine.

11. Conformance

A conforming sink of this extension decodes every scene of the corpus volumetric-conformance/fields/, at every prefix of its levels, to the values digest it states at its points and the frame digest it states at each view; refuses every fault payload with the code it states; gives every level applied to a field the outcome it states (§7.3); and, offered each staged sequence's envelopes in order, reports for each the stage or the reason it states and reaches the final stage, freeze and stage hash stated (§7.4, with a hold bound of zero). The corpus's README.md lists its files.

A conforming decode reproduces every statement; a decode with any one of these faults fails at least one, and both implementations plant each in their own decode and require it:

fault	what it changes
weight_off_by_one	every interpolation fraction w_a one more than §5.3 gives
hash_constant	the third multiplier of §5.2's hash, 805459861, changed by 2
activation_rounding	a layer's R_half_up by wfrac replaced by ⌊acc / 2^wfrac⌋
ray_step	one sample more a ray than S
level_order	a level applied at its own index whatever levels the field holds, not only as the next one (§7.3)

The reference implementation is wai-rs: int_field::{parse_wig1, parse_wil1, Field, View, FieldStages}, Field::{query, render, apply_level}, int_field::corpus, the tool wai_int_field_vectors gen|verify, and tests/int_field.rs. The second implementation, volumetric-conformance/field_verify.py, is written from this text and shares no code with it.

12. Security considerations

A payload is bounded by its header before any section is read: at most 16 levels of at most 2^19 · 8 features, and a network of at most 2^16 multiply-accumulates a point; a level is bounded by its own header, and applies only in the shape its field declared. A render's work is declared before it starts (§3.3), and a sink that renders at views of its own choosing bounds W · H · S by its own limit. Every value fits the width §5.6 and §6.5 give it, so a hostile payload cannot make an implementation overflow, and none of its values steers a loop past those bounds. The spatial hash is fixed and public: shared entries are a property of the field, which its encoder fitted, and a frame digest identifies what every conforming sink shows at a view, not how faithfully the field reproduces any scene it was fitted to.