wai-rs: TON tools-header parser + codestream→z→i_s chain #21

Merged

dcharlot merged 1 commit from feat/jpegai-e2e-chain into main

2026-07-17 12:36:45 -04:00

dcharlot commented

2026-07-17 12:36:44 -04:00

Owner

parse_tools_header decodes the post-filter chain of the TON substream (EFE-linear → ICCI → eICCI → EFE-nonlinear → LEF), each tool gated by its recursive enable flag. Adds BitReader::read_max / bits_for_max — the headers universal width rule bits = ceil(log2(max+1)); some widths are runtime-dependent (a max_symbol decoded earlier sets the width of the weights after it: 65535⇒16 bits, then 99⇒7).

Validated on test256.bin: every TON field matches, end_bit=973 (all consumed, 3-bit pad closes the 63-byte segment). The decoded params ARE the post-filters fixture inputs — B1/100=125.61/130.91 (efl means), luma_min/max=497/25320 + 8 A1 weights (efn inputs), icci_model_signalled_idx=[Y=0,UV=1] (the per-tile eICCI selection). Header parser and post-filters agree from opposite directions.

Plus an end-to-end chain test: test256.bin → container → SOZ → mANS fac → z (2560, 75 nz) → decode_hyper_scale → i_s, byte-exact 40960/40960 vs reference — two independently-built subsystems chained off raw bytes, no fixture input beyond the normative CDF/weight assets.

**parse_tools_header** decodes the post-filter chain of the TON substream (EFE-linear → ICCI → eICCI → EFE-nonlinear → LEF), each tool gated by its recursive enable flag. Adds `BitReader::read_max` / `bits_for_max` — the headers universal width rule `bits = ceil(log2(max+1))`; some widths are runtime-dependent (a `max_symbol` decoded earlier sets the width of the weights after it: 65535⇒16 bits, then 99⇒7). **Validated on test256.bin: every TON field matches, end_bit=973** (all consumed, 3-bit pad closes the 63-byte segment). The decoded params ARE the post-filters fixture inputs — `B1/100`=125.61/130.91 (efl means), luma_min/max=497/25320 + 8 A1 weights (efn inputs), `icci_model_signalled_idx`=[Y=0,UV=1] (the per-tile eICCI selection). Header parser and post-filters agree from opposite directions. **Plus an end-to-end chain test:** test256.bin → container → SOZ → mANS fac → z (2560, 75 nz) → decode_hyper_scale → i_s, **byte-exact 40960/40960** vs reference — two independently-built subsystems chained off raw bytes, no fixture input beyond the normative CDF/weight assets.

dcharlot added 1 commit

2026-07-17 12:36:44 -04:00

wai-rs: jpegai_bitstream — tools-header (TON) parser + codestream→z→i_s chain 0252dc8a82

parse_tools_header decodes the post-filter chain of the TON substream in filters-factory
order (EFE-linear → ICCI → eICCI → EFE-nonlinear → LEF), each tool gated by its own
enable flag per the recursive header walk. Adds BitReader::read_max / bits_for_max: the
header's universal width rule, bits = ceil(log2(max_value+1)) (the reference
HeaderCoder::calc_bits_count). No header field has a fixed width, and some are
runtime-dependent — a max_symbol decoded earlier in a segment sets the width of the
weights after it (65535 ⇒ 16 bits, then 99 ⇒ 7).

Validated on real test256.bin, every field matching the reference and end_bit=973 —
all TON fields consumed exactly, up to the 3-bit pad closing the 63-byte segment.
The decoded parameters ARE the post-filters' fixture inputs: B1/100 = 125.61/130.91 =
efl_chroma's plane means; luma_min/max = 497/25320 and the 8 A1 weights = efn_filter's
luma_min_q/luma_max_q/weights_q15; icci_model_signalled_idx[Y]=0,[UV]=1 = the per-tile
eICCI model selection. Header parser and post-filters agree from opposite directions.

Also adds an end-to-end chain test: test256.bin → parse_container → SOZ → mANS fac
decode → −31 → z (2560 vals, 75 non-zero) → decode_hyper_scale → i_s, byte-exact
40960/40960 against the reference. Two independently-built subsystems, chained off raw
codestream bytes with no fixture input beyond the normative CDF/weight assets.

parse_tools_header takes the first filter's bit offset rather than the TON payload start:
_stream_header_part defaults to "tool_header", so other tools also emit into TON ahead of
the filter chain (two such bits precede EFE-linear here); locating that prefix in general
needs the full tool-tree walk.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

dcharlot merged commit 0252dc8a82 into main

2026-07-17 12:36:45 -04:00

~~dcharlot referenced this pull request 2026-11-06 13:54:17 -05:00~~

wai-quantum: an energy ceiling held by a live reading, and the command line can give one (stacked on #668) #670

dcharlot referenced this pull request

2026-11-08 09:49:00 -05:00

wai-quantum: the planner under a contract, its reasoning sealed, a provider's answer shadowed (stacked on #660) #668