open-standards/wai/extensions/media-identify.md

# WAI Extension: Media Identification

> Status: Draft. What a file is, from its bytes alone: the formats they fit, each with the bytes
> that say so and how far they were followed; what a registered probe reads of it; signatures
> found inside it; and measures that hold when no format is recognised. It decodes nothing, and
> what it does not establish it leaves out. The reference implementation is the `identify` module
> of `wai-rs` (feature `identify`), which `wai identify` runs (operation `media.inspect`); the
> corpus is [`identify-conformance/`](../identify-conformance/), and its second implementation is
> `identify-conformance/identify_verify.py`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.

## 1. Scope

§1.1 `wai inspect` reads what an envelope holds. Most files a toolchain meets are not envelopes,
and the first question about one is what it is. Identification answers that for any file, and
it is the first step of working out how a piece of media was made.

§1.2 A result states only what the bytes establish. A claim comes with the offsets of the bytes
that make it, and a walk says where it stopped and why. A format is never guessed from a file
name, and no confidence figure is given: a result has a strength (§4), which says what was
checked. Where the bytes fit no format, the verdict is `unknown` and the measures (§8) are what
is known.

§1.3 What a container states is not what its samples hold. A track row (§5.1) reports what the file
says of a track, and no sample is read to confirm it. No payload is decrypted, and a protected track
is reported as protected and no further.

§1.4 This extension adds no capability and changes no dispatch (SPEC §4). A capability in a result
is the registered capability whose payload a format or track is: it says what `wai wrap` would
carry it under, not that a decoder here accepts it.

## 2. The result

§2.1 A result is a JSON object:

| member | value |
|---|---|
| `format` | `wai.identify/1` |
| `size` | the file's length in bytes |
| `verdict` | `identified`, `ambiguous`, `signature_only` or `unknown` (§4.2) |
| `candidates` | the formats the bytes fit, in the order of §4.3 |
| `probes_accepted` | the capabilities whose probe accepted the bytes (§3.5) |
| `embedded` | signatures found inside the file (§7), each `{at, id}` |
| `embedded_capped` | whether more were found than `embedded` holds |
| `measures` | §8 |

§2.2 A candidate is a JSON object:

| member | value |
|---|---|
| `id` | the format, as §5 and §6 name it, or the capability a probe alone accepted (§3.5) |
| `kind` | `wai`, `container`, `audio`, `image`, `elementary`, `playlist`, `text`, `compression` or `capture` |
| `strength` | `signature`, `structure` or `parameters` (§4.1) |
| `evidence` | the first 8 bytes-that-say-so, each `{at, len, is}`: `len` bytes at offset `at`, which read as `is` |
| `walked_to` | the offset a walk of the file's structures reached, or `null` where the format has no such walk |
| `stopped` | `null`, or `{at, why}`: where a walk stopped before the file's end (§3.3) |
| `facts` | what the format's headers state, by the section that defines it |
| `tracks` | §5.1 |
| `capabilities` | the capabilities it maps to (§1.4), sorted |

## 3. Common rules

§3.1 Integers are read as each section says. `u8`, `be16`, `be24`, `be32` and `be64` are big-endian,
`le16` and `le32` little-endian. All arithmetic of a result is integer arithmetic, but for the
sample
rate of §5.4; two implementations give the same result for the same bytes on any machine.

§3.2 **What is read.** The **head** is the first 1 MiB of the file (1,048,576 bytes), or all of it
if
shorter. The walks of §5.5, §5.9, §5.10, §5.13, §5.17, §5.18, §5.19 and §5.20, the table of §6 and
the
probes read the head alone. The other walks read the file where their lengths lead. The **scan** is
the first 16 MiB of the file (16,777,216 bytes), or all of it if shorter: §7 and §8 read it.

§3.3 **Walks.** A walk reads a format's structures in order and is bounded: at most 4,096 structures
(§5.6: 256), and 64 tracks. A walk that stops before the file's end says why, in `stopped`:

| `why` | meaning |
|---|---|
| `overrun` | a structure declares, or needs, more than the file holds |
| `invalid` | a structure's bytes are not what the format allows there |
| `crc` | a checksum the format defines does not hold |
| `limit` | the walk reached a bound of this extension, or the end of the head where it reads no further |
| `unknown_size` | a structure of unknown size, whose end only reading it would find (§5.4) |

`walked_to` and a stop's `at` are never past the end of the file: an offset past it is taken as the
file's length. A walk that reads the head alone and reaches its end while the file goes on has
stopped
there, `limit`.

§3.4 **Strings.** A string read from the file enters a result only through this filter: its first
64 bytes, each of 0x20 to 0x7E but `\` as itself, and any other byte as `\xNN` in lowercase
hexadecimal.
A file is untrusted input: a reader **MUST** treat these strings as data, and a tool that gives a
result to an agent **MUST** mark it as holding them.

§3.5 **Probes.** The probes of the table of [format-probes](format-probes.md) §2 that can name a
format from the bytes (`wai.image.jxs`, `wai.video.jxs`, `wai.video.hevc`, `wai.video.mvhevc`,
`wai.video.vvc`, `wai.video.lcevc`, `wai.audio.mpegh`, `wai.video.avs3`, `wai.audio.avs3`, in that
order) each read the head. The other two read a payload declared to be theirs, have no signature,
and say nothing of a file they are merely given, so they are not run. For each probe, in order:

- Where it accepts, the capability joins `probes_accepted`. The first candidate of kind `elementary`
  that maps to that capability (or, for `wai.video.mvhevc`, to `wai.video.hevc`) has its strength
raised
  to `parameters`, the capability added to its `capabilities`, and the probe's result set in its
  `facts.probe` under the capability. With no such candidate, a candidate of kind `elementary` is
added
  whose `id` is the capability, of strength `parameters`, with evidence `{at: 0, len: <head's
length>,
  is: "probe"}`, that capability and the same `facts.probe`.
- Where it refuses with `no_signature`, nothing is recorded.
- Where it refuses otherwise, and a candidate of kind `elementary` already maps to that capability,
  its
  `facts.probe_refused` holds the code under the capability.

## 4. Strength and verdict

§4.1 **Strength** says how far a candidate's bytes were followed:

- `signature`: a format's fixed bytes are where the format puts them. Nothing else was checked, and
  short ones occur by chance: `evidence` shows how many bytes it was.
- `structure`: what follows the signature is what the format's section requires of it.
- `parameters`: a probe of format-probes read the format's headers and accepted them.

§4.2 **The verdict** comes from the candidates at the highest strength present: `unknown` if there
are
no candidates; `ambiguous` if more than one; `signature_only` if one, of strength `signature`; else
`identified`. Two formats that both fit are both reported, and no ranking between them is invented.

§4.3 **Order.** Candidates are in order of strength, strongest first. Within a strength they keep
the
order of the sections that define them: §5.2 to §5.20, then §6's table in its order, then those a
probe alone added, in the order of §3.5.

## 5. Walks

§5.1 **Tracks.** A row of `tracks` holds `track` (the identifier the container gives, or `null`),
`media` (`video`, `audio`, `text` or `other`), `stated` (what the container calls it, through §3.4),
`codec` (the name of the table below, or `null` where this extension has no name for it), and
`width`,
`height`, `channels`, `sample_rate` and `protected`, each `null` where not stated. `protected`, on a
track the container says is protected, is `{scheme, original}`, each `null` where not stated.

A row's codec adds to its candidate's `capabilities` the capability of this table, where it has one:

| codec | capability |
|---|---|
| `avc` | `wai.video.avc` |
| `hevc` | `wai.video.hevc` |
| `vvc` | `wai.video.vvc` |
| `av1` | `wai.video.av1` |
| `vp8` | `wai.video.vp8` |
| `vp9` | `wai.video.vp9` |
| `evc` | `wai.video.evc` |
| `lcevc` | `wai.video.lcevc` |
| `jpeg-xs` | `wai.video.jxs` |
| `jpeg2000` | `wai.video.j2k` |
| `aac` | `wai.audio.aac` |
| `opus` | `wai.audio.opus` |
| `flac` | `wai.audio.flac` |
| `ac4` | `wai.audio.ac4` |
| `mpegh` | `wai.audio.mpegh` |
| `iamf` | `wai.audio.iamf` |

The names `mp3`, `vorbis`, `theora`, `speex`, `ac3`, `eac3`, `webvtt` and `ttml` map to none.

§5.2 **WAI envelopes** (`wai`, kind `wai`). The head begins `WAI1` or `WAI2`: evidence `magic:WAI1`
or
`magic:WAI2`, 4 bytes at 0, and `facts.version`. `le32` at 4 is the manifest's length. If it is
absent
the walk stops at 0 (`overrun`). If it exceeds 1 MiB the walk stops at 8 (`limit`); if the file
holds
less the walk stops at 8 (`overrun`); if the manifest is not JSON, at 8 (`invalid`). Evidence
`manifest`. For `WAI1`: `facts.capability` is `model_requirement.capability`, and `le32` after the
manifest is the payload's length, with evidence `payload_length`; where it or the payload does not
fit
the walk stops there (`overrun`) with `walked_to` there too, else `walked_to` is past the payload.
For
`WAI2`: `le16` after the manifest is `facts.renditions`, evidence `rendition_count`, and `walked_to`
is
past it. A walk that reaches this is `structure`. `wai inspect` reads the rest.

§5.3 **ISO base media** (`isobmff`, kind `container`). The four bytes at 4 are one of `ftyp`,
`styp`,
`moov`, `mdat`, `moof`, `free`, `skip`, `wide`, `sidx` or `pnot`. Boxes are walked from 0 to the end
of
the file. A box: `be32` size and four bytes of type. Every type byte is 0x20 to 0x7E or 0xA9, else
`invalid`. Size 0 runs to the end of the file; size 1 means `be64` at 8 is the size, which must be
at
least 16; any other size below 8 is `invalid`. Fewer bytes than a header needs is `overrun`. Each
box whose header
reads gives evidence `box:<type>`, 4 bytes at its offset plus 4, and its type joins `facts.boxes`
(distinct, first 16); a box that does not end within the file stops the walk there (`overrun`).
`moof`
boxes are counted in `facts.fragments`, present when there are any.

`ftyp` that fits: fewer than 8 content bytes is `invalid`; else `facts.major_brand` is its first
four,
and `facts.compatible_brands` the first 16 of the four-byte brands from its ninth. A major or
compatible brand `avif` or `avis` (among its first 4096 bytes) adds `wai.image.avif`.

`moov` that fits and is at most 64 MiB is read for `trak` boxes (above that, `facts.moov_not_read`
is
`true`). Inside a box, children are read as the top level is, stopping at the first that does not
fit.
A track needs `mdia`, and in it `hdlr` (the handler type is at content offset 8) and
`minf`/`stbl`/`stsd`
with a first entry (the entries start at content offset 8), else it has no row. `track` is from
`tkhd`:
`be32` at content offset 12 if its first byte is 0, else at 20. `media` from the handler: `vide`,
`auxv`,
`pict` is `video`; `soun` is `audio`; `text`, `subt`, `sbtl`, `clcp` is `text`; else `other`.
`stated` is the
entry's type. For `video`: `width` `be16` at entry content 24, `height` at 26, child boxes from 78.
For
`audio`: `channels` `be16` at 16, `sample_rate` `be32` at 24 shifted right 16, and, where `be16` at
8 is
0, child boxes from 28.

An entry of type `encv`, `enca`, `enct` or `encs` is protected: `protected.scheme` is from
`sinf`/`schm`
(content bytes 4 to 8) and `protected.original` from `sinf`/`frma` (its first four), and the codec
is
named from `original` where it is present. `mp4a` is named from its `esds`: `aac` for object type
0x40,
0x66, 0x67 or 0x68, `mp3` for 0x69 or 0x6B, else `null`. The object type is reached as MPEG-4
Systems
lays it out: after four version bytes, tag 3 and its size (one to four bytes of 7 bits, high bit set
on
all but the last); two bytes of identifier and one of flags, then 2 bytes if flag 0x80, a length
byte
and that many if 0x40, and 2 bytes if 0x20; tag 4 and its size; the next byte. Other entry types:

| entry | codec | entry | codec |
|---|---|---|---|
| `avc1` `avc2` `avc3` `avc4` `dvav` `dva1` | `avc` | `hvc1` `hev1` `dvh1` `dvhe` | `hevc` |
| `vvc1` `vvi1` | `vvc` | `av01` `dav1` | `av1` |
| `vp08` | `vp8` | `vp09` | `vp9` |
| `evc1` | `evc` | `lvc1` | `lcevc` |
| `jxsm` | `jpeg-xs` | `mjp2` | `jpeg2000` |
| `Opus` | `opus` | `fLaC` | `flac` |
| `ac-3` | `ac3` | `ec-3` | `eac3` |
| `ac-4` | `ac4` | `mhm1` `mhm2` `mha1` `mha2` | `mpegh` |
| `iamf` | `iamf` | `wvtt` | `webvtt` |
| `stpp` | `ttml` | | |

`structure`: at least two box headers read.

§5.4 **Matroska and WebM** (`matroska`, kind `container`). The head begins `1A 45 DF A3`: evidence
`ebml`, 4 bytes at 0. An element is an identifier of 1 to 4 bytes (its length in the leading zeros
of
its first byte, the marker kept) and a size of 1 to 8 bytes (the same, the marker removed; all ones
is unknown). The header must read in the head, with a known size: else the walk stops at 0
(`invalid` for an
element that does not parse, `overrun` for one that does not fit). `facts.doc_type` is its 0x4282,
through
§3.4, and `facts.doc_type_version` its 0x4287. The next element must be the Segment, 0x18538067:
evidence
`segment`; else the walk stops there, `overrun` if fewer than 5 bytes remain and `invalid`
otherwise. A
Segment with a known size that passes the file's end sets `facts.segment_overruns_by`, and the walk
ends at
the file's end.

Its children are walked to its end. A child that does not parse stops the walk (`overrun` if fewer
than 12
bytes remain, else `invalid`); an unknown size, `unknown_size`; a child that passes the file's end,
`overrun`.
Clusters (0x1F43B675) are counted in `facts.clusters`, present when there are any, one that passes
the file's
end included. Tracks (0x1654AE6B) of at most 1 MiB gives evidence `tracks` and a row for each 0xAE:
`track` 0xD7;
`media` from 0x83: 1 `video`, 2 `audio`, 17 `text`, else `other`; `stated` and `codec` from 0x86;
`width` and
`height` 0xB0 and 0xBA in 0xE0; `channels` 0x9F and `sample_rate` 0xB5 in 0xE1, an IEEE float of 4
or 8 bytes
taken toward zero; `protected` where 0x6D80 holds 0x6240 holding 0x5035. Unsigned values are
big-endian, of 1 to
8 bytes. The codec is `aac` for an identifier beginning `A_AAC`, else:

| identifier | codec | identifier | codec |
|---|---|---|---|
| `V_MPEG4/ISO/AVC` | `avc` | `V_MPEGH/ISO/HEVC` | `hevc` |
| `V_MPEGI/ISO/VVC` | `vvc` | `V_AV1` | `av1` |
| `V_VP8` | `vp8` | `V_VP9` | `vp9` |
| `A_OPUS` | `opus` | `A_FLAC` | `flac` |
| `A_VORBIS` | `vorbis` | `A_AC3` | `ac3` |
| `A_EAC3` | `eac3` | `A_MPEG/L3` | `mp3` |
| `S_TEXT/WEBVTT` | `webvtt` | | |

`structure`: at least one child read.

§5.5 **MPEG transport streams** (`mpeg-ts`, kind `container`). For each layout in order, packet size
and sync
offset (188, 0), (192, 4), (204, 0), and for each start from 0 below the packet size: 0x47 at the
sync offset of
five packets from there. The first that holds is the layout: evidence `sync`, 1 byte at its first
sync,
`facts.packet_size`. Packets are read while 188 bytes from a sync offset lie in the head. A packet
whose sync byte
is not 0x47 stops the walk (`invalid`), at the offset of that byte. PID is 13 bits from byte 1;
`facts.scrambled_packets`
counts those whose byte 3 has either of its top two bits set. With an adaptation field (byte 3 bit
0x20), the payload
is past its length byte and its length. With a payload (bit 0x10), PID 0 and each PID the program
table names are
gathered: a packet with the start bit (byte 1 bit 0x40) begins a section after its pointer field,
and a section's
length is 3 plus the 12 bits from its second byte. A section is used only if at least 12 bytes long
and its MPEG-2
checksum (polynomial 0x04C11DB7, from all ones, not reflected, over the section and its checksum) is
0.

The first section of table 0 on PID 0 is the program table: evidence `pat`, each entry of 4 bytes
from offset 8 to 4 from
its end (the first 64), with program number from the first two bytes and PID from the next 13 bits,
program 0 not being
counted. `facts.programs` is the number of programs. The first section of table 2 on each such PID:
evidence `pmt`.
Program information of the 12 bits from offset 10 starts at 12; streams follow, each a type byte, 13
bits of PID at the
next two bytes and 12 bits of descriptor length at the next two, then that many bytes of
descriptors. A row for each,
after the last packet: `track` the PID, `stated` `stream_type:0x<two lowercase hexadecimal digits>`,
`protected` where a
descriptor with tag 9 is among the program's or the stream's (then `{scheme: null, original:
null}`). Codec and media:

| type | codec | media | type | codec | media |
|---|---|---|---|---|---|
| 0x1B | `avc` | video | 0x24 | `hevc` | video |
| 0x33 | `vvc` | video | 0x0F 0x11 | `aac` | audio |
| 0x03 0x04 | `mp3` | audio | 0x81 | `ac3` | audio |
| 0x87 | `eac3` | audio | 0x01 0x02 0x10 | `null` | video |

Any other type has codec `null` and media `other`. `facts.packets_read` counts the packets. If no
sync byte stopped it,
and the walk's end is before the file's: `limit` if the head is shorter than the file, else
`overrun`. Always `structure`.

§5.6 **Ogg** (`ogg`, kind `container`). The head begins `OggS` and a zero byte. Pages from 0: a
27-byte header beginning the
same, a table of as many bytes as its byte 26, and a body of as many bytes as that table sums to. A
page that does not fit is
`overrun`; a header that does not begin so, `invalid`; a checksum (`le32` at 22) that does not hold
over the page with its checksum
bytes zero, under the polynomial 0x04C11DB7 from zero, not reflected, `crc`. At most 256 pages. Each
gives evidence `page`, 1 page
long, at its offset. Byte 5 bit 0x02 starts a stream: a row with `track` `le32` at 14, read from the
body: `OpusHead` (`stated`
`OpusHead`, `opus`, audio, `channels` byte 9, `sample_rate` `le32` at 12); `\x01vorbis` (`stated`
`vorbis`, `vorbis`, audio, `channels` byte
11, `sample_rate` `le32` at 12); `\x7fFLAC` (`stated` `FLAC`, `flac`, audio); `\x80theora` (`stated`
`theora`, `theora`, video);
`Speex` and three spaces (`stated` `Speex`, `speex`, audio); else `stated` is its first 8 bytes,
media `other`. `facts.pages_read`.
`structure`: two pages, or one that ends at the file's end.

§5.7 **ID3 and FLAC** (`flac`, kind `audio`). An **ID3v2 tag** at 0, for this and §5.9: `ID3`, ten
header bytes, bytes 3 and 4 not 0xFF,
and bytes 6 to 9 each below 0x80. Its length is 10 plus those four as 7 bits each, plus 10 where
byte 5 has bit 0x10. A head that begins
`ID3` with a header that is not one means neither walk applies. `fLaC` at 0, or after the tag:
evidence `fLaC`, and `wai.audio.flac`.
Blocks follow: a byte (bit 0x80: last; low 7 bits: type), then `be24` length. The first must be type
0 of length 34, else `invalid`:
evidence `streaminfo`, 34 bytes at its body, and `facts.sample_rate` (20 bits from body byte 10),
`facts.channels` (3 bits, plus 1),
`facts.bits_per_sample` (5 bits, plus 1) and `facts.samples` (36 bits). A block past the file's end
is `overrun`. After the last block, 2 bytes
0xFF and 0xF8 or 0xF9 are a frame's sync: evidence `frame_sync`, and `structure`. Other bytes there
stop the walk (`invalid`), and no bytes
(`overrun`). `walked_to` is the offset after the last block read.

§5.8 **RIFF** (kind per form). The head begins `RIFF`, with at least 12 bytes. The form at 8: `WAVE`
is `wave`, audio; `AVI ` is `avi`, container;
`WEBP` is `webp`, image; any other is `riff`, container. Evidence `form:<form>`, 4 bytes at 8, and
`facts.form`. The declared end is `le32` at 4 plus 8,
in `facts.declared_size` where it is not the file's size; the walk ends at the lesser of it and the
file's size. Chunks from 12: four bytes of
type and `le32` length, padded to an even length. A header past the end stops the walk (`overrun`)
unless the walk is exactly at the end or beyond it;
each header read gives evidence `chunk:<type>`, and its type joins `facts.chunks` (distinct, first
16). A chunk's content past the end stops the walk there
(`overrun`).

For `wave`: `fmt ` gives, from at most its first 64 bytes, `facts.format_tag` `le16` at 0 (where
that is 0xFFFE, `le16` at 24 instead),
`facts.channels` `le16` at 2, `facts.sample_rate` `le32` at 4 and `facts.bits_per_sample` `le16` at
14, each `null` where absent; `data` gives `facts.data_bytes`.
For `webp`: `VP8 ` with bytes 3 to 5 `9D 01 2A` gives `facts.coding` `lossy`, `facts.width` and
`facts.height` from `le16` at 6 and 8 in their low 14 bits;
`VP8L` with first byte 0x2F and 5 bytes gives `lossless`, and, in `le32` at 1, `width` its low 14
bits plus 1 and `height` the next 14 plus 1; `VP8X` with 10 bytes gives
`extended`, and `width` and `height` the 24-bit little-endian values at 4 and 7, each plus 1.
`walked_to` is the walk's end, at most the end above. `structure`: one chunk read.

§5.9 **MPEG audio** (`mpeg-audio`, kind `audio`). From 0 or after an ID3v2 tag (§5.7). A frame:
0xFF; 3 high bits of the next byte set; version 2 bits (0: 2.5, 2: 2,
3: 1), layer 2 bits (1: III, 2: II, 3: I), the bitrate index in byte 2's high 4 bits (not 0 or 15),
rate index in the next 2 (not 3), padding the next bit; byte 3's high 2 bits are
the channel mode. The version value 1 and layer value 0 are not a frame. Rate is 44100, 48000 or
32000 by index, halved for version 2 and quartered for 2.5. The bitrate in kbit/s, by
index from 1:

| | indexes 1 to 14 |
|---|---|
| version 1, layer I | 32 64 96 128 160 192 224 256 288 320 352 384 416 448 |
| version 1, layer II | 32 48 56 64 80 96 112 128 160 192 224 256 320 384 |
| version 1, layer III | 32 40 48 56 64 80 96 112 128 160 192 224 256 320 |
| version 2 or 2.5, layer I | 32 48 56 64 80 96 112 128 144 160 176 192 224 256 |
| version 2 or 2.5, layers II and III | 8 16 24 32 40 48 56 64 80 96 112 128 144 160 |

Frame length: layer I, (12000 × kbit/s ÷ rate + padding) × 4; layer III of version 2 or 2.5, 72000 ×
kbit/s ÷ rate + padding; otherwise 144000 × kbit/s ÷ rate + padding; division
is integer division. Nothing is a candidate unless the first bytes are a frame and at least one
whole frame is walked. Evidence `id3v2`, 3 bytes at 0, where a tag is there; `frame_header`, 4 bytes
at the first
frame. `facts.version` (`1`, `2` or `2.5`), `facts.layer`, `facts.bitrate_kbps`, `facts.sample_rate`
and `facts.channels` (1 for channel mode 3, else 2), of the first frame. Frames walk through the
head. A frame must
keep the first's version, layer and rate, else the walk stops there (`overrun` if fewer than 4 bytes
remain, else `invalid`). Exactly 128 bytes left beginning `TAG`: evidence `id3v1`, 3 bytes, and the
walk ends after them. A frame past the
file's end is `overrun`, past the head's alone `limit`. `facts.frames_read`. `structure`: three
frames.

§5.10 **ADTS** (`adts`, kind `audio`). A frame: 0xFF; byte 1 with `byte & 0xF6 == 0xF0`; byte 2's
high 2 bits plus 1 the audio object type, its next 4 the rate index (below 13: 96000, 88200, 64000,
48000, 44100, 32000, 24000, 22050,
16000, 12000, 11025, 8000, 7350), and channel configuration from its low bit and byte 3's high 2;
frame length 13 bits from byte 3's low 2, at least 7. Nothing is a candidate unless the head begins
with a frame and at least one whole
frame is walked. Evidence `frame_header`, 7 bytes at 0, `wai.audio.aac`, and
`facts.audio_object_type`, `facts.sample_rate`, `facts.channel_configuration` of the first. Frames
walk as §5.9's do, keeping type and rate (`overrun` under 7 bytes), and with
the same limits. `facts.frames_read`. `structure`: three frames.

§5.11 **PNG** (`png`, kind `image`). The head begins `89 50 4E 47 0D 0A 1A 0A`: evidence
`signature`, and `wai.image.png`. Chunks from 8: `be32` length, four ASCII letters of type (else
`invalid`), data, 4-byte checksum. The first must be `IHDR` of length 13, else
`invalid`; its checksum (polynomial 0xEDB88320 reflected, from all ones, inverted at the end; over
type and data) must hold, else `crc`. Then evidence `ihdr`, 25 bytes, `facts.width`, `facts.height`,
`facts.bit_depth`, `facts.colour_type`, `facts.interlace`, and `structure`.
`acTL` sets `facts.animated`; `cICP` with 4 bytes readable sets `facts.cicp` to them as a list. A
chunk past the file's end, or the end of the file before `IEND`, is `overrun`. `IEND` gives evidence
`iend`, 12 bytes at its offset, and the walk ends after it. `facts.chunks_read`.
Only `IHDR`'s checksum is computed.

§5.12 **JPEG** (`jpeg`, kind `image`). The head begins `FF D8 FF`: evidence `soi`, 2 bytes at 0, and
`wai.image.jpeg`. Markers from 2: 0xFF and a code, a code of 0xFF being a fill byte that advances
one. Fewer than 2 bytes is `overrun`; a first byte not 0xFF is `invalid`. A code of 0xDA gives
evidence `sos`, 2 bytes, and ends the walk there. Others: `be16` length, at least 2 (else
`invalid`), `overrun` if it does not read or the segment passes the file's end. 0xE0 to 0xEF name an
application segment from its first 32 bytes: `JFIF\0` `jfif`, `Exif\0\0` `exif`, `ICC_PROFILE\0`
`icc`, `MPF\0` `mpf`,
the XMP namespace (`http://ns.adobe.com/xap/1.0/\0`) `xmp`, `Adobe` `adobe`, else `other`; <!-- allow-name: the APP14 segment's identifier and the XMP namespace, byte signatures a JPEG carries -->
`facts.applications` holds them, distinct, first 8. Codes 0xC0 to 0xC3, 0xC5 to 0xC7, 0xC9 to 0xCB
and 0xCD to 0xCF are a frame header, needing 6 bytes (else `overrun`): evidence `sof`, the segment's
bytes, `facts.process` (0xC0
`baseline`, 0xC1 `extended`, 0xC2 `progressive`, 0xC3 `lossless`, 0xC9 to 0xCB `arithmetic`, else
`hierarchical`), `facts.precision`, `facts.height`, `facts.width`, `facts.components`, and
`structure`.

§5.13 **GIF** (`gif`, kind `image`). The head begins `GIF87a` or `GIF89a`: evidence
`signature:<it>`, 6 bytes. Fewer than 13 bytes stops the walk at 6. `facts.width` and `facts.height`
from `le16` at 6 and 8; a global table of 3 × 2^(n+1) bytes follows byte 10 flag 0x80 (n its low 3
bits). Blocks from there: 0x3B the
trailer (evidence `trailer`, 1 byte, and the walk ends after it); 0x21 an extension, sub-blocks from
2 on; 0x2C an image, 10 bytes (a local table likewise from its byte 9), one more byte, then
sub-blocks, evidence `image`, 10 bytes, for the first; any other byte is `invalid`. Sub-blocks: a
count byte and that many, ending after a count
of 0. Bytes not in the head stop the walk: `limit` where the head is shorter than the file, else
`overrun`. `facts.frames_read`. `structure`: one image, or the trailer.

§5.14 **IVF** (`ivf`, kind `container`). The head begins `DKIF`: evidence `dkif`, 4 bytes. Under 12
bytes stops at 0 (`overrun`); a version (`le16` at 4) not 0 or a header length (`le16` at 6) under
32 stops at 0 (`invalid`). A row: media `video`, `stated` the four bytes at 8, codec `av1` for
`AV01`, `vp9` for `VP90`, `vp8` for
`VP80`, `width` and `height` `le16` at 12 and 14. Frames from the header length: `le32` size, 8
bytes of time, then that many. `overrun` where one does not fit. `facts.frames_read`. `structure`:
one frame.

§5.15 **YUV4MPEG2** (`y4m`, kind `elementary`). The head begins `YUV4MPEG2 `: evidence `signature`,
10 bytes. A line feed must end the header within 256 bytes: else stop at 0 (`overrun` if the file
ends sooner, else `invalid`). Space-separated tokens: `W` width, `H` height, `C` colour (default
`420`), `F` frame rate; `facts.width`, `facts.height`, `facts.colour`
and `facts.frame_rate`. A colour is a base, and optionally `p` and a decimal depth, 2 bytes a sample
above 8. Bases: `420`, `420jpeg`, `420mpeg2`, `420paldv` (w·h + 2·⌈w/2⌉·⌈h/2⌉ samples a frame);
`422` (w·h + 2·⌈w/2⌉·h); `444` (3·w·h); `mono` (w·h). Any other stops the walk (`invalid`), as do
absent dimensions. Frames: a line beginning `FRAME`, within 256 bytes, then a frame's bytes.
`facts.frames_read`.
`structure`: one frame.

§5.16 **FLV** (`flv`, kind `container`). The head begins `FLV` and 0x01: evidence `signature`, 4
bytes. `be32` at 5, the data offset, must be at least 9, else stop at 0 (`overrun` under 9 bytes,
else `invalid`). Tags from that offset plus 4: 11 header bytes (type in byte 0's low 5 bits, one of
8, 9 or 18, else `invalid`; `be24` length at 1), then the body, then a `be32` that must be 11 plus
the length (else `invalid`; absent, `overrun`).
The first audio tag (type 8, with a body) gives a row: media `audio`, `stated` `sound_format:<n>`
for its first body byte's high 4 bits, codec `mp3` for 2 and `aac` for 10. The first video tag (type
9, with a body) gives a row: where its first body byte has bit 0x80, `stated` is the next four
bytes, with codec `avc` for `avc1`, `hevc` for `hvc1`, `av1` for `av01`, `vp9` for `vp09`; otherwise
`stated` is `codec_id:<n>` for that byte's low 4 bits, with `avc` for 7. The video row comes first.
`facts.tags_read`. `structure`: one tag.

§5.17 **AV1 OBU streams** (`av1-obu`, kind `elementary`). An OBU: a header byte with bits 0x81 clear
and 0x02 set; type its bits 3 to 6, one of 1 to 8 or 15; an extension byte where bit 0x04 is set; a
LEB128 size of at most 8 bytes; that many bytes. The first must be type 1 or 2. They are walked
through the head. A header that does not parse stops the walk at it (`limit` if the head is shorter
than the file and fewer than 8 bytes remain, else `invalid`); one past the file's end, `overrun`;
one past the head's alone, `limit`. The first type 1 gives evidence `sequence_header`, and is read
as AV1 §5.5: `facts.profile`, `facts.level` (the first operating point's `seq_level_idx`),
`facts.width` and `facts.height` (`max_frame_width_minus_1` and `max_frame_height_minus_1`, plus 1).
`facts.units_read`, and `wai.video.av1`. A candidate only if a sequence header reads and at least
two OBUs were walked, and then `structure`: two bytes that merely fit are no claim of AV1.

§5.18 **HLS** (`hls`, kind `playlist`). After an optional UTF-8 byte order mark, `#EXTM3U` and a
line end or the end of the head: evidence `extm3u`, 7 bytes at it. Lines (ended by a line feed, less
a final carriage return) are read: any beginning `#EXT-X-` is a tag; `#EXT-X-STREAM-INF:` counts
`facts.variants`, and the first with `CODECS="` gives `facts.codecs`; `#EXTINF:` counts
`facts.segments`; `#EXT-X-MAP:` counts `facts.maps`; `#EXT-X-ENDLIST` sets `facts.ended`;
`#EXT-X-VERSION:` gives `facts.version` where it is a decimal integer; and the first `#EXT-X-KEY:`
or `#EXT-X-SESSION-KEY:` whose `METHOD=` (to the next comma) is not `NONE` gives `facts.key_method`.
`facts.playlist` is `multivariant` with variants, else `media` with segments, else `other`. Every
one of these facts is present; `structure`: at least one tag.

§5.19 **DASH** (`dash`, kind `playlist`). After an optional byte order mark and any white space,
`<?...?>` and `<!--...-->`, then `<MPD` and white space or `>`: evidence `mpd`, 4 bytes at it.
`urn:mpeg:dash:schema:mpd:2011` anywhere in the head gives evidence `namespace` and `structure`.
`facts.type` is the value of ` type="` in the root tag up to its first `>`. `facts.periods`,
`facts.adaptation_sets`, `facts.representations` and `facts.content_protection` count `<Period`,
`<AdaptationSet`, `<Representation` and `<ContentProtection`; `facts.codecs` holds the distinct
values of `codecs="`, the first 16. It is no XML parser: these are substrings.

§5.20 **Annex B streams** (kind `elementary`). The head holds at least two zero bytes, then 0x01,
and nothing before them. Units are split as format-probes §2 (its Annex B byte streams) does, the
first 4,096 (trailing zero bytes removed). If the byte after the first prefix is 0xBA or 0xB3, §6
has the format, and no reading applies. Each reading below needs every unit's header to hold and its
parameter sets in order, each before the first coded picture of that reading; a stream two readings
fit is reported as both, and no reading is preferred. A reading is `structure`, with
`facts.units_read`. Where the head is shorter than the file, or 4,096 units were read, it stops at
the last unit (`limit`). `walked_to` is `null`.

- `avc-annexb`: one header byte, bit 0x80 clear, type its low 5 bits and 1 to 21; types 5, 7 and 8
  have nonzero bits 0x60. Order 7, 8; coded pictures are types 1 to 5. Its first type 7 is read as
  ITU-T H.264 §7.3.2.1.1, with emulation prevention removed: a `profile_idc` other than 66, 77, 88,
  100, 110, 122, 244, 44, 83, 86, 118, 128, 138, 139, 134 or 135 or a failed reading rules the
  reading out. Evidence `nal:sps`; facts `profile_idc`, `constraint_flags`, `level_idc`,
  `chroma_format_idc`, `width` and `height` less cropping; `wai.video.avc`.
- `hevc-annexb`: two header bytes, bit 0x80 of the first clear, the low 3 bits of the second not
  zero, type the first's bits 1 to 6 and one of 0 to 9, 16 to 21 or 32 to 40. Order 32, 33, 34;
  coded pictures are types 0 to 9 and 16 to 21. Evidence `nal:sps`; `wai.video.hevc`.
- `vvc-annexb`: two header bytes, bits 0xC0 of the first clear, the low 3 bits of the second not
  zero, type the second's high 5 bits and at most 25. Order 15, 16; coded pictures are types 0 to
  10. Evidence `nal:sps`; `wai.video.vvc`.

## 6. Formats known by their signature

§6.1 A format of this table is a candidate where every one of its fixed bytes is at its offset in
the head: `strength` `signature`, evidence `signature` for each, and the capabilities it names.
Bytes are in hexadecimal, each group at the offset before its colon. A format with two rows is
either.

| id | kind | bytes | capability |
|---|---|---|---|
| `aiff` | audio | `0:464f524d 8:41494646` |  |
| `aiff` | audio | `0:464f524d 8:41494643` |  |
| `caf` | audio | `0:636166660001` |  |
| `wavpack` | audio | `0:7776706b` |  |
| `amr` | audio | `0:2321414d520a` |  |
| `amr-wb` | audio | `0:2321414d522d57420a` |  |
| `jpegxl` | image | `0:ff0a` | `wai.image.jxl` |
| `jpegxl` | image | `0:0000000c4a584c200d0a870a` | `wai.image.jxl` |
| `jp2` | image | `0:0000000c6a5020200d0a870a` | `wai.image.jp2` |
| `j2k` | image | `0:ff4fff51` | `wai.image.jp2` |
| `tiff` | image | `0:49492a00` |  |
| `tiff` | image | `0:4d4d002a` |  |
| `bigtiff` | image | `0:49492b00` |  |
| `bigtiff` | image | `0:4d4d002b` |  |
| `openexr` | image | `0:762f3101` |  |
| `dpx` | image | `0:53445058` |  |
| `dpx` | image | `0:58504453` |  |
| `radiance` | image | `0:233f52414449414e43450a` |  |
| `radiance` | image | `0:233f524742450a` |  |
| `mxf` | container | `0:060e2b34020501010d0102` |  |
| `asf` | container | `0:3026b2758e66cf11a6d900aa0062ce6c` |  |
| `mpeg-ps` | container | `0:000001ba` |  |
| `mpeg-video` | elementary | `0:000001b3` |  |
| `webvtt` | text | `0:574542565454` |  |
| `gzip` | compression | `0:1f8b08` |  |
| `zstd` | compression | `0:28b52ffd` |  |
| `xz` | compression | `0:fd377a585a00` |  |
| `bzip2` | compression | `0:425a68` |  |
| `zip` | compression | `0:504b0304` |  |
| `tar` | compression | `257:7573746172` |  |
| `pcap` | capture | `0:d4c3b2a1` |  |
| `pcap` | capture | `0:a1b2c3d4` |  |
| `pcap` | capture | `0:4d3cb2a1` |  |
| `pcap` | capture | `0:a1b23c4d` |  |
| `pcapng` | capture | `0:0a0d0d0a` |  |

## 7. Signatures inside a file

§7.1 A reverse engineer's first question about a blob is what it holds. The scan is searched for
each of these fixed bytes, none shorter than 8 bytes, so that one in 16 MiB of noise is improbable;
each hit past the file's beginning gives `{at, id}`, `at` being where its structure begins, `back`
bytes before them. A hit whose structure would begin at or before the file's beginning is not
reported. Hits are in the order their bytes are found, and within one offset in this table's order.
At most 32 are reported, and `embedded_capped` says whether another was found. A hit says only that
the bytes are there.

| id | bytes | back |
|---|---|---|
| `png` | `89504e470d0a1a0a` | 0 |
| `jp2` | `0000000c6a5020200d0a870a` | 0 |
| `jpegxl` | `0000000c4a584c200d0a870a` | 0 |
| `mxf` | `060e2b34020501010d0102` | 0 |
| `asf` | `3026b2758e66cf11a6d900aa0062ce6c` | 0 |
| `ogg` | `4f67675300020000000000000000` | 0 |
| `flac` | `664c614300000022` | 0 |
| `flac` | `664c614380000022` | 0 |
| `isobmff` | `6674797069736f6d` | 4 |
| `isobmff` | `6674797069736f32` | 4 |
| `isobmff` | `6674797069736f35` | 4 |
| `isobmff` | `6674797069736f36` | 4 |
| `isobmff` | `667479706d703431` | 4 |
| `isobmff` | `667479706d703432` | 4 |
| `isobmff` | `667479704d344120` | 4 |
| `isobmff` | `667479704d345620` | 4 |
| `isobmff` | `6674797071742020` | 4 |
| `isobmff` | `6674797061766966` | 4 |
| `isobmff` | `6674797068656963` | 4 |
| `isobmff` | `66747970636d6663` | 4 |
| `isobmff` | `6674797064617368` | 4 |
| `isobmff` | `667479706d736468` | 4 |

## 8. Measures

§8.1 `measures` holds, over the scan: `scanned`, its length; `distinct_bytes`, how many of the 256
values occur; `zero_bytes`, how many are zero; `chi_square`, the floor of `256 · Σc² / n` less `n`,
where `c` are the counts of the values and `n` the length, and 0 for an empty scan; `text` (§8.2);
`windows` and `uniform_windows` (§8.3); and `periods` (§8.4).

§8.2 `text` is true where the scan is not empty, is valid UTF-8 (a sequence the scan's end cuts
short allowed), and holds no byte below 0x20 but 0x09, 0x0A, 0x0C and 0x0D, and no 0x7F.

§8.3 A **window** is each whole 4096 bytes of the scan from its start. One is **uniform** when the
sum of its counts squared is at most 70,496: its chi-square against a uniform spread is then at most
310, about the 99th percentile for 255 degrees of freedom. A compressed payload and an encrypted one
both tend to be uniform; this measure cannot tell them apart, and says only that the bytes show no
structure in their values.

§8.4 **Repeats.** Over the first 256 KiB of the scan (`n` bytes), for each lag `l` from 1 to `L +
1`, where `L` is the lesser of 4096 and `⌊n/4⌋`: `m = n − l`; `matches` counts the `i` below `m`
with byte `i` equal to byte `i + l`; `expected` is the floor of `m · Σc² / n²`, at least 1, over
this sample's counts; the excess `e` is `matches − expected`; and `z` is, where `e` is positive, the
floor of the square root of the floor of `e² / expected`, otherwise 0. With `L` below 2 there are no
periods.

The floor is 8 times the median of `|e|` over those lags, the element at index `⌊k/2⌋` of the `k` of
them sorted: data that repeats exactly is not independent from lag to lag, and its noise is many
times what `z` assumes. A lag from 2 to `L`, in increasing order, is reported as `{lag, z}` where
all of these hold, at most 8: `z` is at least 8; `e` is at least the floor; `e` exceeds twice the
greatest of its neighbours' excess and 0; if `2l ≤ L + 1`, `z` at `2l` is at least 8; and it is no
multiple of a lag already reported. A repeat is a record size, a sample width or a table's stride,
found with no knowledge of the format.

## 9. Security

§9.1 Identification reads files nobody vouches for. A conforming implementation **MUST NOT** panic,
read outside the file, allocate more than the file's bounded windows and a bounded number of
structures, or follow a URI in a playlist. The bounds of §3 are what make the cost of a hostile file
bounded.

§9.2 A result can hold strings a file's author chose (§3.4). The operation's agent door marks its
results `untrusted_content`.

§9.3 An implementation **MUST NOT** decrypt, or try to recover the keys of, a protected track. It
reports the scheme the file states and stops.

## 10. Conformance

§10.1 The corpus [`identify-conformance/`](../identify-conformance/) holds files written by real
muxers and encoders from synthetic inputs (`ext/`), generated cases and negative controls (`gen/`),
and `vectors.json`, each file's digest and whole result. `prober.json` is what an independent media
prober reads of `ext/`, the reference checked against it wherever both state a value. A conforming
implementation gives each file its result exactly.