The packer writes refinement layers, chooses each piece's representation by its ladder, packs live, and simulates delivery over MoQ; the corpus gains a staged golden that the independent verifier re-derives byte for byte. - Refinements (packaging §5.4): for each refinement rung of a piece's ladder, a wai.video.int_refine object over the int_motion base, written by wai-rs's refinement encoder (staged::wir1::encode) with the manifest `wai stage refine` writes; every layer applied to the decoded base by the reference sink before it is kept. The index states each prefix's tier, digest, SSE and deadline. - A piece's ladder and stills (§5.7, plan::piece_ladder): a rule's then.ladder, or the component's first base rung, its refinement chain and its floors. §3.4 item 10 now requires a rule's ladder to name exactly one base, every refinement over a rung it names and every floor. A wai.image.png base is a still, the piece's first frame held for the piece, written as a PNG whose every byte is stated (png.rs; the verifier's reader inflates with miniz_oxide, the one new dependency). - One coder (pack::Coder) for the on-demand and the live packer; the verifier holds each piece to its ladder and its plan's deadline, and decodes every prefix (a still held, each layer through the sink). - Catalogs: the MoQ catalog is a stage set of moq-streaming-format §4.1, written by staged::moq (wai_staged, a receipts track and an alias per media track, depends on a track of every layer below); `priority` (MOQT's, §8.5) and `wai_layer` leave it, and the registry drops `wai_layer`. HTTP views carry the component's default base capability only: a still's piece plays from the floor there (http-delivery §4.2), and its stage view and MoQ tracks list it. - Live (§9.2, live.rs, `wai-pack live`): frames pushed one at a time, pieces coded as the segmenter closes them, windows sealed by version-2 receipts chained live-window; its plan, bases and refinements are the on-demand package's byte for byte (`wai-pack listing --primary`). - Delivery (§8.9, delivery.rs, `wai-pack sim`): a thin consumer of stage_measure::sim — per-track timeouts, §8.5's priorities, the subscriber's cap, group receipts on each track's chain; delivered objects accepted by Merkle proof, dropped ones held by inclusion only. - Corpus: the golden video-staged-01 (fixture, policy, supplied floor and supplied_floor.py, which states how it was written), staged-mutations/, delivery/; video-base-01 keeps its objects, index and receipt (its MoQ catalog and LISTING change). - pack_verify.py re-derives every layer and still from the source (its own rANS coder and deflate), decodes every prefix (staged_verify.py's WIR1 and PNG readers), checks the MoQ catalog with staged_verify.py's §4.1 check, renders every catalog, and re-derives each delivery case through sim_verify.py's simulator. stage_diff.py runs layers, stills, piece plans and deliveries differentially. - Tests 5, 7, 12 and 13 over the staged golden (tests/staged.rs). Co-Authored-By: Claude Opus 5.5 <[email protected]>
235 lines
12 KiB
Rust
235 lines
12 KiB
Rust
//! Every refusal of the packer this revision names (packaging §7.3, §11.4, §12), each from
|
|
//! the golden's inputs with one thing changed: the supplied floor's reasons, what this
|
|
//! revision does not pack yet, a meter that gives no figure, the encoder's caps, and an
|
|
//! output directory that is not empty.
|
|
|
|
use std::collections::BTreeMap;
|
|
use std::path::{Path, PathBuf};
|
|
|
|
type Files = BTreeMap<String, Vec<u8>>;
|
|
|
|
fn corpus() -> PathBuf {
|
|
Path::new(env!("CARGO_MANIFEST_DIR")).join("../pack-conformance")
|
|
}
|
|
|
|
fn golden_policy() -> String {
|
|
String::from_utf8(std::fs::read(corpus().join("packs/video-base-01/policy.toml")).unwrap()).unwrap()
|
|
}
|
|
|
|
fn golden_floor() -> Vec<u8> {
|
|
std::fs::read(corpus().join("packs/video-base-01/floor/v.obu")).unwrap()
|
|
}
|
|
|
|
/// The golden policy with its floor pinned to the listing of `floor`, and `edit` applied.
|
|
fn policy_for(floor: &Files, edit: &dyn Fn(String) -> String) -> Vec<u8> {
|
|
let listing = wai_pack::floor::listing_sha256(floor).unwrap();
|
|
let p = golden_policy();
|
|
let at = p.find("supplied_listing = \"").unwrap() + "supplied_listing = \"".len();
|
|
let p = format!("{}{listing}{}", &p[..at], &p[at + 64..]);
|
|
edit(p).into_bytes()
|
|
}
|
|
|
|
fn pack(policy: &[u8], floor: Files, meter: Option<Box<dyn wai::energy_meter::Meter>>) -> String {
|
|
let mut o = wai_pack::pack::PackOpts::new(ed25519_dalek::SigningKey::from_bytes(&[0x5a; 32]), "did:key:test", floor);
|
|
o.meter = meter;
|
|
o.power_reader = "model".into();
|
|
o.issued_at = "2026-10-06T00:00:00Z".into();
|
|
let mut src = BTreeMap::new();
|
|
src.insert("v".to_string(), wai_pack::fixture::generate("video-base-01").unwrap());
|
|
match wai_pack::pack::pack(policy, &src, &mut o) {
|
|
Ok(_) => "ok".into(),
|
|
Err(e) => e.code,
|
|
}
|
|
}
|
|
|
|
fn units() -> Vec<Vec<u8>> {
|
|
wai_pack::av1::temporal_units(&golden_floor()).unwrap().iter().map(|t| t.bytes()).collect()
|
|
}
|
|
|
|
fn floor_of(stream: Vec<u8>) -> Files {
|
|
let mut f = BTreeMap::new();
|
|
f.insert("v.obu".to_string(), stream);
|
|
f
|
|
}
|
|
|
|
fn case(stream: Vec<u8>, edit: &dyn Fn(String) -> String) -> String {
|
|
let fl = floor_of(stream);
|
|
pack(&policy_for(&fl, edit), fl, None)
|
|
}
|
|
|
|
/// §7.3: each reason a supplied floor is refused with, before anything is written.
|
|
#[test]
|
|
fn every_floor_reason() {
|
|
let u = units();
|
|
let id = |p: String| p;
|
|
assert_eq!(case(golden_floor(), &id), "ok");
|
|
// No floor rung.
|
|
assert_eq!(
|
|
case(golden_floor(), &|p: String| p.replace("[[ladder]]\nid = \"f0\"\ncomponent = \"v\"\ncapability = \"wai.video.av1\"\nrole = \"floor\"\nparams = { source = \"supplied\" }\n", "")),
|
|
"floor-missing"
|
|
);
|
|
// A listing the policy does not pin.
|
|
let fl = floor_of(golden_floor());
|
|
let mut pol = String::from_utf8(policy_for(&fl, &id)).unwrap();
|
|
let at = pol.find("supplied_listing = \"").unwrap() + "supplied_listing = \"".len();
|
|
pol.replace_range(at..at + 64, &"0".repeat(64));
|
|
assert_eq!(pack(pol.as_bytes(), fl, None), "floor-listing-mismatch");
|
|
// No stream for the component.
|
|
let mut other = BTreeMap::new();
|
|
other.insert("w.obu".to_string(), golden_floor());
|
|
assert_eq!(pack(&policy_for(&other, &id), other, None), "floor-stream-missing");
|
|
// A stream cut inside its last unit; an empty stream.
|
|
let g = golden_floor();
|
|
assert_eq!(case(g[..g.len() - 3].to_vec(), &id), "floor-av1-malformed");
|
|
assert_eq!(case(Vec::new(), &id), "floor-av1-malformed");
|
|
// One unit short.
|
|
assert_eq!(case(u[..u.len() - 1].concat(), &id), "floor-frame-count");
|
|
// The piece of frames 85 to 134 opening on an inter frame, with no key frame
|
|
// anywhere in it.
|
|
let mut inter = u.clone();
|
|
inter[85] = u[86].clone();
|
|
assert_eq!(case(inter.concat(), &id), "floor-keyframe-mismatch");
|
|
// A key frame inside piece 2: refused, unless extra key frames are allowed.
|
|
let mut extra = u.clone();
|
|
extra[86] = u[85].clone();
|
|
assert_eq!(case(extra.concat(), &id), "floor-keyframe-mismatch");
|
|
assert_eq!(case(extra.concat(), &|p: String| p.replace("[floor]\n", "[floor]\nextra_keyframes = true\n")), "ok");
|
|
}
|
|
|
|
/// §11.4: what this revision does not pack yet is refused by name, every
|
|
/// `pack-capability-unsupported` naming its capability; what it packs now (refinement
|
|
/// rungs, a rule's ladder subset, a still) packs.
|
|
#[test]
|
|
fn what_this_revision_does_not_pack() {
|
|
let fl = floor_of(golden_floor());
|
|
let run = |edit: &dyn Fn(String) -> String| pack(&policy_for(&fl, edit), fl.clone(), None);
|
|
// Another medium, and two components.
|
|
assert_eq!(run(&|p: String| p.replace("media = \"video\"", "media = \"image\"")), "pack-media-unsupported");
|
|
assert_eq!(
|
|
run(&|p: String| p.replace("[segmentation]", "[[component]]\nid = \"a\"\nmedia = \"audio\"\n\n[component.timebase]\nnum = 48000\nden = 1\n\n[[ladder]]\nid = \"ab\"\ncomponent = \"a\"\ncapability = \"wai.audio.flac\"\nrole = \"base\"\n\n[segmentation]")),
|
|
"pack-media-unsupported"
|
|
);
|
|
// A refinement rung, and a rule's ladder subset, pack.
|
|
let refinement = |cap: &str, over: &str| {
|
|
let (cap, over) = (cap.to_string(), over.to_string());
|
|
move |p: String| p.replace("[floor]", &format!("[[ladder]]\nid = \"r1\"\ncomponent = \"v\"\ncapability = \"{cap}\"\nrole = \"refinement\"\nrefines = \"{over}\"\nparams = {{ q = 6 }}\n\n[live]\nlayer_lag_ms = [0, 200]\n\n[floor]"))
|
|
};
|
|
assert_eq!(run(&refinement("wai.video.int_refine", "b0")), "ok");
|
|
assert_eq!(run(&|p: String| format!("{p}\n[[rule]]\nid = \"s\"\nwhen = {{ piece_kind = [\"static\"] }}\nthen = {{ ladder = [\"b0\", \"f0\"] }}\n")), "ok");
|
|
// A still as the base, alone or as a second base for a rule to choose: it packs.
|
|
assert_eq!(
|
|
run(&|p: String| p.replace("capability = \"wai.video.int_motion\"\nrole = \"base\"\nparams = { block = 16, search = 4, mv_precision = 1, q_candidates = [16, 12, 8], max_sse_px_milli = 30000 }", "capability = \"wai.image.png\"\nrole = \"base\"")),
|
|
"ok"
|
|
);
|
|
assert_eq!(run(&|p: String| p.replace("[floor]", "[[ladder]]\nid = \"b1\"\ncomponent = \"v\"\ncapability = \"wai.image.png\"\nrole = \"base\"\n\n[floor]")), "ok");
|
|
// A floor to encode.
|
|
assert_eq!(run(&|p: String| p.replace("params = { source = \"supplied\" }", "params = { source = \"encode\" }")), "pack-floor-encode-unsupported");
|
|
// A base of a capability it does not code; a refinement of one, or over a base it does
|
|
// not refine; a floor of another capability.
|
|
assert_eq!(
|
|
run(&|p: String| p.replace("[floor]", "[[ladder]]\nid = \"b1\"\ncomponent = \"v\"\ncapability = \"wai.video.av1\"\nrole = \"base\"\n\n[floor]")),
|
|
"pack-capability-unsupported:wai.video.av1"
|
|
);
|
|
assert_eq!(
|
|
run(&|p: String| p.replace("[floor]", "[[ladder]]\nid = \"r1\"\ncomponent = \"v\"\ncapability = \"wai.image.int_refine\"\nrole = \"refinement\"\nrefines = \"b0\"\n\n[live]\nlayer_lag_ms = [0, 200]\n\n[floor]")),
|
|
"pack-capability-unsupported:wai.image.int_refine"
|
|
);
|
|
assert_eq!(
|
|
run(&|p: String| refinement("wai.video.int_refine", "b1")(p.replace("[floor]", "[[ladder]]\nid = \"b1\"\ncomponent = \"v\"\ncapability = \"wai.image.png\"\nrole = \"base\"\n\n[floor]"))),
|
|
"pack-capability-unsupported:wai.video.int_refine"
|
|
);
|
|
assert_eq!(
|
|
run(&|p: String| p.replace("capability = \"wai.video.av1\"\nrole = \"floor\"\nparams = { source = \"supplied\" }", "capability = \"wai.image.png\"\nrole = \"floor\"")),
|
|
"pack-capability-unsupported:wai.image.png"
|
|
);
|
|
}
|
|
|
|
/// A meter that was asked for and gives no figure refuses the pack: a model past its
|
|
/// range, and one whose figure rounds to nothing.
|
|
#[test]
|
|
fn a_meter_with_no_figure_refuses_the_pack() {
|
|
let fl = floor_of(golden_floor());
|
|
let pol = policy_for(&fl, &|p: String| p);
|
|
let clock = |step: u64| {
|
|
let mut t = 0u64;
|
|
move || {
|
|
let v = t;
|
|
t += step;
|
|
v
|
|
}
|
|
};
|
|
let huge = wai::energy_meter::ModelMeter::new(clock(1_000_000), u64::MAX / 2);
|
|
assert_eq!(pack(&pol, fl.clone(), Some(Box::new(huge))), "pack-meter");
|
|
let tiny = wai::energy_meter::ModelMeter::new(clock(1), 1);
|
|
assert_eq!(pack(&pol, fl.clone(), Some(Box::new(tiny))), "pack-meter");
|
|
let fine = wai::energy_meter::ModelMeter::new(clock(1_000_000), 500);
|
|
assert_eq!(pack(&pol, fl, Some(Box::new(fine))), "ok");
|
|
}
|
|
|
|
/// The encoder's caps (the review's clipcap): a 4K piece of 50 frames is past the clip
|
|
/// cap (more than 2^30 samples), refused for that before anything is allocated or any
|
|
/// frame is read; the packer reports it `pack-encode`.
|
|
#[test]
|
|
fn the_encoder_refuses_a_piece_past_its_caps() {
|
|
let (h, w) = (2160u32, 3840u32);
|
|
let frames: Vec<Vec<u8>> = (0..50).map(|_| Vec::new()).collect();
|
|
let p = wai_pack::encode::MotionParams { block: 16, search: 7, mv_precision: 2, q_candidates: vec![8], max_sse_px_milli: 0 };
|
|
let e = wai_pack::encode::choose_q(&frames, h, w, &p).map(|q| q.q).unwrap_err();
|
|
assert_eq!((e.code(), e.detail()), ("too_large", "the clip exceeds the clip cap"));
|
|
// One frame fewer than the cap allows passes the caps (and is then refused for its
|
|
// frames, which hold nothing).
|
|
let n = ((1u64 << 30) / (3 * h as u64 * w as u64)) as usize;
|
|
let e = wai_pack::encode::choose_q(&frames[..n], h, w, &p).map(|q| q.q).unwrap_err();
|
|
assert_ne!(e.code(), "too_large", "{e}");
|
|
}
|
|
|
|
/// §11.4: `wai-pack pack` writes a package only into a new or empty directory. Into one
|
|
/// that holds a file (a measurement claim of an earlier pack, here) it refuses with
|
|
/// `pack-out-not-empty` before any work, and leaves the directory as it was; into an
|
|
/// empty one it packs a package that verifies.
|
|
#[test]
|
|
fn the_packer_refuses_an_out_directory_that_is_not_empty() {
|
|
let bin = env!("CARGO_BIN_EXE_wai-pack");
|
|
let dir = std::env::temp_dir().join(format!("wai-pack-out-{}", std::process::id()));
|
|
let _ = std::fs::remove_dir_all(&dir);
|
|
std::fs::create_dir_all(&dir).unwrap();
|
|
let src = dir.join("v.y4m");
|
|
std::fs::write(&src, wai_pack::fixture::generate("video-base-01").unwrap()).unwrap();
|
|
let golden = corpus().join("packs/video-base-01");
|
|
let run = |out: &Path| {
|
|
std::process::Command::new(bin)
|
|
.arg("pack")
|
|
.arg("--policy")
|
|
.arg(golden.join("policy.toml"))
|
|
.arg("--out")
|
|
.arg(out)
|
|
.arg("--key")
|
|
.arg(golden.join("key.hex"))
|
|
.arg("--source")
|
|
.arg(format!("v={}", src.display()))
|
|
.output()
|
|
.unwrap()
|
|
};
|
|
// A directory holding an earlier pack's measurement claim.
|
|
let stale = dir.join("stale");
|
|
std::fs::create_dir_all(stale.join("claims")).unwrap();
|
|
std::fs::write(stale.join("claims/old.measurement.json"), b"{}").unwrap();
|
|
let o = run(&stale);
|
|
assert!(!o.status.success());
|
|
let err = String::from_utf8_lossy(&o.stderr);
|
|
assert!(err.contains("pack-out-not-empty"), "{err}");
|
|
let left = wai_pack::verify::read_strict(&stale).unwrap();
|
|
assert_eq!(left.keys().collect::<Vec<_>>(), ["claims/old.measurement.json"], "nothing written");
|
|
// A file where the directory would be.
|
|
let file = dir.join("a-file");
|
|
std::fs::write(&file, b"x").unwrap();
|
|
assert!(String::from_utf8_lossy(&run(&file).stderr).contains("pack-out-not-empty"));
|
|
// An empty directory, and one that does not exist yet.
|
|
std::fs::create_dir_all(dir.join("empty")).unwrap();
|
|
for out in [dir.join("empty"), dir.join("new")] {
|
|
let o = run(&out);
|
|
assert!(o.status.success(), "{}", String::from_utf8_lossy(&o.stderr));
|
|
assert!(wai_pack::verify::verify_dir(&out).is_ok(), "the package verifies");
|
|
}
|
|
let _ = std::fs::remove_dir_all(&dir);
|
|
}
|