bc06242f9f
Several tests of one binary write entries of the same sidecar (three of them write tests/vectors/int_video/sha256.json), each by reading the file, inserting its entry and writing it back. Under parallel test threads two writes could interleave and drop an entry. Each write now holds a process-wide lock. The support module documents the command that regenerates the sidecars under tests/vectors/ and neural-conformance/. Regenerated with that command locally, every sidecar came out byte for byte as committed, and the same tests then passed in check mode. Co-Authored-By: Claude Opus 5.5 <[email protected]>
76 lines
3.4 KiB
Rust
76 lines
3.4 KiB
Rust
//! The SHA-256 conformance digests of decoded output (SPEC §7, *Digests*), kept in a
|
|
//! `sha256.json` sidecar beside each committed vector, so no committed vector file
|
|
//! changes. A sidecar entry names the buffer it covers and gives its SHA-256 as 64
|
|
//! lowercase hexadecimal digits; for a sequence of frames, each frame's digest and
|
|
//! the clip digest, the SHA-256 of the frames' 32-byte digests concatenated in output
|
|
//! order.
|
|
//!
|
|
//! The tests that decode a vector check its legacy diagnostic (FNV-1a or BLAKE3)
|
|
//! first, which an independent reference pinned, and then the SHA-256 of the same
|
|
//! buffer against the sidecar. With `WAI_WRITE_SHA256_SIDECARS=1` they write the
|
|
//! sidecar entry instead — only after the legacy check has passed. Several tests of
|
|
//! one binary write entries of the same sidecar, so each write (read, insert,
|
|
//! write back) holds a process-wide lock and parallel test threads lose no entry.
|
|
//! The sidecars under `tests/vectors/` and `neural-conformance/` are regenerated with
|
|
//!
|
|
//! ```text
|
|
//! WAI_WRITE_SHA256_SIDECARS=1 cargo test --no-default-features \
|
|
//! --features neural_int,video_int,audio_int,world_int,splat_int,generate \
|
|
//! --test int_kernels_vectors --test generate
|
|
//! ```
|
|
//!
|
|
//! and then checked by the same command without the variable.
|
|
#![allow(dead_code)]
|
|
|
|
use std::path::Path;
|
|
use std::sync::Mutex;
|
|
|
|
use serde_json::{json, Value};
|
|
|
|
fn hex(d: &[u8]) -> String {
|
|
d.iter().map(|b| format!("{b:02x}")).collect()
|
|
}
|
|
|
|
/// SHA-256 of `bytes`, hex.
|
|
pub fn sha256_hex(bytes: &[u8]) -> String {
|
|
hex(&wai::pin::sha256(bytes))
|
|
}
|
|
|
|
/// The clip digest of `frames`: SHA-256 over the frames' 32-byte SHA-256 digests.
|
|
pub fn clip_sha256(frames: &[Vec<u8>]) -> String {
|
|
let all: Vec<u8> = frames.iter().flat_map(|f| wai::pin::sha256(f)).collect();
|
|
sha256_hex(&all)
|
|
}
|
|
|
|
/// Held for each sidecar write, so two tests never interleave a read and a write.
|
|
static WRITE: Mutex<()> = Mutex::new(());
|
|
|
|
fn write_or_check(dir: &Path, key: &str, entry: Value) {
|
|
let path = dir.join("sha256.json");
|
|
if std::env::var("WAI_WRITE_SHA256_SIDECARS").as_deref() == Ok("1") {
|
|
// A test that panicked while holding the lock wrote nothing partial that
|
|
// matters here: the next write reads the file afresh.
|
|
let _held = WRITE.lock().unwrap_or_else(|e| e.into_inner());
|
|
let mut doc: serde_json::Map<String, Value> = std::fs::read(&path).ok()
|
|
.and_then(|b| serde_json::from_slice(&b).ok()).unwrap_or_default();
|
|
doc.insert(key.to_string(), entry);
|
|
let mut s = serde_json::to_string_pretty(&Value::Object(doc)).unwrap();
|
|
s.push('\n');
|
|
std::fs::write(&path, s).unwrap();
|
|
return;
|
|
}
|
|
let doc: Value = serde_json::from_slice(&std::fs::read(&path)
|
|
.unwrap_or_else(|e| panic!("read {}: {e}", path.display()))).unwrap();
|
|
assert_eq!(doc[key], entry, "{}: entry {key}", path.display());
|
|
}
|
|
|
|
/// Check (or write) the SHA-256 of one decoded buffer.
|
|
pub fn one(dir: &Path, key: &str, buffer: &str, bytes: &[u8]) {
|
|
write_or_check(dir, key, json!({ "buffer": buffer, "sha256": sha256_hex(bytes) }));
|
|
}
|
|
|
|
/// Check (or write) the per-frame and clip SHA-256 of a decoded sequence.
|
|
pub fn frames(dir: &Path, key: &str, buffer: &str, frames: &[Vec<u8>]) {
|
|
let per: Vec<String> = frames.iter().map(|f| sha256_hex(f)).collect();
|
|
write_or_check(dir, key, json!({ "buffer": buffer, "frames_sha256": per, "clip_sha256": clip_sha256(frames) }));
|
|
}
|