6575ff1eb7
openpay/ (+3 crates)
op-bnpl/ Affirm + Klarna + Afterpay/Clearpay rails. Full lifecycle
typestate (Initiated→Approved→Captured→Settled→Refunded).
Per-provider sandbox+production base URLs, regions (NA/EU/OC
for Klarna; US/AuNz/Gb/Ca/Eu for Afterpay). HMAC-SHA256
webhook verification with constant-time compare. 61 tests.
op-batch/ NACHA ACH (94-char records, PPD/CCD/WEB/TEL/CTX, R01-R99
return codes, Same-Day ACH cutoffs); SEPA pain.001.001.09
(CT) + pain.008.001.08 (DD) with SCT-Inst flagging; Fedwire
MT103/MT202/CHIPS + pacs.008.001.12 / pacs.009.001.10 +
UETR; Bacs Standard 18. Per-rail cutoff scheduler. Exception
taxonomy unified across rails. 51 tests.
op-screening/ OFAC SDN + EU consolidated + UN consolidated + HMT UK
consolidated + Australia DFAT + Canada SEMA + Japan MOF.
Levenshtein + Jaro-Winkler + Soundex + Metaphone + token
Jaccard scoring. Unicode NFKD normalisation + Cyrillic /
Arabic transliteration. Ed25519-signed BLAKE3-chained
audit log. BSA helpers: CTR + SAR pattern detection
(structuring, round-dollar, rapid-in-out). 43 tests.
smart-byte/smart-byte-rs/ (+3 crates)
smart-byte-bbs/ BBS+ signatures (IETF draft-irtf-cfrg-bbs-signatures-08).
BLS12-381 keys, signatures, disclosure proofs with
cryptographic unlinkability. bbs-2023 Data Integrity
cryptosuite. AnonCreds-shaped link secrets +
predicate proofs. SecretKey zeroize-on-drop. 49 tests.
smart-byte-didcomm/ DIDComm v2 messaging per DIF spec. Pack-encrypted
(ECDH-1PU + ECDH-ES via X25519), pack-signed (JWS),
pack-plaintext. Aries protocols: issue-credential v3,
present-proof v3, basic-message, trust-ping, discover-
features, out-of-band, coordinate-mediation, message-
pickup. Mediator role with pluggable storage. 23 tests.
smart-byte-mdl/ ISO/IEC 18013-5 mobile driver's licence + 18013-7
online presentment. mDoc + MSO with salted-hash
selective disclosure (per item; SHA-256 digest tree).
org.iso.18013.5.1 namespace + AAMVA add-on. ES256
device-auth + issuer-auth. QR-code device-engagement
scaffolding. 28 tests.
eoc/eoc-rs/ (+3 crates)
eoc-local/ Local-inference backends behind one NeuralBackend:
llama.cpp (GGUF, Q4/Q5/Q6/Q8/F16/F32 quantisation),
MLX (Apple Silicon, feature-gated), MLC-LLM (TVM, feature-
gated marker), ONNX Runtime. GGUF file-format reader.
HuggingFace tokenizers. Greedy / top-k / top-p / temperature /
Mirostat samplers. Model registry scanning ~/.eoc/models/.
Joule attribution via eoc-meter. 39 tests.
eoc-rerank/ BM25 sparse + HNSW dense + RRF + min-max hybrid fusion.
Cohere v3.5 + Voyage v2 + bge-reranker v2-m3 + ms-marco
MiniLM + ColBERT v2 rerankers. Two-stage pipeline:
retrieve top-K hybrid → cross-encoder rerank → top-N.
KV-stage recovery shim into the cascade. 31 tests.
eoc-eval/ 15 canonical eval harnesses: MMLU, MMLU-Pro, GPQA,
HumanEval (pass@1 with python3 sandbox), BBH, IFEval (25
verifiable constraints), AlpacaEval, AGIEval, HellaSwag,
ARC (Easy + Challenge), TruthfulQA (MC1+MC2), BoolQ,
GSM8K, MATH (boxed-answer equivalence), Winogrande. Real
questions in BuiltinSamples. EvalRunner emits joules-per-
correct alongside accuracy. 49 tests.
Quality:
- cargo check across all three workspaces: clean
- cargo clippy --all-targets -- -D warnings on every new crate: clean
- 374 new tests, all passing
- #![forbid(unsafe_code)] across every new crate
- No unwrap/expect outside #[cfg(test)]
Rust 1.95.0 edition 2024 (inherited via root rust-toolchain.toml).
72 lines
2.1 KiB
Rust
72 lines
2.1 KiB
Rust
//! `eoc-eval` — canonical LLM evaluation harnesses for the EOC stack.
|
|
//!
|
|
//! Each module wraps one published evaluation suite (MMLU, GPQA,
|
|
//! HumanEval, ...) behind a uniform [`Harness`] trait so the
|
|
//! [`runner::EvalRunner`] can drive any harness through any
|
|
//! [`eoc_neural::NeuralBackend`] and emit a comparable score plus
|
|
//! `joules-per-correct`.
|
|
//!
|
|
//! ## Quick start
|
|
//!
|
|
//! ```no_run
|
|
//! # use std::sync::Arc;
|
|
//! # use eoc_eval::{mmlu::Mmlu, runner::EvalRunner};
|
|
//! # use eoc_neural::EchoBackend;
|
|
//! # async fn ex() {
|
|
//! let runner = EvalRunner::new(
|
|
//! Box::new(Mmlu::new()),
|
|
//! Arc::new(EchoBackend::new()),
|
|
//! )
|
|
//! .with_model("echo");
|
|
//! let report = runner.run().await.expect("run");
|
|
//! println!("{} score={:.3} J/correct={}", report.harness, report.score, report.joules_per_correct);
|
|
//! # }
|
|
//! ```
|
|
//!
|
|
//! ## Crate layout
|
|
//!
|
|
//! - [`harness`] — the [`harness::Harness`] trait, [`harness::Metric`],
|
|
//! [`harness::DatasetSource`], [`harness::EvalCase`].
|
|
//! - [`runner`] — drives a harness end-to-end against a backend.
|
|
//! - One module per published harness (`mmlu`, `mmlu_pro`, `gpqa`,
|
|
//! `humaneval`, `bbh`, `ifeval`, `alpaca_eval`, `agi_eval`,
|
|
//! `hellaswag`, `arc`, `truthfulqa`, `boolq`, `gsm8k`, `math`,
|
|
//! `winogrande`).
|
|
//! - [`dataset_loader`] — HuggingFace Hub fetcher (behind the
|
|
//! `download` feature).
|
|
//! - [`builtin_samples`] — small hand-curated sample data embedded in
|
|
//! the crate; available without any features.
|
|
|
|
#![forbid(unsafe_code)]
|
|
#![warn(missing_docs)]
|
|
|
|
pub mod builtin_samples;
|
|
pub mod error;
|
|
pub mod harness;
|
|
pub mod mcq;
|
|
pub mod runner;
|
|
|
|
pub mod agi_eval;
|
|
pub mod alpaca_eval;
|
|
pub mod arc;
|
|
pub mod bbh;
|
|
pub mod boolq;
|
|
pub mod gpqa;
|
|
pub mod gsm8k;
|
|
pub mod hellaswag;
|
|
pub mod humaneval;
|
|
pub mod ifeval;
|
|
pub mod math;
|
|
pub mod mmlu;
|
|
pub mod mmlu_pro;
|
|
pub mod truthfulqa;
|
|
pub mod winogrande;
|
|
|
|
#[cfg(feature = "download")]
|
|
pub mod dataset_loader;
|
|
|
|
pub use error::{EvalError, Result};
|
|
pub use harness::{
|
|
DatasetSource, EvalCase, ExpectedAnswer, Harness, IfEvalConstraint, Metric, Response,
|
|
};
|
|
pub use runner::{EvalReport, EvalRunner};
|