6575ff1eb7
openpay/ (+3 crates)
op-bnpl/ Affirm + Klarna + Afterpay/Clearpay rails. Full lifecycle
typestate (Initiated→Approved→Captured→Settled→Refunded).
Per-provider sandbox+production base URLs, regions (NA/EU/OC
for Klarna; US/AuNz/Gb/Ca/Eu for Afterpay). HMAC-SHA256
webhook verification with constant-time compare. 61 tests.
op-batch/ NACHA ACH (94-char records, PPD/CCD/WEB/TEL/CTX, R01-R99
return codes, Same-Day ACH cutoffs); SEPA pain.001.001.09
(CT) + pain.008.001.08 (DD) with SCT-Inst flagging; Fedwire
MT103/MT202/CHIPS + pacs.008.001.12 / pacs.009.001.10 +
UETR; Bacs Standard 18. Per-rail cutoff scheduler. Exception
taxonomy unified across rails. 51 tests.
op-screening/ OFAC SDN + EU consolidated + UN consolidated + HMT UK
consolidated + Australia DFAT + Canada SEMA + Japan MOF.
Levenshtein + Jaro-Winkler + Soundex + Metaphone + token
Jaccard scoring. Unicode NFKD normalisation + Cyrillic /
Arabic transliteration. Ed25519-signed BLAKE3-chained
audit log. BSA helpers: CTR + SAR pattern detection
(structuring, round-dollar, rapid-in-out). 43 tests.
smart-byte/smart-byte-rs/ (+3 crates)
smart-byte-bbs/ BBS+ signatures (IETF draft-irtf-cfrg-bbs-signatures-08).
BLS12-381 keys, signatures, disclosure proofs with
cryptographic unlinkability. bbs-2023 Data Integrity
cryptosuite. AnonCreds-shaped link secrets +
predicate proofs. SecretKey zeroize-on-drop. 49 tests.
smart-byte-didcomm/ DIDComm v2 messaging per DIF spec. Pack-encrypted
(ECDH-1PU + ECDH-ES via X25519), pack-signed (JWS),
pack-plaintext. Aries protocols: issue-credential v3,
present-proof v3, basic-message, trust-ping, discover-
features, out-of-band, coordinate-mediation, message-
pickup. Mediator role with pluggable storage. 23 tests.
smart-byte-mdl/ ISO/IEC 18013-5 mobile driver's licence + 18013-7
online presentment. mDoc + MSO with salted-hash
selective disclosure (per item; SHA-256 digest tree).
org.iso.18013.5.1 namespace + AAMVA add-on. ES256
device-auth + issuer-auth. QR-code device-engagement
scaffolding. 28 tests.
eoc/eoc-rs/ (+3 crates)
eoc-local/ Local-inference backends behind one NeuralBackend:
llama.cpp (GGUF, Q4/Q5/Q6/Q8/F16/F32 quantisation),
MLX (Apple Silicon, feature-gated), MLC-LLM (TVM, feature-
gated marker), ONNX Runtime. GGUF file-format reader.
HuggingFace tokenizers. Greedy / top-k / top-p / temperature /
Mirostat samplers. Model registry scanning ~/.eoc/models/.
Joule attribution via eoc-meter. 39 tests.
eoc-rerank/ BM25 sparse + HNSW dense + RRF + min-max hybrid fusion.
Cohere v3.5 + Voyage v2 + bge-reranker v2-m3 + ms-marco
MiniLM + ColBERT v2 rerankers. Two-stage pipeline:
retrieve top-K hybrid → cross-encoder rerank → top-N.
KV-stage recovery shim into the cascade. 31 tests.
eoc-eval/ 15 canonical eval harnesses: MMLU, MMLU-Pro, GPQA,
HumanEval (pass@1 with python3 sandbox), BBH, IFEval (25
verifiable constraints), AlpacaEval, AGIEval, HellaSwag,
ARC (Easy + Challenge), TruthfulQA (MC1+MC2), BoolQ,
GSM8K, MATH (boxed-answer equivalence), Winogrande. Real
questions in BuiltinSamples. EvalRunner emits joules-per-
correct alongside accuracy. 49 tests.
Quality:
- cargo check across all three workspaces: clean
- cargo clippy --all-targets -- -D warnings on every new crate: clean
- 374 new tests, all passing
- #![forbid(unsafe_code)] across every new crate
- No unwrap/expect outside #[cfg(test)]
Rust 1.95.0 edition 2024 (inherited via root rust-toolchain.toml).
78 lines
2.8 KiB
Rust
78 lines
2.8 KiB
Rust
//! EOC re-ranking and hybrid retrieval.
|
|
//!
|
|
//! Production RAG systems combine three retrieval signals:
|
|
//!
|
|
//! 1. **Dense retrieval** — cosine / inner-product search over an embedding
|
|
//! space (see [`eoc_embeddings`]). Strong on semantic paraphrase, weak
|
|
//! on rare-token / exact-match queries.
|
|
//! 2. **Sparse retrieval** — Okapi BM25 over an inverted index (see
|
|
//! [`bm25`]) or learned-sparse models like SPLADE (see [`sparse_lex`]).
|
|
//! Strong on rare tokens and exact matches.
|
|
//! 3. **Cross-encoder re-ranking** — a more accurate but slower scorer
|
|
//! that jointly encodes `(query, document)` pairs. Used as a second
|
|
//! stage on the top-K candidate set from the retriever.
|
|
//!
|
|
//! [`hybrid::HybridRetriever`] fuses dense and sparse rankings with
|
|
//! Reciprocal Rank Fusion (RRF) or min-max normalisation.
|
|
//! [`pipeline::RetrievalPipeline`] wires retrieve-then-rerank end to end,
|
|
//! the canonical RAG flow.
|
|
//!
|
|
//! The EOC cascade's KV stage calls into this crate via
|
|
//! [`cascade_integration`] when the cosine-similarity match lands below
|
|
//! a configurable threshold — the cross-encoder gets a chance to recover
|
|
//! candidates the embedder ranked low.
|
|
//!
|
|
//! ## Backends
|
|
//!
|
|
//! Re-ranker backends:
|
|
//!
|
|
//! * **Cohere** — `rerank-english-v3.0`, `rerank-multilingual-v3.0`,
|
|
//! `rerank-v3.5`. See [`cohere_rerank`].
|
|
//! * **Voyage** — `rerank-2`, `rerank-2-lite`. See [`voyage_rerank`].
|
|
//! * **bge-reranker** (local, feature `local`) — `bge-reranker-base`,
|
|
//! `bge-reranker-large`, `bge-reranker-v2-m3`. See [`bge_rerank`].
|
|
//! * **ms-marco MiniLM** (local, feature `local`) — see
|
|
//! [`ms_marco_rerank`].
|
|
//! * **ColBERT v2** (local, feature `local`) — multi-vector late-
|
|
//! interaction. See [`colbert`].
|
|
|
|
#![forbid(unsafe_code)]
|
|
#![warn(missing_docs)]
|
|
|
|
pub mod bm25;
|
|
pub mod cascade_integration;
|
|
pub mod cohere_rerank;
|
|
pub mod dense;
|
|
pub mod error;
|
|
pub mod hybrid;
|
|
pub mod pipeline;
|
|
pub mod reranker;
|
|
pub mod sparse_lex;
|
|
pub mod voyage_rerank;
|
|
|
|
#[cfg(feature = "local")]
|
|
pub mod bge_rerank;
|
|
#[cfg(feature = "local")]
|
|
pub mod colbert;
|
|
#[cfg(feature = "local")]
|
|
pub mod ms_marco_rerank;
|
|
|
|
pub use bm25::{Bm25Config, Bm25Index, Document};
|
|
pub use cohere_rerank::CohereReranker;
|
|
pub use dense::DenseIndex;
|
|
pub use error::{RerankError, RerankResult};
|
|
pub use hybrid::{FusionMethod, HybridRetriever};
|
|
pub use pipeline::RetrievalPipeline;
|
|
pub use reranker::{Candidate, Reranker, Retriever, ScoredCandidate};
|
|
pub use sparse_lex::{SparseLexIndex, SparseVector};
|
|
pub use voyage_rerank::VoyageReranker;
|
|
|
|
#[cfg(feature = "local")]
|
|
pub use bge_rerank::BgeReranker;
|
|
#[cfg(feature = "local")]
|
|
pub use colbert::ColBertIndex;
|
|
#[cfg(feature = "local")]
|
|
pub use ms_marco_rerank::MsMarcoReranker;
|
|
|
|
/// Canonical document identifier — caller-defined opaque string.
|
|
pub type DocId = String;
|