open-standards/joule-code/joule-code-rs/crates/lockstep-quic/MACOS_LAUNCH_CONTEXT.md

Upstream quinn bug: accept-then-dial fails on some Apple-Silicon macOS hosts

This documents why lockstep-quic ships a star topology in addition to the peer-to-peer run mesh. The mesh is correct and works on most hosts and on loopback; it cannot form a full cross-machine mesh across two of our test laptops because of an upstream quinn / macOS issue, not a bug in this crate.

Symptom

A quinn Endpoint that has accepted an incoming connection and then dials an outbound connection to another machine never completes the outbound handshake — connect().await hangs until timeout. A dial without a prior accepted incoming works fine, so the trigger is the combination: (endpoint has an established inbound connection) → (dial outbound to another host).

It tracks the chip generation, not form factor or network config — tested on four Apple-Silicon machines, all macOS 26.5.1 (25F80), same bridge0 link:

Machine	Chip	Form factor	Result
MacBook Pro	M4 Max	laptop (multi-interface)	TIMED OUT
MacBook Pro	M5 Max	laptop	TIMED OUT
Mac Studio	M4 Max	desktop, single-interface	TIMED OUT
Mac Studio	M3 Ultra	desktop	CONNECTED

So it is not the laptops' multi-interface setup — a single-interface M4 Max Mac Studio fails identically. It reproduces on M4 Max and M5 Max, not on M3 Ultra, on the same OS build.

This explains the lockstep behaviour exactly:

• run mesh n=2 where the laptop is the pure dialer (index 0, accepts nothing): works.
• run mesh n≥4 where a laptop node accepts a peer and then dials another: hangs.
• star: the laptops only ever accept (from the hub); the send-capable Studio hub does all dialing — works (cross-machine n=4 BFT, verified).

Ruled out

Raw std::net::UdpSocket bound to the same port, with all of quinn-udp's socket options (IP_RECVTOS, IP_RECVDSTADDR, IP_DONTFRAG), non-blocking, an ECN IP_TOS control message, and a 1200-byte payload, sends to the peer fine. The macOS firewall is off on the laptops; Tailscale is off/absent; the route is a direct bridge0 link (MTU 9000, ARP resolved); ping works. The failure is not reproducible from raw socket operations — only through a quinn endpoint in the accept-then-dial state.

Minimal reproduction (raw quinn, ~60 lines, no joule-moq)

// Cargo.toml: quinn = "0.11", rustls = { version="0.23", features=["ring","std"] },
//             rcgen = "0.14", tokio = { version="1", features=["rt-multi-thread","macros","time"] }
use std::{sync::Arc, time::Duration};
use quinn::{ClientConfig, Endpoint, ServerConfig};

#[derive(Debug)]
struct SkipVerify;
impl rustls::client::danger::ServerCertVerifier for SkipVerify {
    fn verify_server_cert(&self, _: &rustls::pki_types::CertificateDer, _: &[rustls::pki_types::CertificateDer],
        _: &rustls::pki_types::ServerName, _: &[u8], _: rustls::pki_types::UnixTime)
        -> Result<rustls::client::danger::ServerCertVerified, rustls::Error> {
        Ok(rustls::client::danger::ServerCertVerified::assertion()) }
    fn verify_tls12_signature(&self, _: &[u8], _: &rustls::pki_types::CertificateDer, _: &rustls::DigitallySignedStruct)
        -> Result<rustls::client::danger::HandshakeSignatureValid, rustls::Error> {
        Ok(rustls::client::danger::HandshakeSignatureValid::assertion()) }
    fn verify_tls13_signature(&self, _: &[u8], _: &rustls::pki_types::CertificateDer, _: &rustls::DigitallySignedStruct)
        -> Result<rustls::client::danger::HandshakeSignatureValid, rustls::Error> {
        Ok(rustls::client::danger::HandshakeSignatureValid::assertion()) }
    fn supported_verify_schemes(&self) -> Vec<rustls::SignatureScheme> {
        rustls::crypto::ring::default_provider().signature_verification_algorithms.supported_schemes() }
}
fn server_cfg() -> ServerConfig {
    let c = rcgen::generate_simple_self_signed(vec!["localhost".into()]).unwrap();
    let key = rustls::pki_types::PrivatePkcs8KeyDer::from(c.signing_key.serialize_der());
    ServerConfig::with_single_cert(vec![rustls::pki_types::CertificateDer::from(c.cert)], key.into()).unwrap()
}
fn client_cfg() -> ClientConfig {
    let crypto = rustls::ClientConfig::builder().dangerous()
        .with_custom_certificate_verifier(Arc::new(SkipVerify)).with_no_client_auth();
    ClientConfig::new(Arc::new(quinn::crypto::rustls::QuicClientConfig::try_from(crypto).unwrap()))
}

#[tokio::main]
async fn main() {
    let _ = rustls::crypto::ring::default_provider().install_default();
    let a: Vec<String> = std::env::args().collect();
    match a[1].as_str() {
        // host C (dial target): plain server
        "server" => {
            let ep = Endpoint::server(server_cfg(), a[2].parse().unwrap()).unwrap();
            if let Some(i) = ep.accept().await { let _ = i.await; }
            tokio::time::sleep(Duration::from_secs(3)).await;
        }
        // host A: accept ONE incoming (from a feeder), THEN dial host C. Hangs on the laptops.
        "meshnode2" => {
            let mut ep = Endpoint::server(server_cfg(), a[2].parse().unwrap()).unwrap();
            ep.set_default_client_config(client_cfg());
            let _held = ep.accept().await.unwrap().await.unwrap();           // established inbound
            match tokio::time::timeout(Duration::from_secs(5),
                async { ep.connect(a[3].parse().unwrap(), "localhost").unwrap().await }).await {
                Ok(Ok(c)) => println!("CONNECTED to {}", c.remote_address()),
                Ok(Err(e)) => println!("connect error: {e}"),
                Err(_) => println!("TIMED OUT (no handshake response)"),
            }
        }
        // feeder (any send-capable host): give meshnode2 an inbound connection
        "client" => {
            let mut ep = Endpoint::client("0.0.0.0:0".parse().unwrap()).unwrap();
            ep.set_default_client_config(client_cfg());
            let _ = ep.connect(a[2].parse().unwrap(), "localhost").unwrap().await;
            tokio::time::sleep(Duration::from_secs(2)).await;
        }
        _ => {}
    }
}

Run (C = dial target, A = the host under test, F = feeder):

on C:  repro server   0.0.0.0:7503
on A:  repro meshnode2 0.0.0.0:7502 <C>:7503
on F:  repro client   <A>:7502

On the M4/M5 laptops, A prints TIMED OUT. On the M3 Studio, A prints CONNECTED.

Status

• Practical resolution: the star mode (signed votes, send-capable hub) gives BFT-safe cross-machine consensus today — see the crate's star subcommand.
• Upstream: filed as quinn-rs/quinn#2685 (https://github.com/quinn-rs/quinn/issues/2685) with the reproduction above; it is independent of this crate's consensus logic.