# Upstream quinn bug: accept-then-dial fails on some Apple-Silicon macOS hosts This documents why `lockstep-quic` ships a `star` topology in addition to the peer-to-peer `run` mesh. The mesh is correct and works on most hosts and on loopback; it cannot form a full cross-machine mesh across two of our test laptops because of an **upstream quinn / macOS issue**, not a bug in this crate. ## Symptom A quinn `Endpoint` that has **accepted an incoming connection** and then **dials an outbound connection** to another machine never completes the outbound handshake — `connect().await` hangs until timeout. A dial **without** a prior accepted incoming works fine, so the trigger is the combination: *(endpoint has an established inbound connection) → (dial outbound to another host)*. It **tracks the chip generation**, not form factor or network config — tested on four Apple-Silicon machines, all macOS 26.5.1 (25F80), same `bridge0` link: | Machine | Chip | Form factor | Result | |---|---|---|---| | MacBook Pro | M4 Max | laptop (multi-interface) | TIMED OUT | | MacBook Pro | M5 Max | laptop | TIMED OUT | | Mac Studio | M4 Max | desktop, single-interface | TIMED OUT | | Mac Studio | M3 Ultra | desktop | CONNECTED | So it is **not** the laptops' multi-interface setup — a single-interface M4 Max Mac Studio fails identically. It reproduces on **M4 Max and M5 Max**, not on **M3 Ultra**, on the same OS build. This explains the lockstep behaviour exactly: - `run` mesh n=2 where the laptop is the pure dialer (index 0, accepts nothing): **works**. - `run` mesh n≥4 where a laptop node accepts a peer and then dials another: **hangs**. - `star`: the laptops only ever *accept* (from the hub); the send-capable Studio hub does all dialing — **works** (cross-machine n=4 BFT, verified). ## Ruled out Raw `std::net::UdpSocket` bound to the same port, with all of quinn-udp's socket options (`IP_RECVTOS`, `IP_RECVDSTADDR`, `IP_DONTFRAG`), non-blocking, an ECN `IP_TOS` control message, and a 1200-byte payload, sends to the peer **fine**. The macOS firewall is off on the laptops; Tailscale is off/absent; the route is a direct `bridge0` link (MTU 9000, ARP resolved); `ping` works. The failure is **not** reproducible from raw socket operations — only through a quinn endpoint in the accept-then-dial state. ## Minimal reproduction (raw quinn, ~60 lines, no joule-moq) ```rust // Cargo.toml: quinn = "0.11", rustls = { version="0.23", features=["ring","std"] }, // rcgen = "0.14", tokio = { version="1", features=["rt-multi-thread","macros","time"] } use std::{sync::Arc, time::Duration}; use quinn::{ClientConfig, Endpoint, ServerConfig}; #[derive(Debug)] struct SkipVerify; impl rustls::client::danger::ServerCertVerifier for SkipVerify { fn verify_server_cert(&self, _: &rustls::pki_types::CertificateDer, _: &[rustls::pki_types::CertificateDer], _: &rustls::pki_types::ServerName, _: &[u8], _: rustls::pki_types::UnixTime) -> Result { Ok(rustls::client::danger::ServerCertVerified::assertion()) } fn verify_tls12_signature(&self, _: &[u8], _: &rustls::pki_types::CertificateDer, _: &rustls::DigitallySignedStruct) -> Result { Ok(rustls::client::danger::HandshakeSignatureValid::assertion()) } fn verify_tls13_signature(&self, _: &[u8], _: &rustls::pki_types::CertificateDer, _: &rustls::DigitallySignedStruct) -> Result { Ok(rustls::client::danger::HandshakeSignatureValid::assertion()) } fn supported_verify_schemes(&self) -> Vec { rustls::crypto::ring::default_provider().signature_verification_algorithms.supported_schemes() } } fn server_cfg() -> ServerConfig { let c = rcgen::generate_simple_self_signed(vec!["localhost".into()]).unwrap(); let key = rustls::pki_types::PrivatePkcs8KeyDer::from(c.signing_key.serialize_der()); ServerConfig::with_single_cert(vec![rustls::pki_types::CertificateDer::from(c.cert)], key.into()).unwrap() } fn client_cfg() -> ClientConfig { let crypto = rustls::ClientConfig::builder().dangerous() .with_custom_certificate_verifier(Arc::new(SkipVerify)).with_no_client_auth(); ClientConfig::new(Arc::new(quinn::crypto::rustls::QuicClientConfig::try_from(crypto).unwrap())) } #[tokio::main] async fn main() { let _ = rustls::crypto::ring::default_provider().install_default(); let a: Vec = std::env::args().collect(); match a[1].as_str() { // host C (dial target): plain server "server" => { let ep = Endpoint::server(server_cfg(), a[2].parse().unwrap()).unwrap(); if let Some(i) = ep.accept().await { let _ = i.await; } tokio::time::sleep(Duration::from_secs(3)).await; } // host A: accept ONE incoming (from a feeder), THEN dial host C. Hangs on the laptops. "meshnode2" => { let mut ep = Endpoint::server(server_cfg(), a[2].parse().unwrap()).unwrap(); ep.set_default_client_config(client_cfg()); let _held = ep.accept().await.unwrap().await.unwrap(); // established inbound match tokio::time::timeout(Duration::from_secs(5), async { ep.connect(a[3].parse().unwrap(), "localhost").unwrap().await }).await { Ok(Ok(c)) => println!("CONNECTED to {}", c.remote_address()), Ok(Err(e)) => println!("connect error: {e}"), Err(_) => println!("TIMED OUT (no handshake response)"), } } // feeder (any send-capable host): give meshnode2 an inbound connection "client" => { let mut ep = Endpoint::client("0.0.0.0:0".parse().unwrap()).unwrap(); ep.set_default_client_config(client_cfg()); let _ = ep.connect(a[2].parse().unwrap(), "localhost").unwrap().await; tokio::time::sleep(Duration::from_secs(2)).await; } _ => {} } } ``` Run (C = dial target, A = the host under test, F = feeder): ``` on C: repro server 0.0.0.0:7503 on A: repro meshnode2 0.0.0.0:7502 :7503 on F: repro client :7502 ``` On the M4/M5 laptops, A prints `TIMED OUT`. On the M3 Studio, A prints `CONNECTED`. ## Status - Practical resolution: the `star` mode (signed votes, send-capable hub) gives BFT-safe cross-machine consensus today — see the crate's `star` subcommand. - Upstream: filed as **quinn-rs/quinn#2685** () with the reproduction above; it is independent of this crate's consensus logic.