open-standards/joule-code/joule-code-rs/crates/lockstep-quic/MACOS_LAUNCH_CONTEXT.md

# Upstream quinn bug: accept-then-dial fails on some Apple-Silicon macOS hosts

This documents why `lockstep-quic` ships a `star` topology in addition to the
peer-to-peer `run` mesh. The mesh is correct and works on most hosts and on
loopback; it cannot form a full cross-machine mesh across two of our test
laptops because of an **upstream quinn / macOS issue**, not a bug in this crate.

## Symptom

A quinn `Endpoint` that has **accepted an incoming connection** and then **dials
an outbound connection** to another machine never completes the outbound
handshake — `connect().await` hangs until timeout. A dial **without** a prior
accepted incoming works fine, so the trigger is the combination: *(endpoint has
an established inbound connection) → (dial outbound to another host)*.

It **tracks the chip generation**, not form factor or network config — tested on
four Apple-Silicon machines, all macOS 26.5.1 (25F80), same `bridge0` link:

| Machine | Chip | Form factor | Result |
|---|---|---|---|
| MacBook Pro | M4 Max | laptop (multi-interface) | TIMED OUT |
| MacBook Pro | M5 Max | laptop | TIMED OUT |
| Mac Studio | M4 Max | desktop, single-interface | TIMED OUT |
| Mac Studio | M3 Ultra | desktop | CONNECTED |

So it is **not** the laptops' multi-interface setup — a single-interface M4 Max
Mac Studio fails identically. It reproduces on **M4 Max and M5 Max**, not on
**M3 Ultra**, on the same OS build.

This explains the lockstep behaviour exactly:
- `run` mesh n=2 where the laptop is the pure dialer (index 0, accepts nothing):
  **works**.
- `run` mesh n≥4 where a laptop node accepts a peer and then dials another:
  **hangs**.
- `star`: the laptops only ever *accept* (from the hub); the send-capable Studio
  hub does all dialing — **works** (cross-machine n=4 BFT, verified).

## Ruled out

Raw `std::net::UdpSocket` bound to the same port, with all of quinn-udp's socket
options (`IP_RECVTOS`, `IP_RECVDSTADDR`, `IP_DONTFRAG`), non-blocking, an ECN
`IP_TOS` control message, and a 1200-byte payload, sends to the peer **fine**.
The macOS firewall is off on the laptops; Tailscale is off/absent; the route is
a direct `bridge0` link (MTU 9000, ARP resolved); `ping` works. The failure is
**not** reproducible from raw socket operations — only through a quinn endpoint
in the accept-then-dial state.

## Minimal reproduction (raw quinn, ~60 lines, no joule-moq)

```rust
// Cargo.toml: quinn = "0.11", rustls = { version="0.23", features=["ring","std"] },
//             rcgen = "0.14", tokio = { version="1", features=["rt-multi-thread","macros","time"] }
use std::{sync::Arc, time::Duration};
use quinn::{ClientConfig, Endpoint, ServerConfig};

#[derive(Debug)]
struct SkipVerify;
impl rustls::client::danger::ServerCertVerifier for SkipVerify {
    fn verify_server_cert(&self, _: &rustls::pki_types::CertificateDer, _: &[rustls::pki_types::CertificateDer],
        _: &rustls::pki_types::ServerName, _: &[u8], _: rustls::pki_types::UnixTime)
        -> Result<rustls::client::danger::ServerCertVerified, rustls::Error> {
        Ok(rustls::client::danger::ServerCertVerified::assertion()) }
    fn verify_tls12_signature(&self, _: &[u8], _: &rustls::pki_types::CertificateDer, _: &rustls::DigitallySignedStruct)
        -> Result<rustls::client::danger::HandshakeSignatureValid, rustls::Error> {
        Ok(rustls::client::danger::HandshakeSignatureValid::assertion()) }
    fn verify_tls13_signature(&self, _: &[u8], _: &rustls::pki_types::CertificateDer, _: &rustls::DigitallySignedStruct)
        -> Result<rustls::client::danger::HandshakeSignatureValid, rustls::Error> {
        Ok(rustls::client::danger::HandshakeSignatureValid::assertion()) }
    fn supported_verify_schemes(&self) -> Vec<rustls::SignatureScheme> {
        rustls::crypto::ring::default_provider().signature_verification_algorithms.supported_schemes() }
}
fn server_cfg() -> ServerConfig {
    let c = rcgen::generate_simple_self_signed(vec!["localhost".into()]).unwrap();
    let key = rustls::pki_types::PrivatePkcs8KeyDer::from(c.signing_key.serialize_der());
    ServerConfig::with_single_cert(vec![rustls::pki_types::CertificateDer::from(c.cert)], key.into()).unwrap()
}
fn client_cfg() -> ClientConfig {
    let crypto = rustls::ClientConfig::builder().dangerous()
        .with_custom_certificate_verifier(Arc::new(SkipVerify)).with_no_client_auth();
    ClientConfig::new(Arc::new(quinn::crypto::rustls::QuicClientConfig::try_from(crypto).unwrap()))
}

#[tokio::main]
async fn main() {
    let _ = rustls::crypto::ring::default_provider().install_default();
    let a: Vec<String> = std::env::args().collect();
    match a[1].as_str() {
        // host C (dial target): plain server
        "server" => {
            let ep = Endpoint::server(server_cfg(), a[2].parse().unwrap()).unwrap();
            if let Some(i) = ep.accept().await { let _ = i.await; }
            tokio::time::sleep(Duration::from_secs(3)).await;
        }
        // host A: accept ONE incoming (from a feeder), THEN dial host C. Hangs on the laptops.
        "meshnode2" => {
            let mut ep = Endpoint::server(server_cfg(), a[2].parse().unwrap()).unwrap();
            ep.set_default_client_config(client_cfg());
            let _held = ep.accept().await.unwrap().await.unwrap();           // established inbound
            match tokio::time::timeout(Duration::from_secs(5),
                async { ep.connect(a[3].parse().unwrap(), "localhost").unwrap().await }).await {
                Ok(Ok(c)) => println!("CONNECTED to {}", c.remote_address()),
                Ok(Err(e)) => println!("connect error: {e}"),
                Err(_) => println!("TIMED OUT (no handshake response)"),
            }
        }
        // feeder (any send-capable host): give meshnode2 an inbound connection
        "client" => {
            let mut ep = Endpoint::client("0.0.0.0:0".parse().unwrap()).unwrap();
            ep.set_default_client_config(client_cfg());
            let _ = ep.connect(a[2].parse().unwrap(), "localhost").unwrap().await;
            tokio::time::sleep(Duration::from_secs(2)).await;
        }
        _ => {}
    }
}
```

Run (C = dial target, A = the host under test, F = feeder):
```
on C:  repro server   0.0.0.0:7503
on A:  repro meshnode2 0.0.0.0:7502 <C>:7503
on F:  repro client   <A>:7502
```
On the M4/M5 laptops, A prints `TIMED OUT`. On the M3 Studio, A prints
`CONNECTED`.

## Status

- Practical resolution: the `star` mode (signed votes, send-capable hub) gives
  BFT-safe cross-machine consensus today — see the crate's `star` subcommand.
- Upstream: filed as **quinn-rs/quinn#2685**
  (<https://github.com/quinn-rs/quinn/issues/2685>) with the reproduction above;
  it is independent of this crate's consensus logic.