mesh: a cost model that can represent the request it is meant to refuse #70
2026-08-30
mesh: a cost model that can represent the request it is meant to refuse
`BackendSpec::estimate` was `uj_per_token × (prompt_tokens + max_tokens)`: one scalar, linear in token count, blind to how much context precedes each token. For a backend whose per-token cost genuinely does not depend on the prefix that is correct, and it stays expressible. For one that re-reads its context per token it is not — and the error has a **direction**. A linear model applied to a super-linear backend *under*-estimates, always, and the gap widens with context. At a gate, under-estimating means admitting work the ceiling cannot cover: the request passes, runs, and overruns — which is the one outcome a joule ceiling exists to prevent. A ceiling checked against a model that cannot represent the expensive case is not a ceiling, it is a formality that happens to pass. So "refuse a long-context route before it burns the budget" was a capability the docs implied and the code could not deliver, and the test that now pins it shows the same backend admitted under a flat profile and refused under a calibrated one, for the identical request. `CostProfile` carries the *shape* — a fixed base, a flat per-token term, and microjoules per token per 1024 tokens of context — and `CostProfile::linear` reproduces the old arithmetic exactly. **No profile ships as a default, deliberately.** The context term is a property of a particular backend on particular hardware and has to be calibrated by measurement there. The measured study that motivated this change used one model per attention condition, so architecture is confounded with model family and parameter count; lifting its slopes into a constant would inherit that confound and dress it as a fact. Same reason `BandingScale` ships no canonical scale: the type expresses the shape, the operator supplies the numbers. Details worth stating: - The context term is charged against `prompt_tokens`. Output tokens extend the prefix as they are generated, so this remains a lower bound on a growing-context backend — deliberately, because the alternative is to model that growth and then defend a second uncalibrated constant. - Every step saturates. An estimate that wrapped would come back *small*, and a small estimate is exactly what admits an unaffordable request. - `is_context_aware()` is inspectable, so a deployment gating long-context work can notice it is deciding with profiles that cannot represent the case. - The profile has a wire form, so a route decision can be re-derived from the receipt rather than trusted. Backward compatibility is proven, not asserted: `BackendSpec::new` is unchanged in signature and behaviour, and re-running `gen-vectors` leaves the published vectors byte-identical on disk. Found by the 2026-08-29 sweep, which ranked this second and correctly described it as falsifying TX's own cost model. The source paper is cite-only (arXiv non-exclusive licence, no code), and nothing from it is reproduced here. Verified: 38 mesh-rs tests pass (6 new), conformance passes, published vectors unchanged after regeneration, no clippy warnings in mesh-router.