mesh: a cost model that can represent the request it is meant to refuse #70

Merged

dcharlot merged 1 commit from mesh/context-cost-profile into main

2026-09-03 15:11:18 -04:00

dcharlot commented

2026-08-30 15:52:51 -04:00

Owner

Ranked #2 by the 2026-08-29 sweep, which correctly described it as falsifying TX's own cost model.

fn estimate(&self, req: &ServeRequest) -> u64 {
    let tokens = req.prompt_tokens as u64 + req.max_tokens as u64;
    self.uj_per_token.saturating_mul(tokens)      // one scalar, architecture-blind
}

For a backend whose per-token cost genuinely doesn't depend on the prefix, that's correct — and it stays expressible. For one that re-reads its context per token it isn't, and the error has a direction.

A linear model applied to a super-linear backend under-estimates, always, and the gap widens with context. At a gate, under-estimating means admitting work the ceiling cannot cover: the request passes, runs, and overruns — the one outcome a joule ceiling exists to prevent.

A ceiling checked against a model that cannot represent the expensive case is not a ceiling. It is a formality that happens to pass.

"Refuse a long-context route before it burns the budget" was a capability the docs implied and the code could not deliver. The test that now pins it shows the same backend, the same request — admitted under a flat profile, refused under a calibrated one.

The design

CostProfile carries the shape: a fixed base, a flat per-token term, and µJ per token per 1024 tokens of context. CostProfile::linear reproduces the old arithmetic exactly.

No profile ships as a default, deliberately. The context term is a property of a particular backend on particular hardware and must be calibrated by measurement there. The study that motivated this change used one model per attention condition, so architecture is confounded with model family and parameter count — lifting its slopes into a constant would inherit that confound and dress it as a fact. Same reason BandingScale ships no canonical scale: the type expresses the shape, the operator supplies the numbers.

Details worth stating

The context term is charged against prompt_tokens. Output tokens extend the prefix as they're generated, so this stays a lower bound on a growing-context backend — deliberately, because the alternative is modelling that growth and then defending a second uncalibrated constant.
Every step saturates. An estimate that wrapped would come back small, and a small estimate is exactly what admits an unaffordable request.
is_context_aware() is inspectable, so a deployment gating long-context work can notice it's deciding with profiles that cannot represent the case.
The profile has a wire form, so a route decision can be re-derived from the receipt rather than trusted.

Backward compatibility is proven, not asserted

BackendSpec::new is unchanged in signature and behaviour, and re-running gen-vectors leaves the published vectors byte-identical on disk.

The source paper is cite-only (arXiv non-exclusive licence, no code); nothing from it is reproduced here.

Verification

38 mesh-rs tests pass (6 new) · conformance passes · published vectors unchanged after regeneration · no clippy warnings in mesh-router.

Ranked **#2** by the 2026-08-29 sweep, which correctly described it as **falsifying TX's own cost model**. ```rust fn estimate(&self, req: &ServeRequest) -> u64 { let tokens = req.prompt_tokens as u64 + req.max_tokens as u64; self.uj_per_token.saturating_mul(tokens) // one scalar, architecture-blind } ``` For a backend whose per-token cost genuinely doesn't depend on the prefix, that's correct — and it stays expressible. For one that re-reads its context per token it isn't, **and the error has a direction**. A linear model applied to a super-linear backend **under**-estimates, always, and the gap widens with context. At a gate, under-estimating means *admitting* work the ceiling cannot cover: the request passes, runs, and overruns — the one outcome a joule ceiling exists to prevent. > A ceiling checked against a model that cannot represent the expensive case is not a ceiling. It is a formality that happens to pass. "Refuse a long-context route before it burns the budget" was a capability the docs implied and the code could not deliver. The test that now pins it shows **the same backend, the same request** — admitted under a flat profile, refused under a calibrated one. ## The design `CostProfile` carries the *shape*: a fixed base, a flat per-token term, and µJ per token per 1024 tokens of context. `CostProfile::linear` reproduces the old arithmetic exactly. **No profile ships as a default, deliberately.** The context term is a property of a particular backend on particular hardware and must be calibrated by measurement there. The study that motivated this change used **one model per attention condition**, so architecture is confounded with model family and parameter count — lifting its slopes into a constant would inherit that confound and dress it as a fact. Same reason `BandingScale` ships no canonical scale: the type expresses the shape, the operator supplies the numbers. ## Details worth stating - **The context term is charged against `prompt_tokens`.** Output tokens extend the prefix as they're generated, so this stays a *lower bound* on a growing-context backend — deliberately, because the alternative is modelling that growth and then defending a second uncalibrated constant. - **Every step saturates.** An estimate that wrapped would come back *small*, and a small estimate is exactly what admits an unaffordable request. - **`is_context_aware()` is inspectable**, so a deployment gating long-context work can notice it's deciding with profiles that cannot represent the case. - **The profile has a wire form**, so a route decision can be re-derived from the receipt rather than trusted. ## Backward compatibility is proven, not asserted `BackendSpec::new` is unchanged in signature and behaviour, and **re-running `gen-vectors` leaves the published vectors byte-identical on disk**. The source paper is cite-only (arXiv non-exclusive licence, no code); nothing from it is reproduced here. ## Verification 38 mesh-rs tests pass (6 new) · conformance passes · published vectors unchanged after regeneration · no clippy warnings in `mesh-router`.

dcharlot added 1 commit

2026-08-30 15:52:51 -04:00

mesh: a cost model that can represent the request it is meant to refuse 327295732b

`BackendSpec::estimate` was `uj_per_token × (prompt_tokens + max_tokens)`: one
scalar, linear in token count, blind to how much context precedes each token.
For a backend whose per-token cost genuinely does not depend on the prefix that
is correct, and it stays expressible. For one that re-reads its context per
token it is not — and the error has a **direction**.

A linear model applied to a super-linear backend *under*-estimates, always, and
the gap widens with context. At a gate, under-estimating means admitting work
the ceiling cannot cover: the request passes, runs, and overruns — which is the
one outcome a joule ceiling exists to prevent. A ceiling checked against a model
that cannot represent the expensive case is not a ceiling, it is a formality
that happens to pass. So "refuse a long-context route before it burns the
budget" was a capability the docs implied and the code could not deliver, and
the test that now pins it shows the same backend admitted under a flat profile
and refused under a calibrated one, for the identical request.

`CostProfile` carries the *shape* — a fixed base, a flat per-token term, and
microjoules per token per 1024 tokens of context — and `CostProfile::linear`
reproduces the old arithmetic exactly.

**No profile ships as a default, deliberately.** The context term is a property
of a particular backend on particular hardware and has to be calibrated by
measurement there. The measured study that motivated this change used one model
per attention condition, so architecture is confounded with model family and
parameter count; lifting its slopes into a constant would inherit that confound
and dress it as a fact. Same reason `BandingScale` ships no canonical scale: the
type expresses the shape, the operator supplies the numbers.

Details worth stating:

- The context term is charged against `prompt_tokens`. Output tokens extend the
  prefix as they are generated, so this remains a lower bound on a
  growing-context backend — deliberately, because the alternative is to model
  that growth and then defend a second uncalibrated constant.
- Every step saturates. An estimate that wrapped would come back *small*, and a
  small estimate is exactly what admits an unaffordable request.
- `is_context_aware()` is inspectable, so a deployment gating long-context work
  can notice it is deciding with profiles that cannot represent the case.
- The profile has a wire form, so a route decision can be re-derived from the
  receipt rather than trusted.

Backward compatibility is proven, not asserted: `BackendSpec::new` is unchanged
in signature and behaviour, and re-running `gen-vectors` leaves the published
vectors byte-identical on disk.

Found by the 2026-08-29 sweep, which ranked this second and correctly described
it as falsifying TX's own cost model. The source paper is cite-only (arXiv
non-exclusive licence, no code), and nothing from it is reproduced here.

Verified: 38 mesh-rs tests pass (6 new), conformance passes, published vectors
unchanged after regeneration, no clippy warnings in mesh-router.

dcharlot merged commit 5055e69351 into main

2026-09-03 15:11:18 -04:00

dcharlot referenced this pull request from a commit

2026-09-03 15:11:19 -04:00

Merge pull request 'mesh: a cost model that can represent the request it is meant to refuse' (#70) from mesh/context-cost-profile into main

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!70

No description provided.

Rows
Columns