eoc-route-learned: recommend the stage cheapest in expectation, not the likeliest #439

Merged

dcharlot merged 2 commits from fix/eoc-route-cheapest into main 2026-11-01 09:38:36 -05:00

Conversation 0 Commits 2 Files changed 10 +1111 -247

dcharlot commented 2026-10-22 07:23:08 -04:00

Owner

Copy link

The crate says a learned router "predicts the cheapest stage that is likely to satisfy the query". Three of its four routers recommended the stage likeliest to succeed instead. MF and both bandits took an argmax over success, and the bandits also ignored the joule cost passed to observe. Neural succeeds on everything and costs the most, so these routers learned to send every query to Neural. The LinUCB synthetic test asserted exactly that as its goal ("converges to best arm": Neural). This came from the 2026-10-19 sweep (item 17, "the bandit reward ignores joules"). Checked on main, the problem is wider than that item said.

evaluate had a second defect. It charged every row's own recorded cost whatever the router chose, and priced the "full cascade" at 4× that. So every router reported a lift of 3× its spend, whatever it did.

The rule

router::expected_joules prices a skip the way LearnedCascade dispatches it:

Neural      c_N
Kv, Graph   c_s + (1 − p_s) · c_N                              a miss falls through to Neural
Cache       c_C + (1 − p_C)(c_K + (1 − p_K)(c_G + (1 − p_G) c_N))   the full walk

cheapest_in_expectation picks the stage with the lowest expected joules. A tie goes to the cheaper stage. A probability that is NaN or outside [0, 1] can only make a stage cost more, never less.

Changes

• MF picks by the rule, over its per-stage chances and learned costs.
• LinUCB and Thompson learn each stage's cost from observe and pick by the rule. LinUCB uses optimistic chances (the UCB) and Thompson uses sampled ones, so an untried cheap stage still gets tried. Confidence is now the mean belief, not the bound or sample used to explore; otherwise the threshold policy's skip would be a coin toss. States saved before this change import with the default costs.
• observe's contract: each call reports one stage that ran and that stage's own joules. When a dispatch runs several stages, report each one; a miss is evidence too.
• train_logreg labels each query with its cheapest successful stage. Rows with the same embedding count as one query. The classifier's online update no longer reinforces a correct stage costlier than the one it predicts, since every query gets a correct Neural answer.
• evaluate replays each recommendation over per-stage outcomes:
  • It adds full_cascade_microjoules and unscored.
  • A query it cannot price is counted as unscored, not charged zero.
  • joules_per_correct is None when nothing was correct. Zero would read as the best possible figure.

What the tests show

On synthetic data whose coverage is soft and of the family the model fits, priced at each query's true chances:

	expected µJ (2 000 queries)
best possible choices	11 884 M
plain walk	11 889 M
MF with this rule	11 952 M (0.6% over best)
MF with the old argmax rule	85 150 M (7.2×)

With the default costs, Graph is 1% of a Neural call. The plain walk is then within 0.04% of the best, so routing has little to win. The router is held to within 1% of the best.

When Kv and Graph cost a real share of a Neural call, the walk is 4.5% above the best. At this training budget MF lands at about the walk's cost and does not capture that gap. Its learned chances are off by 5–9% per stage, and a wrong skip costs a whole Neural call. Training longer made it beat the walk by 2% while its chances got worse, so this is a limit of the estimator, not of the rule. The test therefore requires only that the router not lose to the walk, and the test doc records the measurement.

The bandits, with each stage that ran reporting its outcome:

• Graph resolves 95% and nothing cheaper resolves: both skip to Graph, at 2.7 M µJ a query.
• The old test's arms (0.1 / 0.2 / 0.3 / 0.8): both walk, where the old bandits picked Neural.
• Nothing short of Neural resolves: both come within 1% of going straight to Neural. They still probe Kv now and then, because a probe costs a thousandth of a Neural call.

Verification (build host)

• cargo test -p eoc-route-learned: 48 pass. cargo clippy -p eoc-route-learned --all-targets --no-deps -D warnings is clean.
• Mutation testing: 25 of 25 mutants caught. The mutants revert each part of the fix: each dispatch formula, the NaN and clamp handling, the tie rule, argmax in MF and Thompson, no optimism in LinUCB, dropping the cost in observe, confidence from the bound or sample, the classifier gate (removed, or made strict), the label choice, each evaluate pricing rule, the serde defaults, the EMA weight, and MF's confidence. Two survived the first run, the strict gate and MF's confidence, and each now has a test.
• cargo test --workspace --no-fail-fast for eoc-rs: 763 pass, 0 fail.
• eoc-cascade's hit_calibration_process_energy is flaky on the shared host, independent of this change. Run alone it failed 1 of 3 runs on main, and 3 of 3 on this branch, which does not touch eoc-cascade. It passed inside the full workspace run.
• Repo checks clean: tests-in-ci, orphan sources, spec refs, vanity metrics, comparative prose, public names.

🤖 Generated with Claude Code

The crate says a learned router "predicts the cheapest stage that is likely to satisfy the query". Three of its four routers recommended the stage **likeliest to succeed** instead. MF and both bandits took an argmax over success, and the bandits also ignored the joule cost passed to `observe`. Neural succeeds on everything and costs the most, so these routers learned to send every query to Neural. The LinUCB synthetic test asserted exactly that as its goal ("converges to best arm": Neural). This came from the 2026-10-19 sweep (item 17, "the bandit reward ignores joules"). Checked on main, the problem is wider than that item said. `evaluate` had a second defect. It charged every row's own recorded cost whatever the router chose, and priced the "full cascade" at 4× that. So every router reported a lift of 3× its spend, whatever it did. ## The rule `router::expected_joules` prices a skip the way `LearnedCascade` dispatches it: ```text Neural c_N Kv, Graph c_s + (1 − p_s) · c_N a miss falls through to Neural Cache c_C + (1 − p_C)(c_K + (1 − p_K)(c_G + (1 − p_G) c_N)) the full walk ``` `cheapest_in_expectation` picks the stage with the lowest expected joules. A tie goes to the cheaper stage. A probability that is NaN or outside [0, 1] can only make a stage cost more, never less. ## Changes - **MF** picks by the rule, over its per-stage chances and learned costs. - **LinUCB and Thompson** learn each stage's cost from `observe` and pick by the rule. LinUCB uses optimistic chances (the UCB) and Thompson uses sampled ones, so an untried cheap stage still gets tried. Confidence is now the mean belief, not the bound or sample used to explore; otherwise the threshold policy's skip would be a coin toss. States saved before this change import with the default costs. - **`observe`'s contract:** each call reports one stage that ran and that stage's own joules. When a dispatch runs several stages, report each one; a miss is evidence too. - **`train_logreg`** labels each query with its cheapest successful stage. Rows with the same embedding count as one query. The classifier's online update no longer reinforces a correct stage costlier than the one it predicts, since every query gets a correct Neural answer. - **`evaluate`** replays each recommendation over per-stage outcomes: - It adds `full_cascade_microjoules` and `unscored`. - A query it cannot price is counted as unscored, not charged zero. - `joules_per_correct` is `None` when nothing was correct. Zero would read as the best possible figure. ## What the tests show On synthetic data whose coverage is soft and of the family the model fits, priced at each query's true chances: | | expected µJ (2 000 queries) | |---|---| | best possible choices | 11 884 M | | plain walk | 11 889 M | | MF with this rule | 11 952 M (0.6% over best) | | MF with the old argmax rule | 85 150 M (7.2×) | With the default costs, Graph is 1% of a Neural call. The plain walk is then within 0.04% of the best, so routing has little to win. The router is held to within 1% of the best. When Kv and Graph cost a real share of a Neural call, the walk is 4.5% above the best. At this training budget MF lands at about the walk's cost and does not capture that gap. Its learned chances are off by 5–9% per stage, and a wrong skip costs a whole Neural call. Training longer made it beat the walk by 2% while its chances got worse, so this is a limit of the estimator, not of the rule. The test therefore requires only that the router not lose to the walk, and the test doc records the measurement. The bandits, with each stage that ran reporting its outcome: - Graph resolves 95% and nothing cheaper resolves: both skip to Graph, at 2.7 M µJ a query. - The old test's arms (0.1 / 0.2 / 0.3 / 0.8): both walk, where the old bandits picked Neural. - Nothing short of Neural resolves: both come within 1% of going straight to Neural. They still probe Kv now and then, because a probe costs a thousandth of a Neural call. ## Verification (build host) - `cargo test -p eoc-route-learned`: 48 pass. `cargo clippy -p eoc-route-learned --all-targets --no-deps -D warnings` is clean. - **Mutation testing: 25 of 25 mutants caught.** The mutants revert each part of the fix: each dispatch formula, the NaN and clamp handling, the tie rule, argmax in MF and Thompson, no optimism in LinUCB, dropping the cost in `observe`, confidence from the bound or sample, the classifier gate (removed, or made strict), the label choice, each `evaluate` pricing rule, the serde defaults, the EMA weight, and MF's confidence. Two survived the first run, the strict gate and MF's confidence, and each now has a test. - `cargo test --workspace --no-fail-fast` for eoc-rs: 763 pass, 0 fail. - `eoc-cascade`'s `hit_calibration_process_energy` is flaky on the shared host, independent of this change. Run alone it failed 1 of 3 runs on main, and 3 of 3 on this branch, which does not touch eoc-cascade. It passed inside the full workspace run. - Repo checks clean: tests-in-ci, orphan sources, spec refs, vanity metrics, comparative prose, public names. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 2 commits 2026-10-22 07:23:08 -04:00

eoc-route-learned: recommend the stage cheapest in expectation, not the likeliest ... 05ec7a0aa7

The crate says a learned router "predicts the cheapest stage that is likely
to satisfy the query". Three of its four routers recommended the stage
likeliest to succeed: MF and both bandits took an argmax over success, and
the bandits ignored the joule cost `observe` passed them. Neural succeeds on
everything and costs the most, so they learned to send every query there.
The synthetic LinUCB test pinned exactly that as the goal.

- router: `expected_joules` prices a skip the way LearnedCascade dispatches
  it (Kv or Graph falling through to Neural, Neural alone, Cache as the full
  walk), and `cheapest_in_expectation` picks the least. A broken probability
  prices a stage dearer, never cheaper. `observe` is the outcome of one
  stage that ran, with that stage's own joules.
- MF picks by that rule over its per-stage chances and learned costs.
- LinUCB and Thompson learn each stage's cost and pick by the rule over
  optimistic and sampled chances. Confidence is the mean belief, not the
  bound or sample used to explore. Older saved states import with the
  default costs.
- train_logreg labels each query with its cheapest successful stage; the
  classifier's online update no longer reinforces a correct stage costlier
  than the one it predicts.
- evaluate replays each recommendation over per-stage outcomes. It charged
  every row's own cost whatever the router chose and priced the walk at 4x
  that, so every router showed a lift of 3x its spend. A query it cannot
  price is counted unscored, not charged zero, and joules_per_correct is
  None when nothing was correct.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-route-learned: pin the classifier's self-reinforcement and MF's confidence ... 690e5c2d1d

Two mutants of the previous commit survived: a correct result on the
predicted stage itself no longer reinforcing it, and MF reporting a fixed
confidence. Each now has a test.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit 8c3de62968 into main 2026-11-01 09:38:36 -05:00

dcharlot referenced this pull request from a commit 2026-11-01 09:38:36 -05:00

Merge pull request 'eoc-route-learned: recommend the stage cheapest in expectation, not the likeliest' (#439) from fix/eoc-route-cheapest into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!439

No description provided.