perf: decide strategy ties in a second solve, not with a weight - #115
perf: decide strategy ties in a second solve, not with a weight#115andig wants to merge 7 commits into
Conversation
andig
changed the title
|
@ekkea this should fix the p99 latency |
9eefe86 to
79481c8
Compare
|
I've had Claude analyze all current long-running (i.e. p99 10s) requests:
Population-weighted: 47.5% slow on main → 20.4% with #115 |
ekkea
left a comment
ekkea
left a comment
There was a problem hiding this comment.
Here are my 2 cents w/o going into intensive testing: If this solves an urgent performance problem for us, I'm ok to apply the change, however, I think half of the changes here will have no predictable positive effect, the other half will have some performance effect but comes with the risk of not solving for proper strategy adherence.
| # the optimization problem | ||
| self.problem = None | ||
| # objective split, filled by _setup_target_function: real money and preferences | ||
| self.cost_objective = 0 |
There was a problem hiding this comment.
splitting the objectives into "real costs" and "preferences" may be a nice approach to improve the code readability but does not add anything new to the model and its structure. There may be arbitrary performance wins by slightly changing ratios between different objective contributions but this is not going to be deterministic over all cases.
There was a problem hiding this comment.
I'm not sure if it is smart to set OBJECTIVE_SCALE to a constant value. It may be a great choice for all cases but I do not see the proof for that.
|
Thanks, the objection about strategy loss is the load bearing one and it holds. I measured it rather than argue it, and it is worse than the PR body implies. The gap swallows the strategy terms on most casesFor every golden case, solved on this branch, the total weighted preference objective in currency units against
On eight of these the entire weighted preference objective is worth less than the absolute gap, several by one to two orders. The preference terms are one signed, so their total magnitude bounds their spread: on those cases every strategy variant sits inside the acceptance window and CBC returns whichever incumbent it happens to hold when the gap closes. The proof you asked for cannot be constructed there, it fails by arithmetic.
The peak and ramp cases survive because The second table in the description was already showing this and I read it too kindly: gap alone moves 15 to 28 MWh of charging away from the deliberate tie break on the flat cases. Weight 3 masks that to zero on five of six, but by branching order, not by construction. Related: the weight does nothing when prices are flat or negativeWhen On the objective splitYou are right that as it stands the split is routing for the weight and nothing more, and my own comment on it is stale, it still claims which no longer happens anywhere. That is a leftover from the two stage version this branch is named after. ProposalRestore the two stage solve, the split is already the infrastructure for it:
Strategy adherence becomes something we solve for instead of something that emerges from coefficient ratios. No weight to tune, no cliff table, and the 0.1679 percent that If the second solve turns out too expensive, the cheaper single solve version is the one already floated in the description: normalize Either way this wants a test asserting On
|
andig
changed the title
|
@ekkea pushed the rework, the description is rewritten around it. Short version of what your second comment turned into: you were right, and no weight could have fixed it. The two are now solved in sequence, cost first with the gap, preferences second under a constraint that keeps the money the first stage found. Measured over 335 unique requests from the slow request dump, at the production limit of 10 s:
The version you reviewed is behind main on strategy adherence on 112 of 335 production requests. That is your objection, on real traffic rather than on a constructed case. No golden case moves in either version, which is why neither of us saw it there. On your first comment: the split is no longer only readability, it is what the second stage optimizes over. On the third, Two things worth knowing beyond this PR, both found while benchmarking:
🤖 Generated with Claude Code |
|
The Short version: there is no proof for the constant, and it does not place what it claims to. With |
The cost neutral tie breakers, not the physics and not the horizon, drive the requests that exhaust OPTIMIZER_TIME_LIMIT. On a flat tariff the cost optimum is a plateau of equally priced schedules, and the strategy terms are far too small to decide between them, so the search walks the plateau instead. OPTIMIZER_GAP_ABS, one cent, stops the search once the remaining gap is worth less than a cent. OPTIMIZER_STRATEGY_WEIGHT, 3, lifts the strategy terms far enough above the cost terms to actually decide the tie. Neither is enough alone: the gap collapses the plateau walk but leaves the tie to wherever the search stopped, the weight decides the tie but still searches. To support the weight, the objective is assembled as cost_objective plus preference_objective, with a test asserting the split stays exhaustive.
OPTIMIZER_GAP_ABS bounds the whole objective, preferences included, and the preferences are orders below any useful gap: measured over the golden cases the entire weighted preference objective is worth less than one cent on 8 of 19, by up to 149x on 018-high-soc-initial. Inside one objective the gap therefore swallows the tie break, and OPTIMIZER_STRATEGY_WEIGHT cannot buy it back: 3 gives a factor 3 against a shortfall of 3 to 150, and 10 already costs 0.4 percent of real money on 023. So the two are solved in sequence instead. The first stage maximizes cost_objective and may stop OPTIMIZER_GAP_ABS short, which is where the speed comes from. The second maximizes preference_objective under a constraint that keeps the money the first stage found, which is where the strategies are decided. Strategy adherence becomes something the model solves for rather than something that emerges from the ratio between coefficients, so the weight and its cliff table are gone. OPTIMIZER_PREFERENCE_BUDGET replaces the weight and defaults to zero, the cost neutrality the strategies are documented to have. It is not a knob to reach for: the second stage is indifferent to money below the bound, so every unit granted is spent, measured exactly at budgets of 1 and 100. Both stages share one wall clock, so the second solve cannot double the response time, and anything short of Optimal there falls back to the first stage schedule whole. A timed out tie break costs preference, never money. 019-unexpected-charge-spikes is refreshed. Its money is unchanged to the last digit, only the reported objective_value moves, the s[0] artifact documented in AGENTS.md. test_app gains an atol of a hundredth of a cent, below which the reported value only carries the slack the cost bound needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
57c2ce9 to
b94b353
Compare
Three things the slow request dump brought out, none of them visible on the golden cases alone. CBC returns a strictly worse schedule and reports it as proven optimal when the first stage solution is handed to it as a MIP start, and it calls the model infeasible over a cost bound that same solution satisfies. Measured on request 4f28656db0ba: preference -0.806 warm against -0.610 cold, where cold matches a single joint solve to the last digit. No warm start, and the infeasible runs that motivated turning presolve off disappear with it, so presolve goes back to its default. Deciding the tie to proven optimality is its own hard problem, as expensive as the cost optimum on exactly the requests this is meant to help: solved cold to the end it puts the median back to 10.05 s over the dump. So the stage gets a quarter of the time limit rather than whatever is left, and what it does not finish is kept anyway. A tie break stopped by the clock still holds an incumbent, and the alternative is the first stage schedule, which is no tie break at all. The improvement and the money are checked here rather than read off the solver status. That check needs a tolerance of its own. CBC treats the cost bound like any other row and misses it by its feasibility tolerance, 1.2e-5 on 024-attenuate-demand-peaks, where rejecting over it threw away a tie break worth four times the peak. It is also what 024 needs: levelling a peak costs round trip efficiency, about 3.5e-5 there, so the strategies are not quite the cost neutral tie breakers the docs claim. 019-unexpected-charge-spikes is back to its stored expectation, no golden case moves. test_app keeps an atol of a hundredth of a cent, which is what the bound gives away. Over 335 unique requests from the slow request dump, at the production limit of 10 s: median 7.65 s to 2.99 s, requests over 10 s 149 to 52, and against main the strategies come out ahead on 120 requests and behind on 79, where the weighted variant of the previous commit is behind on 112 and ahead on 51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b94b353 to
1c53e73
Compare
|
Current status brings p99 significantly down, but the second solve has negative impact on p50/p95: 
|
The split costs a second CBC spawn on every request with a strategy set, and buys nothing on a request that was never degenerate: a model that proves the joint objective decided its own tie, in one solve, with no gap and no preference budget to give anything away. Measured on the stored cases, that is a median of 0.114 s against 0.052 s on main, a doubling paid by all traffic to help the one percent that reaches the time limit. So probe first. Solve the whole objective with no gap on a fifth of the time limit; if it proves optimality, stop there. Only a probe that runs out falls back to the money-then-preferences split, which is the case the split was built for. On the stored cases the median returns to main's 0.052 s, and 020-weird-charging-at-night gets faster than either, 0.53 s against 0.93 s, because the probe correctly routes it to the split. sol_status, not status: pulp reports LpStatusOptimal whenever CBC returns any feasible solution, including one it stopped on at the time limit. On a captured request a 2 s and a 30 s run both said Optimal, with objectives of -682466848 and 59714881. Reading status alone made the probe accept every truncated search and quietly cut the whole population off at the probe budget. Over 335 captured slow requests the probe is a loss, as it should be on a population that is entirely hard: median 4.89 s against 3.03 s for the unconditional split, with 214 of 335 falling through. Requests over the limit stay far below main, 51 against 149. Known gap: one request of the 335 comes back Not Solved, where the probe finds no feasible solution and leaves the cost stage too little clock to find one either. A shorter probe does not fix it. The probe's incumbent is kept when it has one, which covers the rest. OPTIMIZER_PROBE_SECONDS tunes the budget, 0 restores the unconditional split. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Pushed The problem it fixesMeasured over 14 buckets on the live service, against the build before it:
The tail did what it was meant to, and has not touched the limit in 14 buckets. The body got 70 percent slower, because the split spawns a second CBC on every request with a strategy configured, and the median request never needed it. What changedSplitting only pays on a degenerate plateau. A request that proves the joint objective never had one: it decided its own tie, in one solve, with no gap and no preference budget to give anything away. So the solve now probes first, on a fifth of the time limit, and only a probe that runs out falls back to money-then-preferences. On the stored cases, comparing
Back to main's median, and the total beats main because
A pulp trap worth knowing about regardless of this PRpulp reports Known gap, not fixed hereOne request of 335 captured slow ones comes back On the captured slow requests the probe is a net loss, which is what it should be on a population that is entirely hard: median 4.89 s against 3.03 s for the unconditional split, 214 of 335 falling through. Requests over the limit stay far below main, 51 against 149. The gain is on ordinary traffic, which that population by construction contains none of. |
2 participants
Fixes the main part of #114. Rebased onto #104, which changed what these settings are for.
#114 measured that the cost neutral tie breakers, not the physics and not the horizon, drive the requests that exhaust
OPTIMIZER_TIME_LIMIT. Their coefficients sit inside the solver's tolerance band, so they cannot prune anything. On a flat tariff, where the cost optimum is a plateau of equally priced schedules, the search walks that plateau instead of deciding the tie. 4196 of 6245 captured slow requests carry exactly one distinct import price.This PR changed shape after review. It was opened with an absolute MIP gap and a weight on the strategy terms. @ekkea objected that a gap which stops the search a cent short must also swallow the strategy terms, since they are orders below a cent, and asked for proof that this does not end in a loss of strategy. Measured, the objection holds and no weight fixes it, so the two are now solved in sequence instead. The old approach and the numbers behind it are in the review thread.
Why the weight could not carry it
The total weighted preference objective, in currency, against
gap_abs = 0.01, over the golden cases:On eight of nineteen the entire weighted preference objective is worth less than the gap. The preference terms are one signed, so their total magnitude bounds their spread: on those cases every strategy variant sits inside the acceptance window and the solver returns whichever incumbent it happens to hold. The weight buys a factor 3 against a shortfall of 3 to 150, and it cannot be raised, 10 already gives up 0.4 percent of real money on 023.
That is not a tuning problem. It is what an absolute gap on a single objective means.
What it does now
solve()runs two solves over the same model.cost_objectivewithOPTIMIZER_GAP_ABS, one cent. Real money only, and this is where the speed comes from.preference_objectiveunder a constraint that keeps the money the first stage found. This is where the strategies are decided.Strategy adherence becomes something the model solves for rather than something that emerges from the ratio between coefficients, so
OPTIMIZER_STRATEGY_WEIGHTand its cliff table are gone.OPTIMIZER_PREFERENCE_BUDGETtakes its place, defaulting to zero, the cost neutrality the strategies are documented to have. It is not a knob to reach for: the second stage is indifferent to money below the bound, so every unit granted is spent, measured exactly at budgets of 1 and of 100.Both stages share one wall clock. Deciding the tie to proven optimality turns out to be its own hard problem, as expensive as the cost optimum on exactly the requests this is meant to help, so the tie break gets a quarter of the limit rather than whatever is left of it, and what it does not finish is kept anyway: a tie break stopped by the clock still holds an incumbent, and the alternative is the first stage schedule, which is no tie break at all. The improvement and the money are verified in
_solve_preferencesrather than read off the solver status.Measured
335 unique requests from the slow request dump, every one of them a request that exhausted the limit in production, run at the production limit of 10 s:
The variant this PR was opened with is behind main on strategy adherence on 112 of 335 requests, which is the reviewer's objection showing up on production data rather than on a constructed case. The two stage version keeps almost all of the speed and turns that column around.
Money on the requests where both runs converged, in currency: 15 of 175 worse, worst 0.0071, median -2e-6. The rows where either run hit the limit are not evidence in either direction, neither result is converged there.
Nothing in the golden set moves, the stored expectations are untouched.
Three things the dump brought out, which the golden cases hid
PuLP's
warmStart=Truemakes CBC lie. Handed the first stage solution as a MIP start, CBC returns a strictly worse schedule and reports it as proven optimal, and it declares the model infeasible over a cost bound that same start satisfies. On request4f28656db0ba, preference -0.806 warm against -0.610 cold, where the cold value matches a single joint solve to the last digit. Every "CBC presolve is broken" symptom chased here turned out to be this, so presolve stays at its default.CBC misses the cost bound by its feasibility tolerance, like any other row, 1.2e-5 on a bound around 1263. A verification of that bound needs a tolerance of its own or it throws away good solutions,
COST_BOUND_TOLERANCE.The peak strategies are not cost neutral. Levelling burns round trip efficiency:
024-attenuate-demand-peaksneeds about 3.5e-5 currency to level, and refuses to level under a stricter bound. The claim in AGENTS.md that the strategies only decide between economically equal solutions holds for the tie breakers, not for the tier 3 peak and ramp terms. Worth a separate look.What this gives away
Per request, at the default budget of zero:
COST_BOUND_SLACKplusCOST_BOUND_TOLERANCE, so at most 1.1e-4 currency, a hundredth of a cent, and it is spent rather than merely allowed. Three orders below the cent the cost stage may already leave on the table. A third stage that maximizes cost again under a preference bound would reclaim it, at the price of a third solve, which did not look worth it.The third review comment, on
OBJECTIVE_SCALEbeing a constant, is not addressed here. It came in with #104 and stands on its own, so it gets its own PR.