Skip to content

Commit e6e231a

Browse files
feat(planner): add basic scaffolding for optimization problem in planner (#407)
* feat(optimizer): scaffold OptimizerSolution type and translator Adds asap-planner-rs/src/optimizer/ with: - Aqe: atomic query expression wrapper (QueryRequirements + query strings + f_a) - QueryMethod: Neither / Merge{num_windows} / Subtract / Exact - AqeAssignment: maps an AQE to a deployed config + query method + cost estimate - OptimizerSolution: full planner output (deployed configs + assignments + cost) with all_exact() constructor for Phase 1 scaffolding - translator: translate() -> (StreamingConfig, InferenceConfig); Phase 1 stub with TODO for Phase 2 query config population Also adds design doc for the optimization formulation (#405). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(optimizer): add AQE extractor with GCD/min/sum frequency tracking Adds optimizer/aqe_extractor.rs: - Rqe struct (query_string + t_repeat_secs) - extract_aqes(): decomposes RQEs into deduplicated AQEs via recursive binary-op splitting and PromQL pattern matching - Computes three frequency values per AQE: - query_frequency_hz: Σ 1/T_r (total query load for MIP objective) - min_t_repeat_secs: min(T_r) (freshness bound on W ≤ min_t) - t_repeat_gcd_secs: GCD(T_r) (natural slide interval S for candidate gen) - Hand-rolled Euclidean GCD (num-integer not in workspace) - TODO noting duplication with build_query_requirements_promql in query-engine Updates Aqe struct in solution.rs to carry all three frequency fields. 5 unit tests covering single queries, binary splits, scalar arms, dedup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(optimizer): wire all-EXACT end-to-end pipeline (Phase 1) Adds optimizer/pipeline.rs with run_all_exact_pipeline(): ControllerConfig + PromQLSchema → config_to_rqes() (flatten QueryGroups) → extract_aqes() → OptimizerSolution::all_exact() → translate() → (StreamingConfig, InferenceConfig) No streaming configs deployed in this path — every AQE falls back to raw data. Validates end-to-end plumbing before Phase 2 sketch selection. 2 tests: empty streaming config for all-EXACT, group flattening. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(optimizer): add sketch algebraic properties and candidate config generation Adds sketch_properties.rs (mergeable/subtractable/subpopulation_aware per AggregationType) and candidate_gen.rs (enumerates candidate streaming configs per AQE across agg type, param grid, window size, and ingest type). Sliding candidates sweep slide interval S as well as the forced W=range_a, trading ingest cost for freshness. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(optimizer): add analytical cost model for ingest and query cost rates Adds cost_model.rs implementing IngestCost(g) and QueryCost(a,g) from the design doc's formulas, using stub AtomicCosts (real values come from sketch-bench in Phase 3). Includes an EXACT fallback query cost so the greedy/MIP objective has something non-zero to compare sketches against. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(optimizer): wire greedy per-AQE assignment end-to-end (Phase 2) Adds greedy.rs (picks each AQE's independently-cheapest candidate via the cost model), run_greedy_pipeline() in pipeline.rs, and populates InferenceConfig.query_configs in translator.rs for non-Exact assignments. Rebalances CostWeights::default() so memory and CPU costs are on comparable scales (RAM-held-over-time is ~1e6x cheaper per unit than CPU-time in real cloud pricing) — the prior equal weighting made EXACT always win regardless of workload. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(optimizer): add standalone CLI for offline testing of the greedy pipeline Adds asap-optimizer-cli (src/bin/optimizer_cli.rs), a second binary that runs run_greedy_pipeline against a workload YAML and prints the resulting deployed configs and query configs. Lets the optimizer be exercised against real configs without touching Controller::generate(), which still goes through the existing hardcoded generator::generate_plan() path unconditionally. Also adds a doc comment on generate_plan() flagging that the optimizer module exists as a not-yet-wired-in alternative, for anyone reading that file without other context. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fixed some CR comments * fixed Dockerfile, clarified code comment --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent fefca94 commit e6e231a

16 files changed

Lines changed: 2159 additions & 2 deletions

.design_docs/sketch-config-optimization-formulation.md

Lines changed: 493 additions & 0 deletions
Large diffs are not rendered by default.

asap-planner-rs/Cargo.toml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ path = "src/lib.rs"
1111
name = "asap-planner"
1212
path = "src/main.rs"
1313

14+
[[bin]]
15+
name = "asap-optimizer-cli"
16+
path = "src/bin/optimizer_cli.rs"
17+
1418
[dependencies]
1519
asap_types.workspace = true
1620
promql_utilities.workspace = true

asap-planner-rs/Dockerfile

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,8 @@ COPY asap-planner-rs/Cargo.toml ./asap-planner-rs/
1919
# Create dummy source files so Cargo can resolve all workspace members
2020
RUN mkdir -p asap-query-engine/src && echo "fn main() {}" > asap-query-engine/src/main.rs && \
2121
mkdir -p asap-query-engine/benches && echo "fn main() {}" > asap-query-engine/benches/simple_store_bench.rs && \
22-
mkdir -p asap-planner-rs/src && echo "fn main() {}" > asap-planner-rs/src/main.rs && \
22+
mkdir -p asap-planner-rs/src/bin && echo "fn main() {}" > asap-planner-rs/src/main.rs && \
23+
echo "fn main() {}" > asap-planner-rs/src/bin/optimizer_cli.rs && \
2324
echo "pub fn placeholder() {}" >> asap-planner-rs/src/lib.rs
2425

2526
# Build dependencies (this layer will be cached)
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
//! Offline runner for the optimization-based sketch/config selector.
2+
//!
3+
//! Standalone: not wired into `asap-planner`/`Controller::generate()` yet. Lets
4+
//! you exercise `run_greedy_pipeline` against real workload configs while the
5+
//! optimizer module is still under development (Phase 2 of issue #405).
6+
7+
use std::path::PathBuf;
8+
9+
use asap_planner::optimizer::run_greedy_pipeline;
10+
use asap_planner::ControllerConfig;
11+
use clap::Parser;
12+
13+
#[derive(Parser, Debug)]
14+
#[command(
15+
name = "asap-optimizer-cli",
16+
about = "Offline runner for the optimization-based sketch/config selector (not wired into asap-planner yet)"
17+
)]
18+
struct Args {
19+
/// Path to a YAML workload config (same format as `asap-planner --input_config`).
20+
#[arg(long = "input_config")]
21+
input_config: PathBuf,
22+
23+
#[arg(long = "prometheus_scrape_interval")]
24+
prometheus_scrape_interval: u64,
25+
26+
/// Placeholder arrival rate (items/sec) applied uniformly to every candidate's
27+
/// IngestCost. Real per-config rates aren't wired up yet — see the open TODOs
28+
/// in .design_docs/optimizer-v1-implementation-plan.md.
29+
#[arg(long = "rho", default_value = "1.0", value_parser = parse_positive_finite)]
30+
rho: f64,
31+
32+
#[arg(short, long, action = clap::ArgAction::Count)]
33+
verbose: u8,
34+
}
35+
36+
fn parse_positive_finite(s: &str) -> Result<f64, String> {
37+
let v: f64 = s.parse().map_err(|_| format!("not a valid number: {s}"))?;
38+
if !v.is_finite() || v <= 0.0 {
39+
return Err(format!("--rho must be a positive finite number, got {v}"));
40+
}
41+
Ok(v)
42+
}
43+
44+
fn main() -> anyhow::Result<()> {
45+
let args = Args::parse();
46+
47+
tracing_subscriber::fmt()
48+
.with_max_level(if args.verbose > 0 {
49+
tracing::Level::DEBUG
50+
} else {
51+
tracing::Level::INFO
52+
})
53+
.init();
54+
55+
let yaml_str = std::fs::read_to_string(&args.input_config)?;
56+
let config: ControllerConfig = serde_yaml::from_str(&yaml_str)?;
57+
let schema = config.schema_from_hints();
58+
59+
let (streaming, inference) =
60+
run_greedy_pipeline(&config, &schema, args.prometheus_scrape_interval, args.rho);
61+
62+
let deployed = streaming.get_all_aggregation_configs();
63+
println!("=== Deployed streaming configs: {} ===", deployed.len());
64+
for (id, cfg) in deployed {
65+
println!(
66+
" [{id}] {} sub_type={:?} window={}s slide={}s type={:?} metric={} params={:?}",
67+
cfg.aggregation_type,
68+
cfg.aggregation_sub_type,
69+
cfg.window_size,
70+
cfg.slide_interval,
71+
cfg.window_type,
72+
cfg.metric,
73+
cfg.parameters,
74+
);
75+
}
76+
77+
println!("\n=== Query configs: {} ===", inference.query_configs.len());
78+
for qc in &inference.query_configs {
79+
println!(" \"{}\" -> {:?}", qc.query, qc.aggregations);
80+
}
81+
82+
Ok(())
83+
}

asap-planner-rs/src/lib.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ pub mod config;
33
pub mod elastic_dsl;
44
pub mod error;
55
pub mod generator;
6+
pub mod optimizer;
67
pub mod planner;
78
pub mod planner_output;
89
pub mod prometheus_client;

0 commit comments

Comments
 (0)