Refactor get_trace and extend get_opt_trace_by_steps to MOO/constrained#4884
Closed
ltiao wants to merge 1 commit intofacebook:mainfrom
Closed
Refactor get_trace and extend get_opt_trace_by_steps to MOO/constrained#4884ltiao wants to merge 1 commit intofacebook:mainfrom
get_trace and extend get_opt_trace_by_steps to MOO/constrained#4884ltiao wants to merge 1 commit intofacebook:mainfrom
Conversation
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Feb 10, 2026
…rained (facebook#4884) Summary: We have a method `get_opt_trace_by_steps` that was used extensively during our Ax 1.0 benchmarking campaign. It duplicates the basic logic of `get_trace` but differs in that it operates along `(trial_index, MAP_KEY)` pairs and respects ordering by timestamp (i.e. chronological order). However, it is limited to single-objective unconstrained problems, and our current needs (multi-objective and/or constrained) have outgrown it. We reconcile the two by extracting three core building blocks of `get_trace`: 1. `_pivot_data_with_feasibility`: Pivots data to wide format with feasibility information and metric completeness checks. 2. `_compute_trace_values`: Computes per-observation trace values (hypervolume for MOO, objective value for SOO), with cumulative best support. 3. `_aggregate_and_cumulate_trace`: Aggregates values by groups and computes the cumulative best across groups. These are implemented in a more general way that respects arbitrary groupings and orderings. We then refactor `get_trace` (and its helpers `_prepare_data_for_trace` and `get_trace_by_arm_pull_from_data`) to use these building blocks, and leverage them in `get_opt_trace_by_steps` to extend its support to multi-objective and constrained problems. Additionally: - The timestamp-based sorting in `get_opt_trace_by_steps` is preserved, which is critical for correct cumulative hypervolume computation (without this, observations would be processed in `(trial_index, arm_name, MAP_KEY)` order instead of chronological order). - Tests are updated to replace `NotImplementedError` checks with actual MOO and constrained test cases that verify correctness of the new functionality. Differential Revision: D79581270
6f17ca0 to
e18d69f
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4884 +/- ##
=======================================
Coverage 96.84% 96.84%
=======================================
Files 596 596
Lines 63480 63494 +14
=======================================
+ Hits 61478 61492 +14
Misses 2002 2002 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Mar 2, 2026
…rained (facebook#4884) Summary: We have a method `get_opt_trace_by_steps` that was used extensively during our Ax 1.0 benchmarking campaign. It duplicates the basic logic of `get_trace` but differs in that it operates along `(trial_index, MAP_KEY)` pairs and respects ordering by timestamp (i.e. chronological order). However, it is limited to single-objective unconstrained problems, and our current needs (multi-objective and/or constrained) have outgrown it. We reconcile the two by extracting three core building blocks of `get_trace`: 1. `_pivot_data_with_feasibility`: Pivots data to wide format with feasibility information and metric completeness checks. 2. `_compute_trace_values`: Computes per-observation trace values (hypervolume for MOO, objective value for SOO), with cumulative best support. 3. `_aggregate_and_cumulate_trace`: Aggregates values by groups and computes the cumulative best across groups. These are implemented in a more general way that respects arbitrary groupings and orderings. We then refactor `get_trace` (and its helpers `_prepare_data_for_trace` and `get_trace_by_arm_pull_from_data`) to use these building blocks, and leverage them in `get_opt_trace_by_steps` to extend its support to multi-objective and constrained problems. Additionally: - The timestamp-based sorting in `get_opt_trace_by_steps` is preserved, which is critical for correct cumulative hypervolume computation (without this, observations would be processed in `(trial_index, arm_name, MAP_KEY)` order instead of chronological order). - Tests are updated to replace `NotImplementedError` checks with actual MOO and constrained test cases that verify correctness of the new functionality. Reviewed By: dme65 Differential Revision: D79581270
…rained (facebook#4884) Summary: Pull Request resolved: facebook#4884 We have a method `get_opt_trace_by_steps` that was used extensively during our Ax 1.0 benchmarking campaign. It duplicates the basic logic of `get_trace` but differs in that it operates along `(trial_index, MAP_KEY)` pairs and respects ordering by timestamp (i.e. chronological order). However, it is limited to single-objective unconstrained problems, and our current needs (multi-objective and/or constrained) have outgrown it. We reconcile the two by extracting three core building blocks of `get_trace`: 1. `_pivot_data_with_feasibility`: Pivots data to wide format with feasibility information and metric completeness checks. 2. `_compute_trace_values`: Computes per-observation trace values (hypervolume for MOO, objective value for SOO), with cumulative best support. 3. `_aggregate_and_cumulate_trace`: Aggregates values by groups and computes the cumulative best across groups. These are implemented in a more general way that respects arbitrary groupings and orderings. We then refactor `get_trace` (and its helpers `_prepare_data_for_trace` and `get_trace_by_arm_pull_from_data`) to use these building blocks, and leverage them in `get_opt_trace_by_steps` to extend its support to multi-objective and constrained problems. Additionally: - The timestamp-based sorting in `get_opt_trace_by_steps` is preserved, which is critical for correct cumulative hypervolume computation (without this, observations would be processed in `(trial_index, arm_name, MAP_KEY)` order instead of chronological order). - Tests are updated to replace `NotImplementedError` checks with actual MOO and constrained test cases that verify correctness of the new functionality. Reviewed By: dme65 Differential Revision: D79581270
|
This pull request has been merged in 4e345f4. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
We have a method
get_opt_trace_by_stepsthat was used extensively during our Ax 1.0 benchmarking campaign. It duplicates the basic logic ofget_tracebut differs in that it operates along(trial_index, MAP_KEY)pairs and respects ordering by timestamp (i.e. chronological order). However, it is limited to single-objective unconstrained problems, and our current needs (multi-objective and/or constrained) have outgrown it. We reconcile the two by extracting three core building blocks ofget_trace:_pivot_data_with_feasibility: Pivots data to wide format with feasibility information and metric completeness checks._compute_trace_values: Computes per-observation trace values (hypervolume for MOO, objective value for SOO), with cumulative best support._aggregate_and_cumulate_trace: Aggregates values by groups and computes the cumulative best across groups.These are implemented in a more general way that respects arbitrary groupings and orderings. We then refactor
get_trace(and its helpers_prepare_data_for_traceandget_trace_by_arm_pull_from_data) to use these building blocks, and leverage them inget_opt_trace_by_stepsto extend its support to multi-objective and constrained problems.Additionally:
get_opt_trace_by_stepsis preserved, which is critical for correct cumulative hypervolume computation (without this, observations would be processed in(trial_index, arm_name, MAP_KEY)order instead of chronological order).NotImplementedErrorchecks with actual MOO and constrained test cases that verify correctness of the new functionality.Differential Revision: D79581270