Skip to content

Refactor get_trace and extend get_opt_trace_by_steps to MOO/constrained#4884

Closed
ltiao wants to merge 1 commit intofacebook:mainfrom
ltiao:export-D79581270
Closed

Refactor get_trace and extend get_opt_trace_by_steps to MOO/constrained#4884
ltiao wants to merge 1 commit intofacebook:mainfrom
ltiao:export-D79581270

Conversation

@ltiao
Copy link
Contributor

@ltiao ltiao commented Feb 10, 2026

Summary:
We have a method get_opt_trace_by_steps that was used extensively during our Ax 1.0 benchmarking campaign. It duplicates the basic logic of get_trace but differs in that it operates along (trial_index, MAP_KEY) pairs and respects ordering by timestamp (i.e. chronological order). However, it is limited to single-objective unconstrained problems, and our current needs (multi-objective and/or constrained) have outgrown it. We reconcile the two by extracting three core building blocks of get_trace:

  1. _pivot_data_with_feasibility: Pivots data to wide format with feasibility information and metric completeness checks.
  2. _compute_trace_values: Computes per-observation trace values (hypervolume for MOO, objective value for SOO), with cumulative best support.
  3. _aggregate_and_cumulate_trace: Aggregates values by groups and computes the cumulative best across groups.

These are implemented in a more general way that respects arbitrary groupings and orderings. We then refactor get_trace (and its helpers _prepare_data_for_trace and get_trace_by_arm_pull_from_data) to use these building blocks, and leverage them in get_opt_trace_by_steps to extend its support to multi-objective and constrained problems.

Additionally:

  • The timestamp-based sorting in get_opt_trace_by_steps is preserved, which is critical for correct cumulative hypervolume computation (without this, observations would be processed in (trial_index, arm_name, MAP_KEY) order instead of chronological order).
  • Tests are updated to replace NotImplementedError checks with actual MOO and constrained test cases that verify correctness of the new functionality.

Differential Revision: D79581270

@meta-cla meta-cla bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Feb 10, 2026
@meta-codesync
Copy link

meta-codesync bot commented Feb 10, 2026

@ltiao has exported this pull request. If you are a Meta employee, you can view the originating Diff in D79581270.

ltiao added a commit to ltiao/Ax that referenced this pull request Feb 10, 2026
…rained (facebook#4884)

Summary:

We have a method `get_opt_trace_by_steps` that was used extensively during our Ax 1.0 benchmarking campaign. It duplicates the basic logic of `get_trace` but differs in that it operates along `(trial_index, MAP_KEY)` pairs and respects ordering by timestamp (i.e. chronological order). However, it is limited to single-objective unconstrained problems, and our current needs (multi-objective and/or constrained) have outgrown it. We reconcile the two by extracting three core building blocks of `get_trace`:

1. `_pivot_data_with_feasibility`: Pivots data to wide format with feasibility information and metric completeness checks.
2. `_compute_trace_values`: Computes per-observation trace values (hypervolume for MOO, objective value for SOO), with cumulative best support.
3. `_aggregate_and_cumulate_trace`: Aggregates values by groups and computes the cumulative best across groups.

These are implemented in a more general way that respects arbitrary groupings and orderings. We then refactor `get_trace` (and its helpers `_prepare_data_for_trace` and `get_trace_by_arm_pull_from_data`) to use these building blocks, and leverage them in `get_opt_trace_by_steps` to extend its support to multi-objective and constrained problems.

Additionally:
- The timestamp-based sorting in `get_opt_trace_by_steps` is preserved, which is critical for correct cumulative hypervolume computation (without this, observations would be processed in `(trial_index, arm_name, MAP_KEY)` order instead of chronological order).
- Tests are updated to replace `NotImplementedError` checks with actual MOO and constrained test cases that verify correctness of the new functionality.

Differential Revision: D79581270
@codecov-commenter
Copy link

codecov-commenter commented Feb 10, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.84%. Comparing base (71c9895) to head (6868715).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4884   +/-   ##
=======================================
  Coverage   96.84%   96.84%           
=======================================
  Files         596      596           
  Lines       63480    63494   +14     
=======================================
+ Hits        61478    61492   +14     
  Misses       2002     2002           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ltiao ltiao force-pushed the export-D79581270 branch from e18d69f to 2820366 Compare March 2, 2026 22:59
ltiao added a commit to ltiao/Ax that referenced this pull request Mar 2, 2026
…rained (facebook#4884)

Summary:

We have a method `get_opt_trace_by_steps` that was used extensively during our Ax 1.0 benchmarking campaign. It duplicates the basic logic of `get_trace` but differs in that it operates along `(trial_index, MAP_KEY)` pairs and respects ordering by timestamp (i.e. chronological order). However, it is limited to single-objective unconstrained problems, and our current needs (multi-objective and/or constrained) have outgrown it. We reconcile the two by extracting three core building blocks of `get_trace`:

1. `_pivot_data_with_feasibility`: Pivots data to wide format with feasibility information and metric completeness checks.
2. `_compute_trace_values`: Computes per-observation trace values (hypervolume for MOO, objective value for SOO), with cumulative best support.
3. `_aggregate_and_cumulate_trace`: Aggregates values by groups and computes the cumulative best across groups.

These are implemented in a more general way that respects arbitrary groupings and orderings. We then refactor `get_trace` (and its helpers `_prepare_data_for_trace` and `get_trace_by_arm_pull_from_data`) to use these building blocks, and leverage them in `get_opt_trace_by_steps` to extend its support to multi-objective and constrained problems.

Additionally:
- The timestamp-based sorting in `get_opt_trace_by_steps` is preserved, which is critical for correct cumulative hypervolume computation (without this, observations would be processed in `(trial_index, arm_name, MAP_KEY)` order instead of chronological order).
- Tests are updated to replace `NotImplementedError` checks with actual MOO and constrained test cases that verify correctness of the new functionality.

Reviewed By: dme65

Differential Revision: D79581270
…rained (facebook#4884)

Summary:
Pull Request resolved: facebook#4884

We have a method `get_opt_trace_by_steps` that was used extensively during our Ax 1.0 benchmarking campaign. It duplicates the basic logic of `get_trace` but differs in that it operates along `(trial_index, MAP_KEY)` pairs and respects ordering by timestamp (i.e. chronological order). However, it is limited to single-objective unconstrained problems, and our current needs (multi-objective and/or constrained) have outgrown it. We reconcile the two by extracting three core building blocks of `get_trace`:

1. `_pivot_data_with_feasibility`: Pivots data to wide format with feasibility information and metric completeness checks.
2. `_compute_trace_values`: Computes per-observation trace values (hypervolume for MOO, objective value for SOO), with cumulative best support.
3. `_aggregate_and_cumulate_trace`: Aggregates values by groups and computes the cumulative best across groups.

These are implemented in a more general way that respects arbitrary groupings and orderings. We then refactor `get_trace` (and its helpers `_prepare_data_for_trace` and `get_trace_by_arm_pull_from_data`) to use these building blocks, and leverage them in `get_opt_trace_by_steps` to extend its support to multi-objective and constrained problems.

Additionally:
- The timestamp-based sorting in `get_opt_trace_by_steps` is preserved, which is critical for correct cumulative hypervolume computation (without this, observations would be processed in `(trial_index, arm_name, MAP_KEY)` order instead of chronological order).
- Tests are updated to replace `NotImplementedError` checks with actual MOO and constrained test cases that verify correctness of the new functionality.

Reviewed By: dme65

Differential Revision: D79581270
@meta-codesync
Copy link

meta-codesync bot commented Mar 3, 2026

This pull request has been merged in 4e345f4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported Merged meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants