Skip to content

Fix two-sided outcome-constraint metrics showing a single direction - #5255

Open
tg9963 wants to merge 1 commit into
facebook:mainfrom
tg9963:export-D113506683
Open

Fix two-sided outcome-constraint metrics showing a single direction#5255
tg9963 wants to merge 1 commit into
facebook:mainfrom
tg9963:export-D113506683

Conversation

@tg9963

@tg9963 tg9963 commented Jul 24, 2026

Copy link
Copy Markdown

Summary:
A metric constrained on both sides (a two-sided band with both a >= and a
<= outcome constraint, e.g. m >= 0.9 and m <= 1.1) was rendered on the Ax
platform with a single "increase" direction, so the upper bound appeared to be
lost. The constraint op and bound were always stored correctly; the bug was
in the metric's derived lower_is_better.

When building the experiment, Ax derived each metric's lower_is_better from
only the FIRST outcome constraint referencing it (if metric_name not in map).
For a two-sided band the first constraint is >= (GEQ -> lower_is_better=False
-> "Increase"), and the second bound was ignored. A metric bounded on both sides
has no single preferred direction.

  • Extract a shared InstantiationBase._build_lower_is_better_map helper. A
    metric with conflicting constraint ops (both GEQ and LEQ) now maps to
    lower_is_better=None instead of the arbitrary first constraint's direction.
  • Use the helper in both call sites that previously duplicated the buggy block:
    make_experiment (create path) and AxClient.set_optimization_config
    (update path). Drop the now-unused ComparisonOp import in ax_client.

Objective directions and single-sided constraints are unchanged. Objective and
constraint metric names are always disjoint (an objective metric cannot be
constrained), so the two groups never conflict.

Differential Revision: D113506683

@meta-cla meta-cla Bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Jul 24, 2026
@meta-codesync

meta-codesync Bot commented Jul 24, 2026

Copy link
Copy Markdown

@tg9963 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D113506683.

Summary:
A metric constrained on both sides (a two-sided band with both a `>=` and a
`<=` outcome constraint, e.g. `m >= 0.9` and `m <= 1.1`) was rendered on the Ax
platform with a single "increase" direction, so the upper bound appeared to be
lost. The constraint `op` and `bound` were always stored correctly; the bug was
in the metric's derived `lower_is_better`.

When building the experiment, Ax derived each metric's `lower_is_better` from
only the FIRST outcome constraint referencing it (`if metric_name not in map`).
For a two-sided band the first constraint is `>=` (GEQ -> lower_is_better=False
-> "Increase"), and the second bound was ignored. A metric bounded on both sides
has no single preferred direction.

- Extract a shared `InstantiationBase._build_lower_is_better_map` helper. A
  metric with conflicting constraint ops (both GEQ and LEQ) now maps to
  `lower_is_better=None` instead of the arbitrary first constraint's direction.
- Use the helper in both call sites that previously duplicated the buggy block:
  `make_experiment` (create path) and `AxClient.set_optimization_config`
  (update path). Drop the now-unused `ComparisonOp` import in ax_client.

Objective directions and single-sided constraints are unchanged. Objective and
constraint metric names are always disjoint (an objective metric cannot be
constrained), so the two groups never conflict.

Differential Revision: D113506683
@tg9963
tg9963 force-pushed the export-D113506683 branch from 055d610 to fda69e1 Compare July 24, 2026 05:22
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.57%. Comparing base (f5c0218) to head (fda69e1).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5255      +/-   ##
==========================================
- Coverage   96.58%   96.57%   -0.01%     
==========================================
  Files         621      621              
  Lines       70663    70682      +19     
==========================================
+ Hits        68248    68262      +14     
- Misses       2415     2420       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed Do not delete this pull request or issue due to inactivity. meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants