Fix fairness metrics NaN error - #1199
Open
lehendo wants to merge 1 commit into
Open
Conversation
disparate_impact and statistical_parity_difference computed a group's favorable-outcome rate as np.sum(...) / len(...). For an empty group this is a numpy 0/0, which gives numpy.float64(nan) rather than raising -- and since nan == 0 is always False, disparate_impact's existing "if p_fav_unpr == 0" guard never caught it, silently returning nan. The guard also only ever checked the unprotected group; the protected group being empty was never checked at all and produced the same silent nan. statistical_parity_difference had no guard whatsoever. Adds a shared _favorable_outcome_rate() helper that validates group non-emptiness before computing the rate, raising ValueError for a genuinely empty group (0 samples) -- distinct from a non-empty group whose rate happens to be exactly 0, which remains a legitimate value (preserving allow_zero_division's existing behavior in disparate_impact, and needing no special handling in statistical_parity_difference, where a 0 rate is fine for a subtraction). Adds tests/core/test_fairness.py (10 tests, no prior test file existed for this module), covering all four previously-silent-NaN cases, normal non-empty-group operation, and allow_zero_division's intended use case. Verified end-to-end on real hardware, including the public fairness_metrics_fn wrapper.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
0/0 produces nan silently (the intended ValueError guard checks == 0, which nan never equals), poisoning any downstream average across folds/seeds