feat(metric): add Symmetric Relevance Gain (SRG) - #374
Open
adrhill wants to merge 7 commits into
Open
Conversation
Implements the SRG faithfulness metric (Blücher et al., TMLR 2024): the area between the LIF and MIF pixel-flipping curves computed from a shared feature ordering. The random-ordering baseline cancels in the difference, making rankings robust to the occlusion strategy. - `quantus.SymmetricRelevanceGain` evaluates both curves with one concatenated forward pass per occlusion step, plus a torch-resident fast path that keeps perturbed inputs on-device - `n_steps` parameter for coarse stepping; the baseline imputer is computed once per batch from the unperturbed input - registered in `AVAILABLE_METRICS`, docs page added, fixture-based tests and invariant tests (sign-flip antisymmetry, shared curve endpoints, torch path vs. numpy path equivalence) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-ups for the SRG metric: - declare TensorFlow support in `model_applicability`: the numpy path only uses the framework-agnostic `ModelInterface` API - make the constant imputer the single perturbation contract: `perturb_func` is applied once per batch to the unperturbed input and all occlusion steps copy from that snapshot, so passing the default function explicitly now matches `perturb_func=None` and the torch fast path works with any perturbation function (regression test added) - replace `n_steps` with the conventional `features_in_step` knob and assert against `x_batch.shape[2:]` like `PixelFlipping` - drop the `last_mif_curves`/`last_lif_curves` accessors and per-call curve storage so `get_params()` only reports configuration - add the `warn_perturbation_caused_no_change` check, document the "random" baseline option, drop the redundant `int` from the `perturb_baseline` annotation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
For inputs normalized to zero channel mean (standard ImageNet preprocessing), imputing zeros exactly reproduces the paper's channel-wise data set mean imputer, whereas the previous default `perturb_baseline="mean"` (per-sample mean over flattened features) only approximated it. Documented the normalization assumption and the alternatives for unnormalized inputs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Drop "random" from the documented `perturb_baseline` options: it is deprecated and `get_baseline_value` rejects it with a `ValueError`. - Replace the "array of channel means" advice: with the all-indices call pattern of `batch_baseline_replacement_by_indices`, an `np.ndarray` baseline must be 0-dimensional; also clarify that "mean" is the per-sample mean over all features, not the paper's channel-wise mean. - Make the `abs` guidance method-conditional in the docstring and the parameterisation warning: signed ranking presumes the attribution's sign encodes evidence for/against the class (LRP, Shapley, IG); for sensitivity maps whose sign is a direction in color space (raw gradients), `abs=True` or channel aggregation is appropriate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The `warn_parameterisation` template hardcodes "is likely to be
sensitive to the choice of {sensitive_params}", so listing
`perturb_baseline` and `features_in_step` followed by "(SRG rankings
are designed to be robust to both)" rendered as a sentence that negated
itself. Headline `abs` as the genuinely sensitive parameter instead,
and state the robustness to occlusion-strategy choices — SRG's main
selling point — in its own sentence, qualified by the paper's finding
that absolute scores still vary across setups.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
annahedstroem
marked this pull request as ready for review
June 18, 2026 08:12
Member
|
Hi @adrhill is this ready for review? Thank you so much! |
There was a problem hiding this comment.
Pull request overview
Adds a new faithfulness metric, Symmetric Relevance Gain (SRG), to Quantus, along with registry wiring, tests, and API documentation so it can be used like existing pixel-flipping-based metrics.
Changes:
- Introduces
quantus.metrics.faithfulness.SymmetricRelevanceGainimplementing SRG with both a torch fast-path and a numpy fallback. - Registers SRG in the faithfulness package exports and
AVAILABLE_METRICS. - Adds SRG-focused tests and Sphinx API docs for the new module.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
quantus/metrics/faithfulness/symmetric_relevance_gain.py |
New SRG metric implementation (including torch fast-path) and docstring/API surface. |
quantus/metrics/faithfulness/__init__.py |
Exports SymmetricRelevanceGain from the faithfulness metrics package. |
quantus/helpers/constants.py |
Registers SRG in AVAILABLE_METRICS under Faithfulness. |
tests/metrics/test_faithfulness_metrics.py |
Adds SRG test coverage + minor assertion formatting changes. |
docs/source/docs_api/quantus.metrics.faithfulness.symmetric_relevance_gain.rst |
Adds generated API docs page for the new module. |
docs/source/docs_api/quantus.metrics.faithfulness.rst |
Includes SRG in the faithfulness docs toctree. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Author
I haven't found the time to check my local results yet, but I'll ping you once I've confirmed them! |
Member
|
Hi @adrhill, can you please have a look at the failing test 3.10? So I can go ahead with a merge! |
Copilot flagged that the SRG formula read like an unnormalised AUC difference, which would contradict the documented [-1, 1] score range. Pin the convention: curves are traced over the occluded feature fraction on [0, 1], so the area between them does not scale with the number of occlusion steps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
Looks to me like the CI failures are unrelated. They seem to come from |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
SRG = AUC(LIF) − AUC(MIF). The random-ordering baseline cancels in the difference, which makes attribution rankings largely insensitive to the occlusion strategy (baseline value, step size).Implemented changes
quantus.SymmetricRelevanceGaininquantus/metrics/faithfulness/symmetric_relevance_gain.py, registered inAVAILABLE_METRICSand the faithfulness__init__.pyfeatures_in_step,perturb_func,perturb_baseline, ...); supports both PyTorch and TensorFlow modelsperturb_funcis applied once per batch to the unperturbed input and every occlusion step copies values from that snapshot, so stochastic baselines are drawn once and results are independent of how the default perturbation function is passedperturb_baseline=0.0reproduces the paper's channel-wise data set mean imputer for inputs normalized to zero channel mean (e.g. standard ImageNet preprocessing); the docstring documents this assumption and the alternatives for unnormalized inputsabsdocstring and parameterisation warning give method-conditional guidance: keepabs=Falsewhere the attribution's sign encodes evidence for/against the class (e.g. LRP, Shapley, IG); useabs=Truefor sensitivity maps whose sign reflects a direction in color space (e.g. raw gradients)tests/metrics/test_faithfulness_metrics.py(sign-flip antisymmetry, shared curve endpoints, torch-path vs. numpy-path equivalence, explicit defaultperturb_funcequalsperturb_func=None)docs/source/docs_api/CC @bluecher31 (paper author)
🤖 Generated with Claude Code