Fix HintSVM query direction to select boundary-nearest samples#199
Open
CoolJosh0221 wants to merge 2 commits into
Open
Fix HintSVM query direction to select boundary-nearest samples#199CoolJosh0221 wants to merge 2 commits into
CoolJosh0221 wants to merge 2 commits into
Conversation
HintSVM.make_query selected the unlabeled point FARTHEST from the hinted
decision boundary (argmax|f(x)|), the opposite of what the algorithm intends.
Li, Ferng & Lin, "Active Learning with Hinted Support Vector Machine" (ACML
2012, Algorithm 1 Query step / §4.2) query the instance CLOSEST to the query
boundary, i.e. min|f(x)| — standard uncertainty sampling against the hinted
classifier. The old orientation also contradicted the _get_scores contract
("higher = more informative").
The bug dates to the original 2015 port (12305e6); PR ntucllab#197 (cac4c1e) only
repackaged make_query into _get_scores and, misleadingly, documented the
inverted orientation as intended.
Fix: negate the score (scores = -abs(decision_value)) so higher = closer to
the boundary = more informative; make_query keeps argmax and therefore now
selects min|f(x)|. This is an intentional query-behavior change and also
affects strategies that build on HintSVM (ActiveLearningByLearning as a
sub-strategy; DensityWeightedMeta composition).
Tests:
- test_hintsvm.py: add a skip guard so the module collects without the C
extension; assert argmax(scores) == make_query()'s pick (RNG re-seeded so
both calls see the same hint pool); assert direction — the on-boundary point
out-scores deep-in-cluster points and scores are <= 0 (p=0 isolates the
score direction from the hint-sampling mechanism). Real labels are -1/+1
because the C extension reserves label 0 as a hint marker.
- test_realdata.py: update the HintSVM golden query sequence, which encoded
the old (farthest-first) behavior, to the corrected deterministic sequence.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This fixes HintSVM so it queries the sample closest to the hinted decision boundary.
Previously,
HintSVM.make_query()usedargmax(abs(decision_value)), which selects the point farthest from the boundary. When_get_scores()was added, it preserved that behavior and returnedabs(decision_value). That made HintSVM scores go against the convention used by the other query strategies. The previous algorithm resulted in larger scores corresponding to points farther from the boundary, which are less instead of more informative.Specifically, this PR changes the score to
-abs(decision_value). With the existingargmax(scores)selection, HintSVM now picks the point with the smallest absolute decision value, which is the point closest to the boundary and represnts the most informative sample.This is an intentional behavior fix for HintSVM, so the expected result in the test is updated.
Tests
Added coverage that checks:
argmax(scores)returns the same sample asmake_query()