fix: intrinsic tests and add some safeguards for future adapters changes #1078
fix: intrinsic tests and add some safeguards for future adapters changes #1078jakelorocco wants to merge 3 commits into
Conversation
…nsics Signed-off-by: Jake LoRocco <jake.lorocco@ibm.com> Assisted-by: CLAUDE:OPUS
Signed-off-by: Jake LoRocco <jake.lorocco@ibm.com> Assisted-by: CLAUDE:OPUS
|
The PR description has been updated. Please fill out the template for your PR to be reviewed. |
…ests; small nits Signed-off-by: Jake LoRocco <jake.lorocco@ibm.com> Assisted-by: CLAUDE:OPUS
jakelorocco
force-pushed
the
test/fix-intrinsic-tests
branch
from
84f4af5 to
33401f4
Compare
|
@frreiss, this PR will require your review when you get a chance. Thank you. I think the main part is to make sure I added support for req-check and certainty correctly. I think I have previously mentioned the versioning checks to you. |
jakelorocco
changed the title
jakelorocco
added
bug
planetf1
left a comment
planetf1
left a comment
There was a problem hiding this comment.
Two tests need @pytest.mark.integration; stale last_validated_commit SHAs on the four new entries need verifying before merge.
| int(os.environ.get("CICD", 0)) == 1, | ||
| reason="Don't cause CICD pipelines to fail due to adapter version changes alone.", | ||
| ) | ||
| @pytest.mark.huggingface |
There was a problem hiding this comment.
huggingface is reserved for GPU/transformers tests and isn't in conftest's _NON_UNIT, so without a tier marker this test auto-becomes unit despite making live HF Hub API calls. integration is the right tier — real external boundary, no GPU needed.
| @pytest.mark.huggingface | |
| @pytest.mark.integration |
| the expected output | ||
| """ | ||
| cfg = yaml_json_combo_no_alora | ||
| _xfail_if_drifted(cfg) |
There was a problem hiding this comment.
_xfail_if_drifted makes a live HF Hub API call on first use per session. test_canned_input has no tier marker so auto-becomes unit. Add @pytest.mark.integration to the function.
|
|
||
| # Same cases as test_canned_input | ||
| cfg = yaml_json_combo_with_lora_model | ||
| _xfail_if_drifted(cfg) |
There was a problem hiding this comment.
Same as test_canned_input — _xfail_if_drifted makes a live HF Hub call but this test auto-becomes unit. Add @pytest.mark.integration.
| inputs_file=_INPUT_JSON_DIR / "requirement_check.json", | ||
| task="requirement-check", | ||
| repo_id="ibm-granite/granitelib-core-r1.0", | ||
| last_validated_commit="6b9a42d5e23364b3aca0ae334fbbea57c510623a", |
There was a problem hiding this comment.
The recorded SHA 6b9a42d5 is already behind current main on granitelib-core-r1.0 — verified against live HF Hub:
requirement-check/granite-4.1-3b/{lora,alora}→d0a2a96auncertainty/granite-4.1-3b/{lora,alora}→1e568b00
All four entries will immediately xfail on first run. Were the canned outputs generated against the current adapter? If so, update last_validated_commit to the current SHAs.
|
|
||
| # Explicitly don't check drift here. Ollama models don't have their own yaml combo | ||
| # that we can track. | ||
| # _xfail_if_drifted(cfg) |
There was a problem hiding this comment.
NIT: the prose comment above already explains why drift isn't checked here — dead code, can be removed.
2 participants
Misc PR
Type of PR
Description
Link to Issue: Fixes update intrinsic tests #1029
Adds back the uncertainty and requirement-check tests
Adds last_validated_commit for adapters we test for so that we can catch future version changes in our nightlies
Testing
Attribution