Skip to content

phd: be picky about allowed test skips#1089

Open
iximeow wants to merge 2 commits intomasterfrom
phd-predicate-thingy
Open

phd: be picky about allowed test skips#1089
iximeow wants to merge 2 commits intomasterfrom
phd-predicate-thingy

Conversation

@iximeow
Copy link
Member

@iximeow iximeow commented Mar 25, 2026

the two commits here are distinct but related (in particular, second builds on the first), so I plan to do the rare "rebase and merge" on this one. because of that, the useful PR description is really just those two messages:


phd: getting guest OS kinds should not involve locks

Artifact types are known ahead of time when the PHD artifact manifest is
parsed. These won't change throughout a test, really only the path that
those artifacts are locally cached at is the only element of an artifact
that may change - and even then, only if the artifact is not already
local in manifest or cache.

So, move this mutable component of StoredArtifact behind a mutex, and
let the rest of the artifact's description be accessible any time.

The corresponding shuffling means that accessing a guest OS kind no
longer implies downloading the referenced aritfact, an edge case which
no one has noticed and no one will miss probably!


phd: be pedantic about what test skips are OK

This gives us some additional confidence that all tests that run either:

  • pass
  • skip, but were expected to given the test hardware/guest configuration

Importantly, not all PHD tests pass on all hardware/guest OS
combinations, and not all PHD tests even run on all hardware/guest
combinations. As a bit of a fluke, Intel happens to run one additional
test that is skipped on AMD processors, even.

The strategy is to run "skipped" tests and verify they skip because if
guest artifacts are changed unexpectedly we may need to better
understand how the artifact differs from PHD expectations. Or if
Propolis/PHD implementation changes such that the skip is no longer
taken, failing the test instead of silently passing will force us to
notice if that change is wanted, if the test should pass, etc.

As an example, the test Alpine image is currently expected to be an
alpine-virt on an ISO 9660 (read-only) filesystem. Since this means
the EFI system partition is also read-only,
guest_can_adjust_boot_order is not a meaningful test (for now). We
run this test anyway and expect a skip, because if it passed then
either:

  • the guest Alpine is on a writable root FS; are we missing test
    coverage involving read-only guest images?
  • we've moved EFI NvVars to out-of-guest storage, and efivarfs writes
    succeed.

In either case, the corresponding changes should have some consideration
for how they relate Propolis and guest operations.


I've only really given this a run with the default Alpine guest for the time being. from the phd_skip!() I found I'm pretty sure the predicates generalize to other guests/configs/etc and any other stuff we can suss out if/when we find it.

Artifact types are known ahead of time when the PHD artifact manifest is
parsed. These won't change throughout a test, really only the path that
those artifacts are locally cached at is the only element of an artifact
that *may* change - and even then, only if the artifact is not already
local in manifest or cache.

So, move this mutable component of StoredArtifact behind a mutex, and
let the rest of the artifact's description be accessible any time.

The corresponding shuffling means that accessing a guest OS kind no
longer implies downloading the referenced aritfact, an edge case which
no one has noticed and no one will miss probably!
@iximeow iximeow added the testing Related to testing and/or the PHD test framework. label Mar 25, 2026
Comment on lines +353 to +354
let entry = self.get_artifact(artifact_name)?;
match entry.description.kind {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is where i momentarily had a tokio::task::block_in_place(|| let rt = ...; rt.block_on(entry.lock())) that was so terrible it would have caused @hawkw to close the tab, but i shoved some mutexes around and now it's just normal code

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spiffy!

@iximeow iximeow mentioned this pull request Mar 25, 2026
@iximeow iximeow force-pushed the phd-predicate-thingy branch from a87bbc5 to 49d2e1b Compare March 25, 2026 18:37
This gives us some additional confidence that all tests that run either:
* pass
* skip, but were expected to given the test hardware/guest configuration

Importantly, not all PHD tests pass on all hardware/guest OS
combinations, and not all PHD tests even *run* on all hardware/guest
combinations. As a bit of a fluke, *Intel* happens to run one additional
test that is skipped on AMD processors, even.

The strategy is to run "skipped" tests and verify they skip because if
guest artifacts are changed unexpectedly we may need to better
understand how the artifact differs from PHD expectations. Or if
Propolis/PHD implementation changes such that the skip is no longer
taken, failing the test instead of silently passing will force us to
notice if that change is wanted, if the test *should* pass, etc.

As an example, the test Alpine image is currently expected to be an
`alpine-virt` on an ISO 9660 (read-only) filesystem. Since this means
the EFI system partition is also read-only,
`guest_can_adjust_boot_order` is not a meaningful test (for now). We
run this test anyway and *expect* a skip, because if it passed then
either:
* the guest Alpine is on a writable root FS; are we missing test
  coverage involving read-only guest images?
* we've moved EFI NvVars to out-of-guest storage, and efivarfs writes
  succeed.

In either case, the corresponding changes should have some consideration
for how they relate Propolis and guest operations.
@iximeow iximeow force-pushed the phd-predicate-thingy branch from 49d2e1b to 6b0a6a5 Compare March 25, 2026 20:26
@iximeow
Copy link
Member Author

iximeow commented Mar 25, 2026

I'd fixed the clippy lints, and caught cargo fmt for my efforts.. the force push is a trivial diff, sorry: https://github.com/oxidecomputer/propolis/compare/49d2e1b422226458a90aa1a28b1b810c9de4e71c..6b0a6a589069f236536c79762099bfb21035b991

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Related to testing and/or the PHD test framework.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants