test/e2e: inject synthetic ext4 kmsg lines instead of a real fault#1305
test/e2e: inject synthetic ext4 kmsg lines instead of a real fault#1305DigitalVeer wants to merge 1 commit into
Conversation
kubernetes-prow
Bot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: DigitalVeer The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kubernetes-prow
Bot
added
the
cncf-cla: yes
kubernetes-prow
Bot
added
the
needs-ok-to-test
|
Hi @DigitalVeer. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Tip We noticed you've done this a few times! Consider joining the org to skip this step and gain Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
kubernetes-prow
Bot
added
the
size/M
DigitalVeer
force-pushed
the
fix/970-ext4-kmsg-injection
branch
from
d6454d7 to
5ae5fc6
Compare
kubernetes-prow
Bot
added
size/L
DigitalVeer
force-pushed
the
fix/970-ext4-kmsg-injection
branch
from
5ae5fc6 to
97d816b
Compare
kubernetes-prow
Bot
added
the
do-not-merge/invalid-commit-message
DigitalVeer
force-pushed
the
fix/970-ext4-kmsg-injection
branch
from
97d816b to
839319b
Compare
kubernetes-prow
Bot
removed
the
do-not-merge/invalid-commit-message
DigitalVeer
force-pushed
the
fix/970-ext4-kmsg-injection
branch
2 times, most recently
from
637fe50 to
02b5e71
Compare
The Ext4FilesystemError problem-maker wrote to /sys/fs/ext4/sda1/trigger_fs_error, a real fault on the boot disk. On kernel 6.5+ a real ext4 error shuts down the whole filesystem instead of remounting it read-only, so sshd dies and the e2e test loses its ssh connection. The ext4 metric assertions have been skipped because of this (issue 970). NPD only reads /dev/kmsg, so it does not need a real fault. OOMKill and DockerHung already inject their log lines into kmsg. Do the same for ext4. - write the two ext4 error lines to /dev/kmsg instead of triggering a fault - un-skip the ext4 metrics test; drop the "stays healthy" check (with no real fault there is nothing to survive, and the metric scrape already proves NPD is alive) - update the problem-maker README Signed-off-by: Veer Singh <8453348+digitalveer@users.noreply.github.com>
DigitalVeer
force-pushed
the
fix/970-ext4-kmsg-injection
branch
from
02b5e71 to
ea78733
Compare
kubernetes-prow
Bot
added
size/S
kubernetes-prow
Bot
removed
the
do-not-merge/work-in-progress
1 participant
The ext4 e2e test got skipped in #970.
problem-makertriggered a real ext4 error on the boot disk, and on kernel 6.5+ this shut down the whole filesystem instead of just remounting it read-only. This caused sshd to die, making the test lose its ssh connection.Since NPD only reads /dev/kmsg, it does not need a real fault, and the other existing problem makers do not attempt to emulate a real fault (they inject into kmsg). This reenables the test, drops a stale "stays healthy" check, and updates the problem-maker README to reflect the synthetic injection.
fixes #970