Skip to content

[libvirt_manager] Add retries to controller SSH key inject tasks#4057

Open
skovili wants to merge 1 commit into
openstack-k8s-operators:mainfrom
skovili:fix/inject-key-retries
Open

[libvirt_manager] Add retries to controller SSH key inject tasks#4057
skovili wants to merge 1 commit into
openstack-k8s-operators:mainfrom
skovili:fix/inject-key-retries

Conversation

@skovili

@skovili skovili commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds retries: 5, delay: 30 to both "Inject private key" and "Inject public key" tasks in manage_vms.yml
  • Matches the retry pattern already used for OCP SSH access (line 62-75, retries: 5, delay: 60)
  • Fixes a race condition where cloud-init hasn't written authorized_keys before the inject task attempts SSH, causing Permission denied (publickey) on controller VMs

Root cause

SSHD starts independently of cloud-init on RHEL 9. The SSH port opens ~16s after VM boot, but cloud-init may not finish writing SSH keys until 30-60s post-boot. Without retries, the inject task fails immediately on first attempt.

Observed in testproject build 5ac3f032 on devin07 (reused hypervisor, routable IPs through physical bridge), but this is a latent race condition that could affect any environment where cloud-init is slower than usual.

Test plan

  • Verify via ci-framework-testproject build with Depends-On: pointing to this PR
  • Confirm inject tasks succeed with retries on devin07 testproject

Related-Issue: #OSPRH-32065

🤖 Generated with Claude Code

Cloud-init may not finish writing authorized_keys before the inject
tasks attempt SSH, causing "Permission denied (publickey)" on
controller VMs. This is the same race the OCP SSH task already handles
with retries.

Related-Issue: #OSPRH-32065

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci

openshift-ci Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tosky for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/d066cd195b724690bfa99b689b485c9c

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 32m 27s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 35m 29s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 48m 18s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 15m 54s
✔️ cifmw-pod-zuul-files SUCCESS in 6m 33s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 10m 35s
✔️ cifmw-pod-pre-commit SUCCESS in 10m 08s
cifmw-molecule-libvirt_manager FAILURE in 32m 08s
cifmw-molecule-reproducer FAILURE in 11m 32s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant