Skip to content

[master] core: services: ardupilot_manager: Cap consecutive start failures - #4129

Open
joaoantoniocardoso wants to merge 1 commit into
bluerobotics:masterfrom
joaoantoniocardoso:fix/autopilot-start-failure-cap
Open

[master] core: services: ardupilot_manager: Cap consecutive start failures#4129
joaoantoniocardoso wants to merge 1 commit into
bluerobotics:masterfrom
joaoantoniocardoso:fix/autopilot-start-failure-cap

Conversation

@joaoantoniocardoso

Copy link
Copy Markdown
Member

Brings #4121 into master.

When no board can be started, auto_restart_ardupilot() retries every 5s
forever, even though nothing about the failure changes between attempts.
On a system whose SD card moved from a Pi with a Navigator to one
without, those retries only spend CPU and grow the service log by tens
of MB per day.

The same loop is what escalated the recently fixed i2c file descriptor
leak into an unrecoverable state: the retries exhausted the descriptors,
after which uvicorn's accept loop logged a traceback per retry, pegging
the CPU and taking the API down with it. That removed the only recovery
path, since selecting SITL requires a working /available_boards.

Give up after 10 consecutive failures instead. Transient failures still
self-heal, and an explicit start or board change re-arms the watchdog.

The counter lives in __init__ rather than setup(), because setup() runs
on every start attempt and would reset it.

Fixes bluerobotics#4070
@joaoantoniocardoso
joaoantoniocardoso requested a review from a team August 11, 2026 22:10
@github-actions

Copy link
Copy Markdown

Automated PR Review

0. Summary

  • Verdict: MINOR SUGGESTIONS ✏️

Adds a consecutive-failure counter to auto_restart_ardupilot() so that after _max_start_failures (10) failed start attempts, the loop stops retrying and clears should_be_running. Follows the existing _heartbeat_fail_count / _max_heartbeat_failures pattern. The counter is initialized in __init__ (not setup(), which runs per start attempt) and reset when is_running() observes the autopilot as up.

1. Correctness & Implementation Bugs

  • 1.1 [minor] core/services/ardupilot_manager/autopilot_manager.py:222 (new code) — after the cap trips, the counter is only reset via elif self.is_running(), which requires a successful start to be observed. If a user manually calls start_ardupilot() after being told "Start the autopilot or change the board to try again" and the first manual attempt fails transiently, start_ardupilot()'s finally (line 681) sets should_be_running = True, the auto-restart loop retries once, increments the (still-at-10) counter to 11, and immediately re-caps — before the user has a real chance to see recovery. Consider resetting _start_fail_count inside start_ardupilot() when it's invoked externally (or when should_be_running transitions False→True), so a manual restart gets a fresh budget.

6. Code Quality & Style

  • 6.1 [nit] core/services/ardupilot_manager/autopilot_manager.py:49-50 (new) — _max_start_failures = 10 is an instance attribute that never varies. This mirrors the existing _max_heartbeat_failures pattern (see setup() at line 163), so consistency-wise it's fine, but both would read better as module-level constants (e.g. MAX_START_FAILURES = 10). Out of scope for this PR; noting for future cleanup.

7. Tests

  • 7.1 [nit] No pytest coverage was added for the new failure-cap path. core/services/ardupilot_manager/ has no existing tests for auto_restart_ardupilot(), so this is consistent with the surrounding code rather than a regression, but adding a small async test that drives the loop with a mocked start_ardupilot would guard against future regressions in the counter/reset behavior.

Generated by PR Review Bot. This is advisory, a human reviewer must still approve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant