feat(orchestrator): track and report last status change timestamp#2980
feat(orchestrator): track and report last status change timestamp#2980jakubno wants to merge 10 commits into
Conversation
PR SummaryMedium Risk Overview Reviewed by Cursor Bugbot for commit 5bd7c94. Bugbot is set up for automated code reviews on this repo. Configure here. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
Retrieving the node or service status and its corresponding status change timestamp via separate, non-atomic method calls introduces a race condition where the status can transition between the calls, resulting in mismatched status and timestamp values in the API responses. To ensure consistency, atomic getters should be implemented and used to retrieve both fields under a single read lock.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1312408d6e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ef7bcb57c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b638e5acc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Add service_status_changed_at to ServiceInfoResponse so the API can see when the orchestrator's status last changed. The timestamp is updated on actual status transitions and initialized to the service startup time.
Consume service_status_changed_at from the orchestrator and track it on nodes and cluster instances. Expose it as statusChangedAt on Node and NodeDetail in the admin API.
- Return status and changed-at atomically under a single lock in both the orchestrator ServiceInfo and the API node manager to avoid mismatched pairs. - Track transitions of the connectivity-derived (reported) node status so statusChangedAt stays consistent with the status returned to callers. - Fall back to time.Now() in cluster instance sync when the status changes but the orchestrator does not report the timestamp.
Introduce ServiceStatus (orchestrator) and StatusInfo (api nodemanager) so status and its change timestamp are stored and returned as a single value instead of separate getters. Drop the instance sync time.Now() fallback since orchestrators report the timestamp.
Drop the time.Now() fallbacks that fabricated a change timestamp when the orchestrator does not report one. Keep time.Now() only for locally observed changes (sync failure, connectivity-derived transitions).
Store node status as a single StatusInfo field and drop the reported status transition tracking. The connectivity-derived status overlay only affects the returned value; the timestamp always refers to the last stored status change.
Drop the newer-timestamp merge logic in setStatus and store the orchestrator-reported value unconditionally; the orchestrator is the source of truth for when its status changed.
jakubno
force-pushed
the
feat/orchestrator-status-changed-at
branch
from
5b638e5 to
8f8a752
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8f8a752. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8f8a75297b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
2 participants
Add service_status_changed_at to ServiceInfoResponse so the API can see when the orchestrator's status last changed. The timestamp is updated on actual status transitions and initialized to the service startup time.