scrimlet-reconcilers status (PR 3/3): add omdb command and bootstrap-lockstep API to display status - #11058
Open
jgallagher wants to merge 10 commits into
Open
scrimlet-reconcilers status (PR 3/3): add omdb command and bootstrap-lockstep API to display status#11058jgallagher wants to merge 10 commits into
jgallagher wants to merge 10 commits into
Conversation
Contributor
Author
|
This command requires specifying the bootstrap address of a sled-agent; the easiest thing to do from a switch zone is to ask your own sled-agent, for multiple reasons:
This branch deployed on Asking for the other scrimlet's status is fine too, after finding its IP: Asking for the status from a non-scrimlet sled "works" but there's nothing interesting there: |
jgallagher
force-pushed
the
john/collapse-enum-variants
branch
from
August 12, 2026 15:14
c7d7161 to
e407b6d
Compare
jgallagher
force-pushed
the
john/scrimlet-reconcilers-status
branch
from
August 12, 2026 17:26
af9ea45 to
8b5c9db
Compare
karencfv
reviewed
Aug 12, 2026
| ScrimletReconcilersStatus::WaitingForSledAgentNetworkingInfo => { | ||
| write!( | ||
| f, | ||
| "not running: sled-agent has not yet initialized \ |
Contributor
There was a problem hiding this comment.
Shouldn't this be not yet running if we're still waiting?
Contributor
Author
There was a problem hiding this comment.
Not necessarily; we may never start running (e.g., if we're not a scrimlet).
Contributor
There was a problem hiding this comment.
Then how about changing the sled-agent has not yet initialized bit then? It implies that it will start running eventually no?
Comment on lines
+175
to
+184
| if let Some(last_completion) = last_completion { | ||
| writeln!(f, "last completion:")?; | ||
| writeln!( | ||
| IndentWriter::new(INDENT, f), | ||
| "{}", | ||
| ReconciliationCompletedStatusDisplay(&last_completion) | ||
| )?; | ||
| } else { | ||
| writeln!(f, "last completion: none")?; | ||
| } |
Contributor
There was a problem hiding this comment.
Nit (take it or leave it): It'd be nice if the "none" had the same indentation as when there is a last_completion
last completion:
none
Contributor
Author
There was a problem hiding this comment.
I don't love this because it makes the overall output vertically longer (and it's already annoying to have to scroll to see it all). There are several other places that have a something: none that would also bump up the vertical space if we applied this everywhere.
Comment on lines
+761
to
+762
| "{name} created / updated / deleted: \ | ||
| {created} / {updated} / {deleted}" |
Contributor
There was a problem hiding this comment.
Might be a little easier to read something like:
routers:
created: 0
updated: 0
deleted: 0
Contributor
Author
There was a problem hiding this comment.
Hmm, this will really blow up the vertical space too; BGP has a ton of counters to report. I could special case "all zero" and make it either
routers: unchanged
or break out counters by line if any of them are nonzero? That might be the best way to go - it simplifies the output in the common case (nothing changed) while breaking out the counters if there's anything interesting?
| { | ||
| writeln!( | ||
| f, | ||
| "v4 routes deleted / added: {deleted} / {added}" |
Contributor
There was a problem hiding this comment.
Same, how about:
v4 routes:
deleted: 0
added: 0| "format": "uuid" | ||
| }, | ||
| "RackInitializeRequest": { | ||
| "description": "Configuration for the \"rack setup service\".\n\nThe Rack Setup Service should be responsible for one-time setup actions, such as CockroachDB placement and initialization. Without operator intervention, however, these actions need a way to be automated in our deployment.", |
Contributor
There was a problem hiding this comment.
We most often refer to it as RSS, so it'd be nice to have that here so people new to the codebase make the connection
Suggested change
| "description": "Configuration for the \"rack setup service\".\n\nThe Rack Setup Service should be responsible for one-time setup actions, such as CockroachDB placement and initialization. Without operator intervention, however, these actions need a way to be automated in our deployment.", | |
| "description": "Configuration for the \"rack setup service\".\n\nThe Rack Setup Service (RSS) should be responsible for one-time setup actions, such as CockroachDB placement and initialization. Without operator intervention, however, these actions need a way to be automated in our deployment.", |
Contributor
Author
There was a problem hiding this comment.
I don't necessarily disagree, but this doesn't have anything to do with this PR. 😅 This is coming from a docstring on some RSS-related type I'm not touching.
Contributor
There was a problem hiding this comment.
Ahhhhhh the diff makes it look like this type wasn't in the OpenAPI doc before, but it was. nvm
| ] | ||
| }, | ||
| { | ||
| "description": "`id` will be none if the rack was already initialized on startup.", |
Contributor
There was a problem hiding this comment.
Can this description be more explicit about what the object is? Also, this type is a little light on the documentation. It's be nice to have docs on each variant
Contributor
Author
There was a problem hiding this comment.
Same as #11058 (comment), this isn't from changes in this PR. I don't think I want to mix in doc cleanup for unrelated types?
Contributor
Author
|
@karencfv Thanks for the review - this is ready for a second pass. I took several of your suggestions and left others out; here's example output from a racklette where we see several nonzero values from BGP settings being applied: and here's a steady-state output later when the reconciler has nothing to do, so we collapse the counts down to |
karencfv
reviewed
Aug 14, 2026
| "format": "uuid" | ||
| }, | ||
| "RackInitializeRequest": { | ||
| "description": "Configuration for the \"rack setup service\".\n\nThe Rack Setup Service should be responsible for one-time setup actions, such as CockroachDB placement and initialization. Without operator intervention, however, these actions need a way to be automated in our deployment.", |
Contributor
There was a problem hiding this comment.
Ahhhhhh the diff makes it look like this type wasn't in the OpenAPI doc before, but it was. nvm
| ScrimletReconcilersStatus::WaitingForSledAgentNetworkingInfo => { | ||
| write!( | ||
| f, | ||
| "not running: sled-agent has not yet initialized \ |
Contributor
There was a problem hiding this comment.
Then how about changing the sled-agent has not yet initialized bit then? It implies that it will start running eventually no?
Comment on lines
+835
to
+847
| writeln!(f, "routes unchanged: {unchanged}")?; | ||
|
|
||
| if let (&Ok(deleted), &Ok(added)) = | ||
| (delete_v4_result, add_v4_result) | ||
| { | ||
| if deleted == 0 && added == 0 { | ||
| writeln!(f, "v4 routes: unchanged")?; | ||
| } else { | ||
| writeln!(f, "v4 routes:")?; | ||
| let mut f = IndentWriter::new(INDENT, &mut f); | ||
| writeln!(f, "deleted: {deleted}")?; | ||
| writeln!(f, "added: {added}")?; | ||
| } |
Contributor
There was a problem hiding this comment.
This bit is a little confusing to me. In the sample output below it says there are routes unchanged: 0 which I assume means something changed, but then it says all v4 and v6 routes are unchanged. So what changed then?
routes unchanged: 0
v4 routes: unchanged
v6 routes: unchanged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is a lot smaller than it looks: ~2400 lines are changes to the lockstep API document, and another ~800 lines are
Displayimpls insideomdb. TheDisplayimpls are worth a quick review but aren't load bearing; I'll put example output from a racklette in a comment below.