FM: how to handle inventory collection in sitrep inputs?

[hawkw]

The current inventory collection is an input to FM analysis. Presently, when we create a sitrep, we record the collection ID of the inventory that was current when the sitrep was produced. I think that diagnosis engines and such will eventually want to be able to compare (and especially, diff) the current inventory with the one that was input to the prior sitrep, so that we can see if the state of the system has changed. Unfortunately, inventory collections are large and get deleted very aggressively, so it is possible for a sitrep to be produced where the parent sitrep's inventory doesn't exist at all. This is also an issue when trying to interpret old sitreps in e.g. OMDB or archived debugging records. I wrote a bit about this in RFD 603, but didn't come up with a solution...

I wonder if we ought to be copying the entire contents of the inventory collection into a big pile of fm_inv_$WHATEVER tables, or something, so that the sitrep can be completely self-describing. In general, this is the approach that feels conceptually correct: the sitrep should contain all the data necessary to understand the sitrep. But, on the other hand, actually doing it in practice seems horrifically gross because the inventory is already a huge pile of tables, and we'd essentially have to duplicate all the schemas and model types and such, and then make sure we can reassemble them into the same domain types representing an inventory as the ones that come from the actual inventory tables. All of this would also require updating as more stuff is added to the inventory. So, um. Man. That sucks. And, in addition to being painful to implement, it also means we 're duplicating all this stuff on disk in CRDB.

Alternatively, we might consider modifying the inventory collection GC logic to retain inventory collections which were input to at least the current sitrep and possibly some buffer of prior ones. This is also a bit sketchy, especially because, well...we were deleting them for a reason. Hm.

@smklein and @jgallagher, any thoughts?

[hawkw]

It's also possible I'm kinda overthinking it and we don't actually need to be diffing the whole inventory, and that any information we care about comparing to the present will just be contained in the parent sitrep. I dunno.

[jgallagher]

I wonder if we ought to be copying the entire contents of the inventory collection into a big pile of fm_inv_$WHATEVER tables, or something, so that the sitrep can be completely self-describing.

I would very strongly vote against this path, for all the reasons you listed. Updating one set of inventory tables when making changes (even simple ones like additions) is already a relatively painful / laborious step without duplicating everything.

I think in principle the blueprint planner has similar issues here. It takes the inventory collection as one of its inputs, and emits both a blueprint and a planning report that can only be 100% truly understood / recreated if the input inventory is available, but it's generally not due to the aggressive GC. This hasn't been a problem for us in practice; I think when it's come up, there have been enough breadcrumbs left behind that we've been able to infer whatever bits of inventory the planner looked at must have been at the time.

I'd suggest these options in order of preference:

1. Just let the GC run, and make sure there's enough info in the sitrep that you won't regret not having a copy (I have no idea if this is reasonable for this case!)
2. Save a copy of the inventory in a debug-only way, potentially outside of CRDB entirely. @davepacheco is working on this for reconfigurator now, where I believe we'll dump an UnstableReconfiguratorState JSON blob (which contains all the currently-available inventory collections, plus a bunch of other stuff - everything we'd need to load up the system in reconfigurator-cli offline and poke at it) to a debug dropbox on disk that eventually gets rotated out to the debug dataset like logs
3. Pin some collections as "not GC'able" (IIRC we have talked about doing this for blueprints at various points, but haven't bothered because of the "in practice it's been fine" bit above, and I think given the debug work going on we probably wouldn't ever do this). I think this might be moderately annoying in practice, since you'd have to go back and "unpin" them eventually, which is sort of its own mini GC pass?

[smklein]

• We have, for the inventory, stated that "it is lossy", so for better or worse, we do need to cope with the possibility of reading an inventory that only has partial information.
• We could probably tweak GC to make it "much less likely that a reasonable sitrep planner ever loses inventory information". This won't help for the historical case, but it can at least make it less likely for the latest iteration of the sitrep to miss inventory info.

I suspect that in the common case, we'll do the following:

• As part of the preparation phase, we'll load the latest inventory from the database. We hope it isn't lossy. We'll need to cope with it being lossy anyway.
• We'll pass an in-memory representation of the inventory from preparation to the planning phase.
• If we make decisions based off of the inventory, we can "write them down" in the output sitrep. "We did X because we saw Y". Or, if there's a specific part of the inventory we want to mention, we should be explicit about that in the sitrep, rather than wholesale copying the input. For example: "We're seeing something odd with a disk that has this serial number, this firmware revision, etc" - basically re-structure the pieces of data we care about from inventory -> sitrep.

All this is to say: We hopefully won't need to re-serialize the inputs (we could save them as JSON for visibility, as @jgallagher suggested for debugging - but that basically means we're not expecting to re-read them outside of the use case of human eyeballs). And if we infer something from the input, and want to save it in the output, it perhaps should be a new type?

I feel like I'm perhaps saying a lot of the same thing @jgallagher is saying. I think a lot of this is contingent on:

and make sure there's enough info in the sitrep that you won't regret not having a copy (I have no idea if this is reasonable for this case!)

I also think that this perhaps challenges your assertion in the original issue:

I think that diagnosis engines and such will eventually want to be able to compare (and especially, diff) the current inventory with the one that was input to the prior sitrep, so that we can see if the state of the system has changed.

Do we actually need to have a full diff the raw inventory collections? Or could we model this as "the sitrep is monitoring for some condition in the inventory", which we can re-evaluate against a new inventory?

If we could re-frame this, then we could save "the condition we're looking for" in the sitrep, instead of the raw original inventory.

[hawkw]

Thanks for the discussion, this is really helpful. I definitely agree that duplicating an entire inventory collection in the sitrep would be quite unfortunate, and would much rather avoid doing that. Similarly, the GC pinning thing also feels like it would be uncomfortably complex and I'd also prefer to not do that --- if we can get away with just copying whatever data is actually part of the decisions made in a given sitrep, that would be ideal.

@smklein re:

I also think that this perhaps challenges your assertion in the original issue:

I think that diagnosis engines and such will eventually want to be able to compare (and especially, diff) the current inventory with the one that was input to the prior sitrep, so that we can see if the state of the system has changed.

Do we actually need to have a full diff the raw inventory collections? Or could we model this as "the sitrep is monitoring for some condition in the inventory", which we can re-evaluate against a new inventory?

So, the condition I was imagining we would need to diff inventory collections in order to detect is changes in the inventory of the rack, at a high level, like a sled being added or removed. We would expect to get ereports for at least some of the events involved in a rack inventory change, but I think we also would want to be able to detect those changes through other means in case some ereports are missed. But, I suppose we don't actually need the entire inventory collection for that; we could instead just create sitrep tables representing just, say, which sleds are present based on the input inventory. And, I'm not even sure if we would really need to track stuff like this as part of FM outside of trying to track the resolution of existing cases, so...hm.

[smklein]

And, I'm not even sure if we would really need to track stuff like this as part of FM outside of trying to track the resolution of existing cases, so...hm.

I think that having some concrete examples here would help us a lot.

You're for sure correct that "there are going to be cases watching for sled additions/removals", but I'm not sure how that'll fit into higher-order "case opening, case identification, etc" processes.

Reading the database tables related to sled inventory, I could imagine a few problems:

1. Problem: Sled dies, goes offline. Was a part of inventory previously. It isn't there now.
2. Problem: Sled is boot looping. It'll appear in inventory, then disappear, then re-appear. The SP may or may not be able to give us ereports about this. If the sled shows up in the inventory, there might be some boot information about reading boot partitions.
3. Problem: Sled agent has a mupdate override file written to it, which for whatever reason, cannot be cleared, and thats blocking it from getting subsequent updates. This info would appear in the inventory.

In all of these cases, I think there would be valuable information in e.g. the inv_sled_config_reconciler table, which I can imagine the diagnosis engine wanting to see - and it totally should be able to read the inventory.

But for these examples, we'd want to see:

1. Sled dying, going offline: Ideally we'd see an explicit ereport, but "not seeing an entry for a specific sled" while others exist in the most recent inventory collection would be a hint.
2. Sled boot looping: Again, ideally we'd get ereports from the SP, but we could record "we saw the sled trying to boot from partition X, using image Y". I'm not sure this actually should be a diff. It might just be a list of "here's are all the boot attempts we've observed from that sled".
3. For the mupdate override case, the status of "trying to clear mupdate override" is directly accessible in inventory. There shouldn't really need to be a diff here.

Are there other scenarios you're imagining where "a diff from inventory A to B" would be the observation we actually want, rather than:

• "View the contents of inventory A" -> make some immediate decision, recorded in the sitrep
  or
• "View the contents of inventory A, B, C, D, etc" -> make a decision based on the cumulative things seen in all of them (e.g., "here are all the boot attempts we've seen on this sled, so far")

[smklein]

anyway, backing up a sec - the reason I'm poking at that question of "when do we need a diff?" is because I am trying to unravel that question of "do we actually need to pin/duplicate inventory tables" vs "can FM store other tables that are more specific about what it wants". Truthfully, between the two, I dunno which is more viable. Hoping it's the latter! but that seems very contingent on what the diagnosis engine needs to properly create/update cases.