chore: Improved StateView and StateSnapshot lifetime logging

[sergerad]

Summary

Closes #2451.

Currently, if a snapshot has lasted beyond some threshold, we do a single warning log when that snapshot expires / drops. This means we might not hear about the lag promptly.

We also never report which LoC was responsible for a long-held (2s) StateView.

This PR adds

• per-block warning for snapshot log based on block distance; and
• StateView::drop log that shows the LoC where the StateView was created, E.G.:

WARN block_builder.build_block:block_builder.get_block_inputs: 
state view held for excessive time, pinning its snapshot generation 
caller=crates/block-producer/src/block_builder/mod.rs:233:14 block_num=46 
view.lifetime_ms=0 block.number=47 block.batches.count=0 block.batch.ids=None 
block.transactions.ids=None block.transactions.count=0

The above output was tested with both view() and with_view() calls.

Changelog

changelog = "none"
reason    = "Internal change only."

[Mirko-von-Leipzig]

I'm not sure if this is where we would want to do this because there is a lack of actionable information.

What do we now do once we receive this? We need to identify which query this is, but we have no way of doing so..

Perhaps we could explore a timer within the actual snapshot itself, and each snapshot taken automatically gets the caller LoC information embedded?

[sergerad]

I have added a LoC WARN log on long-held StateViews (StateSnapshots are not instantiated per-query, StateViews are).

The per-block log is still important in case we ever get snapshots or views that never end.

[Mirko-von-Leipzig]

Those two things are essentially synonyms. We should try improve our naming here. I assume its a snapshot because rocksdb calls them snapshots? Perhaps SmtView?

Is there a downside to having just a single one, instead of separate types? I can't imagine a snapshot is expensive to hold temporarily.

[sergerad]

StateViews and StateSnapshots are not synonyms - any number of views (one per API request) can map to a single snapshot. The view is there to enforce the invariants / API appropriate for accessing snapshot + SQL data consistently (block scoped requests). The snapshot is the non-SQL data itself.

[Mirko-von-Leipzig]

I understand that's what they are in our code. I'm saying the word view and the word snapshot mean almost exactly the same thing, and are therefore not good names for us to use.

If I say StateSnapshot everyone would assume that means a snapshot of our state at a moment in time. If I say StateView everyone would assume that means a view of our state at a moment in time.