Add sequencer queue and tx size metrics

[magicxyyz]

Resolves NIT-4418

This PR adds:

• sequencer metrics considering tx size: arb/sequencer/block/txsize and arb/sequencer/transactions/txsize histograms
• sequencer queue metrics: arb/sequencer/queue/normal, arb/sequencer/queue/timeboost, arb/sequencer/queue/histogram/normal, arb/sequencer/queue/histogram/timeboost, arb/sequencer/waitfortx
• sequencer block counter metrics: arb/sequencer/block/gasfull, arb/sequencer/block/datafull, arb/sequencer/block/txexhausted

[codecov]

Codecov Report

❌ Patch coverage is 88.88889% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.55%. Comparing base (890f0e2) to head (bd7bf05).
⚠️ Report is 58 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #4317       +/-   ##
===========================================
+ Coverage   33.98%   57.55%   +23.56%     
===========================================
  Files         488      488               
  Lines       57783    57807       +24     
===========================================
+ Hits        19640    33268    +13628     
+ Misses      34672    19598    -15074     
- Partials     3471     4941     +1470     

[github-actions]

❌ 5 Tests Failed:

Tests completed	Failed	Passed	Skipped
4179	5	4174	0

View the top 3 failed tests by shortest run time

TestSetLatestSnapshotUrl

Stack Traces | 0.070s run time

... [CONTENT TRUNCATED: Keeping last 20 lines]
        runtime/debug.Stack()
        	/opt/hostedtoolcache/go/1.25.6/x64/src/runtime/debug/stack.go:26 +0x5e
        github.com/offchainlabs/nitro/util/testhelpers.RequireImpl({0x2a20dd0, 0xc000405c00}, {0x29fa3e0, 0x3b473d0}, {0xc0003a6ac0, 0x1, 0x1})
        	/home/runner/work/nitro/nitro/util/testhelpers/testhelpers.go:29 +0x55
        github.com/offchainlabs/nitro/cmd/nitro/init.Require(0xc000405c00, {0x29fa3e0, 0x3b473d0}, {0xc0003a6ac0, 0x1, 0x1})
        	/home/runner/work/nitro/nitro/cmd/nitro/init/init_test.go:1273 +0x5d
        github.com/offchainlabs/nitro/cmd/nitro/init.startFileServer.func2()
        	/home/runner/work/nitro/nitro/cmd/nitro/init/init_test.go:362 +0xa5
        created by github.com/offchainlabs/nitro/cmd/nitro/init.startFileServer in goroutine 83
        	/home/runner/work/nitro/nitro/cmd/nitro/init/init_test.go:359 +0x285
        
    init_test.go:362: �[31;1m [failed to shutdown server] context canceled �[0;0m
INFO [02-09|17:48:36.279] Set latest snapshot url                  url=http://127.0.0.1:43223/arb1/2024/21/archive.tar.gz
    init_test.go:308: running test case latest file with http url
INFO [02-09|17:48:36.283] Set latest snapshot url                  url=http://some.domain.com/arb1/2024/21/archive.tar.gz
    init_test.go:308: running test case latest file with https url
INFO [02-09|17:48:36.285] Set latest snapshot url                  url=https://some.domain.com/arb1/2024/21/archive.tar.gz
    init_test.go:308: running test case chain and contents with upper case
INFO [02-09|17:48:36.286] Set latest snapshot url                  url=http://127.0.0.1:33261/arb1/2024/21/archive.tar.gz
--- FAIL: TestSetLatestSnapshotUrl (0.07s)

TestBroadcastClientConfirmedMessage

Stack Traces | 5.010s run time

=== RUN   TestBroadcastClientConfirmedMessage
=== PAUSE TestBroadcastClientConfirmedMessage
=== CONT  TestBroadcastClientConfirmedMessage
INFO [02-09|17:48:33.798] arbitrum websocket broadcast server is listening address=[::]:33617
    broadcastclient_test.go:327: broadcasting seq 0 message
INFO [02-09|17:48:33.798] connecting to arbitrum inbox message broadcaster url=ws://127.0.0.1:33617/
    broadcastclient_test.go:338: Received Message, Sequence Message: {0 {0xc0001d7800 0} <nil> [211 237 172 167 71 111 123 237 11 139 172 115 73 26 252 98 187 118 140 103 81 70 155 163 231 177 38 195 204 40 211 0 103 237 104 116 138 103 242 35 158 89 163 47 63 158 209 30 75 26 156 71 47 237 75 48 41 234 252 176 159 71 179 31 0] [] 0}
INFO [02-09|17:48:33.807] Feed connected                           feedServerVersion=2 chainId=8744 requestedSeqNum=0
INFO [02-09|17:48:33.809] Feed connected                           feedServerVersion=2 chainId=8744 requestedSeqNum=0
WARN [02-09|17:48:33.809] confirmed sequence number is past the end of stored messages "confirmed sequence number"=42 "last stored sequence number"=0
    broadcastclient_test.go:359: Client did not receive confirm message
--- FAIL: TestBroadcastClientConfirmedMessage (5.01s)

TestPrimaryToSecondaryFailover

Stack Traces | 5.430s run time

=== RUN   TestPrimaryToSecondaryFailover
=== PAUSE TestPrimaryToSecondaryFailover
=== CONT  TestPrimaryToSecondaryFailover
INFO [02-09|17:48:32.158] arbitrum websocket broadcast server is listening address=[::]:34435
    broadcastclients_test.go:206: Primary broadcaster listening on: [::]:34435
INFO [02-09|17:48:32.159] arbitrum websocket broadcast server is listening address=[::]:39795
    broadcastclients_test.go:207: Secondary broadcaster listening on: [::]:39795
    broadcastclients_test.go:230: Primary URL: ws://127.0.0.1:34435
    broadcastclients_test.go:231: Secondary URL: ws://127.0.0.1:39795
INFO [02-09|17:48:32.159] connecting to arbitrum inbox message broadcaster url=ws://127.0.0.1:34435
INFO [02-09|17:48:32.160] Feed connected                           feedServerVersion=2 chainId=1234 requestedSeqNum=0
    broadcastclients_test.go:278: Phase 1: Sending messages from primary broadcaster
INFO [02-09|17:48:37.369] secondary feed started                   url=ws://127.0.0.1:39795 startingFromSeq=4
INFO [02-09|17:48:37.369] connecting to arbitrum inbox message broadcaster url=ws://127.0.0.1:39795
INFO [02-09|17:48:37.370] Feed connected                           feedServerVersion=2 chainId=1234 requestedSeqNum=4
    broadcastclients_test.go:308: Timed out waiting for message 5/5 from primary
--- FAIL: TestPrimaryToSecondaryFailover (5.43s)

📣 Thoughts on this report? Let Codecov know! | Powered by Codecov

[eljobe]

@tsahee, there is some sensitivity for this review. Please take a look. I've reviewed it, and I don't see anything I'd like to change. But, I know you might be able to spot something I've missed.

[tsahee]

I'd check and if that's less than 1ms (or possibly config.ReadFromTxQueueTimeout), put in zero. We don't need to deal with history of how long it took to do the first loop.

[magicxyyz]

I'm adding the capping below 1ms.
I thin the config.ReadFromTxQueueTimeout is a different timeout that starts when wait for tx ends.

[joshuacolvin0]

LGTM