Skip to content

fix: recover from panics in LoadSectorStates#1115

Open
frrist wants to merge 2 commits intomasterfrom
frrist/recover-panic-loadsectorstate
Open

fix: recover from panics in LoadSectorStates#1115
frrist wants to merge 2 commits intomasterfrom
frrist/recover-panic-loadsectorstate

Conversation

@frrist
Copy link
Copy Markdown
Member

@frrist frrist commented Jan 19, 2023

birdychang
birdychang approved these changes Jan 19, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@birdychang birdychang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@davidgasquez
Copy link
Copy Markdown

davidgasquez commented Jan 23, 2023

I'm having some issues with this PR. This is the lily.log.

{"level":"info","ts":"2023-01-22T12:48:26.438Z","logger":"lily/index/manager","caller":"integrated/manager.go:125","msg":"task success","height":"2399050","reporter":"walk_1674391696","task":"verified_registry_verif
ier","status":"OK","duration":9.576490392}
{"level":"error","ts":"2023-01-22T12:48:26.536Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01944708","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01944708.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"info","ts":"2023-01-22T12:48:26.766Z","logger":"lily/index/processor","caller":"processor/state.go:362","msg":"processor ended","task":"miner_sector_post","height":"2399050","reporter":"walk_1674391696","d
uration":9.904742493}
{"level":"info","ts":"2023-01-22T12:48:26.766Z","logger":"lily/index/manager","caller":"integrated/manager.go:125","msg":"task success","height":"2399050","reporter":"walk_1674391696","task":"miner_sector_post","sta
tus":"OK","duration":9.904740094}
{"level":"error","ts":"2023-01-22T12:48:27.338Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01983523","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:27.395Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01915730","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01915730.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:27.544Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01283930","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01283930.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:27.591Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01931245","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01931245.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:27.827Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f014409","code":"fil
/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014409.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change typ
e modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"info","ts":"2023-01-22T12:48:27.835Z","logger":"lily/index/processor","caller":"processor/state.go:362","msg":"processor ended","task":"data_cap_balance","height":"2399050","reporter":"walk_1674391696","du
ration":10.973147105}
{"level":"info","ts":"2023-01-22T12:48:27.835Z","logger":"lily/index/manager","caller":"integrated/manager.go:125","msg":"task success","height":"2399050","reporter":"walk_1674391696","task":"data_cap_balance","stat
us":"OK","duration":10.973145203}
{"level":"error","ts":"2023-01-22T12:48:28.536Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01907880","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01907880.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:28.565Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0396473","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f0396473.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change t
ype modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:29.248Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01915033","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01915033.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:29.248Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01915033","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:29.248Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01915033","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:29.410Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0126037","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f0126037.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change t
ype modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:29.493Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01983523","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01983523.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.319Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0215497","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f0215497.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change ty
pe modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.414Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01964132","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01964132.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.496Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01172955","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01172955.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.496Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01172955","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:30.496Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01172955","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:30.794Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0883206","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f0883206.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change ty
pe modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.813Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f010088","code":"fil
/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f010088.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change type
 modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:31.342Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01157027","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01157027.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:31.527Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01790067","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01790067.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:31.527Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01790067","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:31.527Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01790067","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:31.683Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01157018","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01157018.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:31.864Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01154901","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:31.864Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01154901","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:31.867Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01154901","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01154901.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:32.016Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01858258","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01858258.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:32.077Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0413563","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f0413563.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change ty
pe modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:32.077Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0413563","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: context canceled"}

The daemon is kept running but the walk is not completing. The list command outputs this:

[
        {
                "ID": 1,
                "Name": "walk_1674391696",
                "Type": "walk",
                "Error": "",
                "Tasks": [
                        "block_header",
                        "block_parent",
                        "drand_block_entrie",
                        "data_cap_balance",
                        "miner_beneficiary",
                        "miner_sector_deal",
                        "miner_sector_infos_v7",
                        "miner_sector_infos",
                        "miner_sector_post",
                        "miner_pre_commit_info",
                        "miner_sector_event",
                        "miner_current_deadline_info",
                        "miner_fee_debt",
                        "miner_locked_fund",
                        "miner_info",
                        "market_deal_proposal",
                        "market_deal_state",
                        "message",
                        "block_message",
                        "receipt",
                        "message_gas_economy",
                        "parsed_message",
                        "internal_messages",
                        "internal_parsed_messages",
                        "vm_messages",
                        "multisig_transaction",
                        "chain_power",
                        "power_actor_claim",
                        "chain_reward",
                        "actor",
                        "actor_state",
                        "id_addresses",
                        "derived_gas_outputs",
                        "chain_economics",
                        "chain_consensus",
                        "multisig_approvals",
                        "verified_registry_verifier",
                        "verified_registry_verified_client"
                ],
                "Running": true,
                "RestartOnFailure": false,
                "RestartOnCompletion": false,
                "RestartDelay": 0,
                "Params": {
                        "maxHeight": "2399050",
                        "minHeight": "2396162",
                        "storage": "CSV",
                        "window": "0s"
                },
                "StartedAt": "2023-01-22T12:48:16.859521259Z",
                "EndedAt": "0001-01-01T00:00:00Z"
        }
]

@frrist
Copy link
Copy Markdown
Member Author

frrist commented Jan 23, 2023

@davidgasquez is it failing for all epochs or just some?

@davidgasquez
Copy link
Copy Markdown

davidgasquez commented Jan 23, 2023

I think is just some (will check tomorrow when I'm back). That said, it happens in most of them and should be easy to reproduce in any recent partial archive.

@davidgasquez
Copy link
Copy Markdown

davidgasquez commented Jan 24, 2023

I've used historical-exports_snapshot_100800_103682_1666950560.car.zst locally and was able to reproduce the issue by walking any range of epochs.

The visor_processing_report.csv is full (many rows per epoch) of lines like this one:

100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceca3b27wyrbg6y5rowz3lik74gq7f3eiknem7cb5jscyfvujfaymy"",""Address"":""f018469"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f018469.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedreddfw7c5l3tlxxfrgsc34g7vjja4nb64zyvlyhmd6xnd7226t2"",""Address"":""f07125"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07125.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb6p62a5v3hyogiv37qs4k5w7ayyobdxsiiuszurh4ctdp2csh3q4"",""Address"":""f021755"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f021755.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaced33gej7xvwo6aqzvxfreteetezkciuok276yv3bc5ag53zrhfhyq"",""Address"":""f017484"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f017484.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f04748"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04748.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecwgrhywq4h2dqmkxfcbxpt4v3d6lmlpfvcipzbw3wmnsztmlblhk"",""Address"":""f07673"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07673.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebmzdvcnagkwq3vq43xee2ebxm4dcabegpot2ew7pg6uvdoxgqycc"",""Address"":""f04417"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04417.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb2n2l5ksc5jblp5ukvgxzcs7ringfjp2jwcoov2bp2sl335uejzw"",""Address"":""f06974"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06974.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebrpyuhnvrnutgzqa4garkwgqmqkwadfq2iflesc2tjohebxijr5m"",""Address"":""f015772"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f015772.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebzurbfrykvuzvh2ksrg5i3etnp4ry42gkzje2sqa3z3hqxenrbim"",""Address"":""f017314"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f017314.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceapm44p4ul5vwla27v2vt2vb2secfittiumtick2shn3ozx2its7g"",""Address"":""f02159"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f02159.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedypbk2cchvsq6wwyzagpoeo2cu7bmh4p5vnlnj5skjer7criisag"",""Address"":""f07543"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07543.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07608"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07608.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07089"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07089.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceasjwnxgnrcswzduuu667ewl4wkge5dhktgoudoiqqlslavlmei6w"",""Address"":""f012056"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f012056.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea2k6pvhwp6b6qubzd2irseteyepcp2vw7gfcfcqtqhwqibrz65ge"",""Address"":""f06469"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06469.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07634"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07634.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebaaeo6fus4t3h2hajf4ljbfqhzznsw5ixk3ehvru7exebo32smya"",""Address"":""f08895"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f08895.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedfmo4t7bgc5bjnaqeag2zf5ba6eap3fg4qg42vzb3rgxj53xpc4a"",""Address"":""f03778"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f03778.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceabr32exarjf4frpmz7nnytjdgd4mdqtvpswhj3nmf7g3dzx62mwq"",""Address"":""f011606"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f011606.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb66z6dgo32u5xbju2s2fi7pftaq7aqayi4htuiiyst3u6fzdek62"",""Address"":""f014682"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014682.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaced5c7gjsiqrzriopbb5uv7vbvodxixlbynxittd65zf7bel36krri"",""Address"":""f018422"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f018422.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedqqqf2jnzdxh6ie3juo5hxx4ufervorjs7qhqeil6tylsm55rzye"",""Address"":""f04519"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04519.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaxh66opmtlekt6mcpsovx6zdi2buyx7bfjs7flwq2onlyukwpnay"",""Address"":""f020146"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f020146.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceckmqxgee3y5x4pn3p2rxnbvq2ocgpfdbqp242n6wmfvk3z25lr7a"",""Address"":""f05065"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05065.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaqc6ip4evgj7irwfdruthwxdep2f6unh4qh73qul5n54e5kadcea"",""Address"":""f06883"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06883.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceawrzb5rtgbeq25ljbibi3adk6zhocot3mvdvelmmytuaoplsihby"",""Address"":""f04297"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04297.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebtw3vey4cxsjma7lxan5zejo73huu43bs2sqcrbndswqaiiaj77u"",""Address"":""f018790"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f018790.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebik77jtdpzbreekpl6w3mpozed7poila5su5vusa3w5aaakkb2io"",""Address"":""f05766"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f05766.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebcecqsgmq4hcybquaoskottlgrghe7nhnfe45cbqojhhrkebyaau"",""Address"":""f05509"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f05509.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebnycpqatkwhny46aww7azwvfxsdl2qbuqf4kyv7m2ljtbv6l27bw"",""Address"":""f019317"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f019317.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacec67coauakuxgkv4b5s3kjdq5ckd5t4p37fe5dqhwix65yxgrztmg"",""Address"":""f010346"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f010346.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceckmqxgee3y5x4pn3p2rxnbvq2ocgpfdbqp242n6wmfvk3z25lr7a"",""Address"":""f07042"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07042.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceagdury54uijonwrzvasnkbhrs5pf2bggnqaciskt736jnlnmbmec"",""Address"":""f015748"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f015748.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceagymfhc5tv4ulcthjdtnla7z54tpprqhdbtflzvjsixe3ehkfzek"",""Address"":""f015771"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f015771.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacearowuwpzp7zjbfzzxfadhlygozwpkd7d7oc7x3rw76dmuzbzr32e"",""Address"":""f04688"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04688.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacec6xmpyail2rlpy5vis6j3l7dnnwt2bdar2nnwrlys43wqebppj66"",""Address"":""f07594"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07594.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceavmvddz2e2nwoja36nqgmaewx7jgirdcwsiefwgxowjvnxyhfaug"",""Address"":""f07467"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07467.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb5d56vudzrpcaf7pgi2wacfo4wacg6lokyce55qokaxt7ww6k43k"",""Address"":""f012067"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012067.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea2k6pvhwp6b6qubzd2irseteyepcp2vw7gfcfcqtqhwqibrz65ge"",""Address"":""f06716"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06716.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebrkycr6iso7ujfmg3xc4qpjghzri7ah57sce7gqpq3eeojaht4li"",""Address"":""f06963"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06963.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f06340"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06340.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacectsnumtiverwj7owv4cme5wlfmw5bmeg5f23ulsqeyy3htwk5vp6"",""Address"":""f010224"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f010224.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedpgyirsxcvo7tbbasttkssmpphvtnunvuc7kngf6aogxxjrpd5v2"",""Address"":""f012509"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f012509.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebzhytzk2v2x4q3gx5yxi2xkiiqcv2cxobvv4nygwrwsz4o27t2ew"",""Address"":""f013057"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f013057.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f04676"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04676.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecbfijzx7bqhl6yutgw6f3pryfrmgkryugeuwfa7vuio7bba65dmi"",""Address"":""f04155"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f04155.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedct2ex4tpioukkbrkwc6c7lnkrvnlullz4yyytg7befolvyoq5u2"",""Address"":""f04903"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04903.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaszfxuxlipsi7qlmbtimsdekei6stgxl5gcma3dzx3tfg4jfcmqy"",""Address"":""f04588"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04588.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea4pwqlr32w4mvo56dkwnpmozlzn3t4sn4yds3mzoaqg3wertd2sg"",""Address"":""f014419"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f014419.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceadzscpxothvbyw5r65q7ih63jp3mrjl67truiv72atuehx6r6sro"",""Address"":""f01652"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01652.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedtgbbumq7y4o5vefp7pjke23l7bfu5akyfv7znffzry3vchbebpq"",""Address"":""f011712"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f011712.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceck4zeazi3f22spyd2trzqov3rbe6ltedpadjvvexaxpimq7tyupc"",""Address"":""f05681"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05681.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedgqzrqcutottoas4f3wnwqoxs52wrnj625xkn75seizuuy6wxlpo"",""Address"":""f04257"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f04257.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebik77jtdpzbreekpl6w3mpozed7poila5su5vusa3w5aaakkb2io"",""Address"":""f05374"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05374.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebcecqsgmq4hcybquaoskottlgrghe7nhnfe45cbqojhhrkebyaau"",""Address"":""f07625"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07625.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecj5jx6ft77ycxhlp2jayys35ouuyohyui3n6e2men6v32fvoyyes"",""Address"":""f05131"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05131.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecwgrhywq4h2dqmkxfcbxpt4v3d6lmlpfvcipzbw3wmnsztmlblhk"",""Address"":""f06393"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06393.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedeahyjfo4cb6puhvasijej4bzlkx2b3c5rovhzyhmj42yvnqez7m"",""Address"":""f08993"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f08993.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaksxc4fdbhmi2qbxj7eslfjzut5wwyurpjwqbg5lmoqoszrsprm2"",""Address"":""f012588"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012588.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceac5uhbv4h5wpc5ozyjhklusau2h64wljibwqmtv6ckcot5mk3jg2"",""Address"":""f014561"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014561.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceavizu74t5bsazc4rk4tne2al5xsn3i4hb5aop4jgy5jby3zlc26s"",""Address"":""f014749"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014749.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceas5567stmr7v4kgqpre6a5izgw744u3zt433hqcu7xdiqvcn3meo"",""Address"":""f012445"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012445.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebc35qfrxz22e6doc7awjlxpmbbmq3nqi22evlp3446qtpqpkj7ws"",""Address"":""f016519"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f016519.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedd535k7kwm7kfg3y6c32fdhwb4zuf7ur55jt44dxx6ozfu753rhe"",""Address"":""f012022"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012022.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceause7762ca2nbs4xkhcy6zql63qbjhaovmfx3jndgnmy2netwhbg"",""Address"":""f07705"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07705.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",

There is also a bunch of empty files:

-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_fee_debts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_infos.csv
-rw-r--r-- 1 root  root  449K Jan 24 16:28 arch0124-2020-09-28-miner_locked_funds.csv
-rw-r--r-- 1 root  root  1.6M Jan 24 16:28 arch0124-2020-09-28-miner_pre_commit_infos.csv
-rw-r--r-- 1 root  root  6.9K Jan 24 16:28 arch0124-2020-09-28-miner_sector_deals.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_sector_events.csv
-rw-r--r-- 1 root  root  1.8M Jan 24 16:28 arch0124-2020-09-28-miner_sector_infos.csv
-rw-r--r-- 1 root  root   19M Jan 24 16:28 arch0124-2020-09-28-miner_sector_posts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-multisig_approvals.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-multisig_transactions.csv
-rw-r--r-- 1 root  root   26M Jan 24 16:28 arch0124-2020-09-28-parsed_messages.csv
-rw-r--r-- 1 root  root  250K Jan 24 16:28 arch0124-2020-09-28-power_actor_claims.csv
-rw-r--r-- 1 root  root  2.5M Jan 24 16:28 arch0124-2020-09-28-receipts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-verified_registry_verified_clients.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-verified_registry_verifiers.csv

Finally, in lily.log, we can see some websocket: close 1000 (normal) errors.

{"level":"error","ts":"2023-01-24T15:27:57.608Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":100904,"actor":"fil/1/storageminer","address":"f02303","code":"fil/1/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f02303.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100904: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"debug","ts":"2023-01-24T15:27:57.798Z","logger":"lily/tasks/miner","caller":"miner/sector_events.go:109","msg":"diff sectors","miner":"f02731","duration":3.888790994}
{"level":"error","ts":"2023-01-24T15:27:57.798Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":100904,"actor":"fil/1/storageminer","address":"f02731","code":"fil/1/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f02731.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100904: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"debug","ts":"2023-01-24T15:27:57.808Z","logger":"rpc","caller":"go-jsonrpc@v0.1.8/websocket.go:624","msg":"websocket error","error":"websocket: close 1000 (normal)"}
{"level":"debug","ts":"2023-01-24T15:27:57.999Z","logger":"watchdog","caller":"go-watchdog@v1.3.0/watermarks.go:32","msg":"watermark policy: evaluating; utilization: 10881658880/33573044224 (used/limit)"}

That said, the walk seems to continue without problems after these websocket errors.

1 similar comment
@davidgasquez
Copy link
Copy Markdown

davidgasquez commented Jan 24, 2023

I've used historical-exports_snapshot_100800_103682_1666950560.car.zst locally and was able to reproduce the issue by walking any range of epochs.

The visor_processing_report.csv is full (many rows per epoch) of lines like this one:

100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceca3b27wyrbg6y5rowz3lik74gq7f3eiknem7cb5jscyfvujfaymy"",""Address"":""f018469"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f018469.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedreddfw7c5l3tlxxfrgsc34g7vjja4nb64zyvlyhmd6xnd7226t2"",""Address"":""f07125"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07125.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb6p62a5v3hyogiv37qs4k5w7ayyobdxsiiuszurh4ctdp2csh3q4"",""Address"":""f021755"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f021755.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaced33gej7xvwo6aqzvxfreteetezkciuok276yv3bc5ag53zrhfhyq"",""Address"":""f017484"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f017484.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f04748"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04748.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecwgrhywq4h2dqmkxfcbxpt4v3d6lmlpfvcipzbw3wmnsztmlblhk"",""Address"":""f07673"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07673.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebmzdvcnagkwq3vq43xee2ebxm4dcabegpot2ew7pg6uvdoxgqycc"",""Address"":""f04417"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04417.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb2n2l5ksc5jblp5ukvgxzcs7ringfjp2jwcoov2bp2sl335uejzw"",""Address"":""f06974"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06974.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebrpyuhnvrnutgzqa4garkwgqmqkwadfq2iflesc2tjohebxijr5m"",""Address"":""f015772"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f015772.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebzurbfrykvuzvh2ksrg5i3etnp4ry42gkzje2sqa3z3hqxenrbim"",""Address"":""f017314"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f017314.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceapm44p4ul5vwla27v2vt2vb2secfittiumtick2shn3ozx2its7g"",""Address"":""f02159"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f02159.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedypbk2cchvsq6wwyzagpoeo2cu7bmh4p5vnlnj5skjer7criisag"",""Address"":""f07543"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07543.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07608"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07608.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07089"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07089.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceasjwnxgnrcswzduuu667ewl4wkge5dhktgoudoiqqlslavlmei6w"",""Address"":""f012056"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f012056.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea2k6pvhwp6b6qubzd2irseteyepcp2vw7gfcfcqtqhwqibrz65ge"",""Address"":""f06469"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06469.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07634"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07634.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebaaeo6fus4t3h2hajf4ljbfqhzznsw5ixk3ehvru7exebo32smya"",""Address"":""f08895"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f08895.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedfmo4t7bgc5bjnaqeag2zf5ba6eap3fg4qg42vzb3rgxj53xpc4a"",""Address"":""f03778"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f03778.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceabr32exarjf4frpmz7nnytjdgd4mdqtvpswhj3nmf7g3dzx62mwq"",""Address"":""f011606"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f011606.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb66z6dgo32u5xbju2s2fi7pftaq7aqayi4htuiiyst3u6fzdek62"",""Address"":""f014682"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014682.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaced5c7gjsiqrzriopbb5uv7vbvodxixlbynxittd65zf7bel36krri"",""Address"":""f018422"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f018422.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedqqqf2jnzdxh6ie3juo5hxx4ufervorjs7qhqeil6tylsm55rzye"",""Address"":""f04519"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04519.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaxh66opmtlekt6mcpsovx6zdi2buyx7bfjs7flwq2onlyukwpnay"",""Address"":""f020146"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f020146.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceckmqxgee3y5x4pn3p2rxnbvq2ocgpfdbqp242n6wmfvk3z25lr7a"",""Address"":""f05065"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05065.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaqc6ip4evgj7irwfdruthwxdep2f6unh4qh73qul5n54e5kadcea"",""Address"":""f06883"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06883.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceawrzb5rtgbeq25ljbibi3adk6zhocot3mvdvelmmytuaoplsihby"",""Address"":""f04297"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04297.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebtw3vey4cxsjma7lxan5zejo73huu43bs2sqcrbndswqaiiaj77u"",""Address"":""f018790"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f018790.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebik77jtdpzbreekpl6w3mpozed7poila5su5vusa3w5aaakkb2io"",""Address"":""f05766"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f05766.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebcecqsgmq4hcybquaoskottlgrghe7nhnfe45cbqojhhrkebyaau"",""Address"":""f05509"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f05509.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebnycpqatkwhny46aww7azwvfxsdl2qbuqf4kyv7m2ljtbv6l27bw"",""Address"":""f019317"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f019317.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacec67coauakuxgkv4b5s3kjdq5ckd5t4p37fe5dqhwix65yxgrztmg"",""Address"":""f010346"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f010346.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceckmqxgee3y5x4pn3p2rxnbvq2ocgpfdbqp242n6wmfvk3z25lr7a"",""Address"":""f07042"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07042.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceagdury54uijonwrzvasnkbhrs5pf2bggnqaciskt736jnlnmbmec"",""Address"":""f015748"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f015748.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceagymfhc5tv4ulcthjdtnla7z54tpprqhdbtflzvjsixe3ehkfzek"",""Address"":""f015771"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f015771.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacearowuwpzp7zjbfzzxfadhlygozwpkd7d7oc7x3rw76dmuzbzr32e"",""Address"":""f04688"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04688.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacec6xmpyail2rlpy5vis6j3l7dnnwt2bdar2nnwrlys43wqebppj66"",""Address"":""f07594"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07594.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceavmvddz2e2nwoja36nqgmaewx7jgirdcwsiefwgxowjvnxyhfaug"",""Address"":""f07467"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07467.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb5d56vudzrpcaf7pgi2wacfo4wacg6lokyce55qokaxt7ww6k43k"",""Address"":""f012067"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012067.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea2k6pvhwp6b6qubzd2irseteyepcp2vw7gfcfcqtqhwqibrz65ge"",""Address"":""f06716"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06716.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebrkycr6iso7ujfmg3xc4qpjghzri7ah57sce7gqpq3eeojaht4li"",""Address"":""f06963"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06963.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f06340"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06340.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacectsnumtiverwj7owv4cme5wlfmw5bmeg5f23ulsqeyy3htwk5vp6"",""Address"":""f010224"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f010224.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedpgyirsxcvo7tbbasttkssmpphvtnunvuc7kngf6aogxxjrpd5v2"",""Address"":""f012509"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f012509.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebzhytzk2v2x4q3gx5yxi2xkiiqcv2cxobvv4nygwrwsz4o27t2ew"",""Address"":""f013057"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f013057.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f04676"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04676.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecbfijzx7bqhl6yutgw6f3pryfrmgkryugeuwfa7vuio7bba65dmi"",""Address"":""f04155"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f04155.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedct2ex4tpioukkbrkwc6c7lnkrvnlullz4yyytg7befolvyoq5u2"",""Address"":""f04903"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04903.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaszfxuxlipsi7qlmbtimsdekei6stgxl5gcma3dzx3tfg4jfcmqy"",""Address"":""f04588"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04588.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea4pwqlr32w4mvo56dkwnpmozlzn3t4sn4yds3mzoaqg3wertd2sg"",""Address"":""f014419"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f014419.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceadzscpxothvbyw5r65q7ih63jp3mrjl67truiv72atuehx6r6sro"",""Address"":""f01652"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01652.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedtgbbumq7y4o5vefp7pjke23l7bfu5akyfv7znffzry3vchbebpq"",""Address"":""f011712"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f011712.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceck4zeazi3f22spyd2trzqov3rbe6ltedpadjvvexaxpimq7tyupc"",""Address"":""f05681"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05681.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedgqzrqcutottoas4f3wnwqoxs52wrnj625xkn75seizuuy6wxlpo"",""Address"":""f04257"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f04257.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebik77jtdpzbreekpl6w3mpozed7poila5su5vusa3w5aaakkb2io"",""Address"":""f05374"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05374.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebcecqsgmq4hcybquaoskottlgrghe7nhnfe45cbqojhhrkebyaau"",""Address"":""f07625"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07625.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecj5jx6ft77ycxhlp2jayys35ouuyohyui3n6e2men6v32fvoyyes"",""Address"":""f05131"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05131.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecwgrhywq4h2dqmkxfcbxpt4v3d6lmlpfvcipzbw3wmnsztmlblhk"",""Address"":""f06393"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06393.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedeahyjfo4cb6puhvasijej4bzlkx2b3c5rovhzyhmj42yvnqez7m"",""Address"":""f08993"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f08993.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaksxc4fdbhmi2qbxj7eslfjzut5wwyurpjwqbg5lmoqoszrsprm2"",""Address"":""f012588"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012588.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceac5uhbv4h5wpc5ozyjhklusau2h64wljibwqmtv6ckcot5mk3jg2"",""Address"":""f014561"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014561.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceavizu74t5bsazc4rk4tne2al5xsn3i4hb5aop4jgy5jby3zlc26s"",""Address"":""f014749"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014749.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceas5567stmr7v4kgqpre6a5izgw744u3zt433hqcu7xdiqvcn3meo"",""Address"":""f012445"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012445.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebc35qfrxz22e6doc7awjlxpmbbmq3nqi22evlp3446qtpqpkj7ws"",""Address"":""f016519"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f016519.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedd535k7kwm7kfg3y6c32fdhwb4zuf7ur55jt44dxx6ozfu753rhe"",""Address"":""f012022"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012022.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceause7762ca2nbs4xkhcy6zql63qbjhaovmfx3jndgnmy2netwhbg"",""Address"":""f07705"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07705.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",

There is also a bunch of empty files:

-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_fee_debts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_infos.csv
-rw-r--r-- 1 root  root  449K Jan 24 16:28 arch0124-2020-09-28-miner_locked_funds.csv
-rw-r--r-- 1 root  root  1.6M Jan 24 16:28 arch0124-2020-09-28-miner_pre_commit_infos.csv
-rw-r--r-- 1 root  root  6.9K Jan 24 16:28 arch0124-2020-09-28-miner_sector_deals.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_sector_events.csv
-rw-r--r-- 1 root  root  1.8M Jan 24 16:28 arch0124-2020-09-28-miner_sector_infos.csv
-rw-r--r-- 1 root  root   19M Jan 24 16:28 arch0124-2020-09-28-miner_sector_posts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-multisig_approvals.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-multisig_transactions.csv
-rw-r--r-- 1 root  root   26M Jan 24 16:28 arch0124-2020-09-28-parsed_messages.csv
-rw-r--r-- 1 root  root  250K Jan 24 16:28 arch0124-2020-09-28-power_actor_claims.csv
-rw-r--r-- 1 root  root  2.5M Jan 24 16:28 arch0124-2020-09-28-receipts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-verified_registry_verified_clients.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-verified_registry_verifiers.csv

Finally, in lily.log, we can see some websocket: close 1000 (normal) errors.

{"level":"error","ts":"2023-01-24T15:27:57.608Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":100904,"actor":"fil/1/storageminer","address":"f02303","code":"fil/1/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f02303.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100904: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"debug","ts":"2023-01-24T15:27:57.798Z","logger":"lily/tasks/miner","caller":"miner/sector_events.go:109","msg":"diff sectors","miner":"f02731","duration":3.888790994}
{"level":"error","ts":"2023-01-24T15:27:57.798Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":100904,"actor":"fil/1/storageminer","address":"f02731","code":"fil/1/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f02731.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100904: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"debug","ts":"2023-01-24T15:27:57.808Z","logger":"rpc","caller":"go-jsonrpc@v0.1.8/websocket.go:624","msg":"websocket error","error":"websocket: close 1000 (normal)"}
{"level":"debug","ts":"2023-01-24T15:27:57.999Z","logger":"watchdog","caller":"go-watchdog@v1.3.0/watermarks.go:32","msg":"watermark policy: evaluating; utilization: 10881658880/33573044224 (used/limit)"}

That said, the walk seems to continue without problems after these websocket errors.

@davidgasquez
Copy link
Copy Markdown

davidgasquez commented Jan 25, 2023

Tested things locally and seems its working fine. I've got some errors but I think they're unrelated with this PR:

$ cat lily.log | grep "error" | jq .
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebarb7znoskcwvl6up2avtcu44keoi2c7cfiat7knyvgwqy3iahak"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecn3bzbox5h4x3viqwtgwvsz6glfmgm4tkxq2bvqdgddzzu3gx3ws"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebdm3jeqsoa74de7qthlxvcjjkvp62yxri7auv7ggwyemfllwntgi"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebrgzpswiwdqs5wvmw6zfrsvrdgeqpeozsbkhwuwgsswcfsdrdxto"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacea7lqx55xxzdsp7use33hgeeh33tyry4m6zg3dsin6lznopk4yq3w"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebby6ag7qk743wti5bp6azdcffptrw5ws3u5gwa5c6kiepjt5ayqs"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecifg623mszaimdfbooxpj54etykqjy7sjec3xd2ycoflqn5akke2"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacea4624zeqqzdrdwoe33ejxxz32zq5fr5ysmbuu3vofskgmegkx3qc"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceaoez3byuzqkblaa2gfj4zi2ubh74zj7m7jipkatbpjzoxdofmqvu"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebmd7l7vevx56qkh2ehnugkwtmkysf4ryx3si4xwknanneqiybmhu"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceabtnahyz77s6kzhccls6mjfv43y4snibrn4zrbhfirltwvh6wa3u"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacedjqujbo32cqfkdw3zvhqlswo4lhom4b5v6jnvd54oy3aps5jznle"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacea5wktkeap5ur4czmrbx4xpwp4edfihrxm2n25adnclo3v6nakl4g"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceddzlla5vs4jxknncyoexprtm2hcn3xz37mrkfzjxtc2yiiepmd66"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebnisrg7tjtzd5rwnakvwabhjg7j7m4rgoseae3fopl3cjjsgxbvk"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecqx2oxwfbq2iwmkk4l2xvc6kajc6lndly57b6p7vxkojhzrcwgfa"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacechwhzxskk5cw4qvbaxcyolku6tu6q6prag4kqzlccbzmu7rmvkci"}
{"level":"error","ts":"2023-01-25T08:53:19.391Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceciazen2ggajc573g256ill54lk2jf6b6o6jvig2ezxbncseord22"}
{"level":"error","ts":"2023-01-25T08:53:19.391Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceb4v2dpsfp7skdoscjdcfzaxc5isxtz43doan4xowqrkmurcwpm44"}
{"level":"error","ts":"2023-01-25T08:53:19.391Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacediy2l4ycnctl6esyuwtid5kjdg42k77c5uunxw52nksxqooumue2"}
{"level":"error","ts":"2023-01-25T08:53:19.392Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacediy2l4ycnctl6esyuwtid5kjdg42k77c5uunxw52nksxqooumue2"}
{"level":"error","ts":"2023-01-25T08:53:19.393Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebwmgedad75xmpjbpdfkjndeppcoed3b6rjmmyta7dnkaet6g5732"}
{"level":"error","ts":"2023-01-25T08:53:19.393Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecm5axfyvqzehha6sp3sme6ripsgo7t6bchgt3ulybai3vz4afsik"}
{"level":"error","ts":"2023-01-25T08:53:19.393Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecrfir5yaawy7sdgmksusi7woxhheqmtej7x7hiz3e5mxccwk3j7u"}

Will send some jobs and keep an eye on things.

@davidgasquez
Copy link
Copy Markdown

davidgasquez commented Jan 25, 2023

Heads up that I just got the first websocket issue. Seems it might be still happening. I can't dig much for now though. Will leave a proper report when I'm back next week.

@hsanjuan
Copy link
Copy Markdown
Contributor

hsanjuan commented Jan 26, 2023

@frrist do you have any ideas about why it might be panicking in the first place?

@frrist
Copy link
Copy Markdown
Member Author

frrist commented Jan 26, 2023

@hsanjuan current theory is filecoin-project/filet#22 (comment)

@hsanjuan
Copy link
Copy Markdown
Contributor

hsanjuan commented Jan 26, 2023

The "maybe there are blocks missing in a datastore" doesn't fit the randomness that seems to affect this issue.

@frrist
Copy link
Copy Markdown
Member Author

frrist commented Feb 3, 2023

@davidgasquez can you point me to the github issue for the websocket bug?

@davidgasquez
Copy link
Copy Markdown

davidgasquez commented Feb 3, 2023

@davidgasquez can you point me to the github issue for the websocket bug?

There is one higher level in Filet and another in Lotus.

@birdychang
Copy link
Copy Markdown
Contributor

birdychang commented Mar 27, 2023

@frrist should we merge?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@birdychang birdychang birdychang approved these changes

@davidgasquez davidgasquez Awaiting requested review from davidgasquez

@kasteph kasteph Awaiting requested review from kasteph

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@frrist @davidgasquez @hsanjuan @birdychang