When recovery is enabled, a transient error fetching a full tile causes the tiled worker to discard its current in-memory progress. After the five-second retry delay, the worker creates a new StaticCTClient using the worker’s original startup index.
This can make a tiled log replay millions of already-processed entries repeatedly, flooding WebSocket clients with duplicates and consuming substantial CPU and bandwidth.
Version
Observed on based on master commit d2cb099.
Recovery was enabled.
Observed behavior
A public tiled log occasionally returned HTTP 404 for a full tile near the current head:
Error processing tiled log updates for 'https://luoshu2027.trustasia.com/luoshu2027':
processing tile 259211: fetching tile: request failed: unexpected status code 404
Worker for 'https://luoshu2027.trustasia.com/luoshu2027'
sleeping for 5 seconds due to error
Restarting worker for 'https://luoshu2027.trustasia.com/luoshu2027'
Each cycle eventually reached the head, received another transient 404, and restarted from the same old index.
Restarting the entire certstream process loaded a newer persisted recovery index and reduced the replay window to approximately 100,000 entries, but it did not fix the problem. The same retry loop continued every 20–35 seconds.
Suspected cause
The outer worker retry loop calls runTiledWorker() again after an error:
ct-watcher.go lines 373–413
Each invocation creates a new StaticCTClient using w.ctIndex:
ct-watcher.go lines 469–489
During processing, progress is stored in StaticCTClient.ctIndex:
ct-tiled.go lines 318–350
However, when Monitor() returns an error, that client is discarded:
ct-tiled.go lines 250–257
The updated client index is never copied back to w.ctIndex. Although the metrics/recovery state is updated while entries are broadcast, the outer retry path does not reload it before constructing the replacement client.
The effective flow is:
w.ctIndex = X
client := NewStaticCTClient(..., X)
client.ctIndex advances from X to Y
tile fetch fails
client is discarded
sleep 5 seconds
client := NewStaticCTClient(..., X)
entries X through Y are emitted again
Suggested fix
The simplest fix may be to handle transient fetch errors inside StaticCTClient.Monitor() and retain the same client instance and its ctIndex:
- Log the error.
- Apply the existing backoff.
- Continue monitoring with the same StaticCTClient.
- Return only for cancellation or a genuinely fatal error.
Alternatively, copy staticCTClient.ctIndex back to the worker before returning from runTiledWorker(), or initialize each retry from the latest in-memory recovery index rather than the worker’s original index.
Related issue
This is related to, but distinct from #104: Excessive partial tile fetching for Static CT logs. That issue concerns polling and partial-tile request volume; this report concerns loss of in-memory progress after a full-tile or network error.
When recovery is enabled, a transient error fetching a full tile causes the tiled worker to discard its current in-memory progress. After the five-second retry delay, the worker creates a new StaticCTClient using the worker’s original startup index.
This can make a tiled log replay millions of already-processed entries repeatedly, flooding WebSocket clients with duplicates and consuming substantial CPU and bandwidth.
Version
Observed on based on master commit d2cb099.
Recovery was enabled.
Observed behavior
A public tiled log occasionally returned HTTP 404 for a full tile near the current head:
Each cycle eventually reached the head, received another transient 404, and restarted from the same old index.
Restarting the entire certstream process loaded a newer persisted recovery index and reduced the replay window to approximately 100,000 entries, but it did not fix the problem. The same retry loop continued every 20–35 seconds.
Suspected cause
The outer worker retry loop calls runTiledWorker() again after an error:
ct-watcher.go lines 373–413
Each invocation creates a new StaticCTClient using w.ctIndex:
ct-watcher.go lines 469–489
During processing, progress is stored in StaticCTClient.ctIndex:
ct-tiled.go lines 318–350
However, when Monitor() returns an error, that client is discarded:
ct-tiled.go lines 250–257
The updated client index is never copied back to w.ctIndex. Although the metrics/recovery state is updated while entries are broadcast, the outer retry path does not reload it before constructing the replacement client.
The effective flow is:
Suggested fix
The simplest fix may be to handle transient fetch errors inside StaticCTClient.Monitor() and retain the same client instance and its ctIndex:
Alternatively, copy staticCTClient.ctIndex back to the worker before returning from runTiledWorker(), or initialize each retry from the latest in-memory recovery index rather than the worker’s original index.
Related issue
This is related to, but distinct from #104: Excessive partial tile fetching for Static CT logs. That issue concerns polling and partial-tile request volume; this report concerns loss of in-memory progress after a full-tile or network error.