fix(checkpoint): FTW truncated shard raises OSError instead of silent garbage load - #25
Conversation
…ilent garbage load
|
Triaged while assembling a merged deployment branch for a 2× RTX 6000 Ada / 2× Xeon Gold 6526Y Linux box serving DeepSeek-V4-Flash with offloaded experts, to benchmark the open PRs together. FTW truncated-shard error handling. Not merged: this deployment serves HF safetensors directly and never exercises the FTW reader, so I would have been testing nothing. The change reads correctly — turning a bare OSError into a diagnosable message is the right instinct, and it is the same class of problem as the pin failure in #56. Flagging only so the absence of a report from me is not read as a problem found — I merged and benchmarked #30, #48, #56, #69, #70, #71 and #81, and left this one out deliberately. |
|
ftw.py looks good! Just drop the tests and ready to merge. |
Done |
|
@RuixiangMa Great job! |
Problem
_pread_into in ftw.py silently broke out of its read loop on EOF (os.preadv returning 0), leaving the destination buffer partially filled with uninitialized garbage. A truncated FTW shard file (from interrupted conversion, disk full, or corruption) would load partial weights with no error anywhere, producing silently corrupted model outputs.
Fix
cc @andy-yang-1 @jason-fxz