Skip to content

fix(checkpoint): FTW truncated shard raises OSError instead of silent garbage load - #25

Merged
jason-fxz merged 2 commits into
FlashML-org:mainfrom
RuixiangMa:fix-ftw-silent-corruption
Aug 24, 2026
Merged

fix(checkpoint): FTW truncated shard raises OSError instead of silent garbage load#25
jason-fxz merged 2 commits into
FlashML-org:mainfrom
RuixiangMa:fix-ftw-silent-corruption

Conversation

@RuixiangMa

@RuixiangMa RuixiangMa commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Problem

_pread_into in ftw.py silently broke out of its read loop on EOF (os.preadv returning 0), leaving the destination buffer partially filled with uninitialized garbage. A truncated FTW shard file (from interrupted conversion, disk full, or corruption) would load partial weights with no error anywhere, producing silently corrupted model outputs.

Fix

  • O_DIRECT path (_pread_into): break → raise OSError with bytes read/expected and offset.
  • mmap fallback (FTWReader.read_into): added length check before slice assignment with shard file name diagnostics. Previously raised a cryptic ValueError.
  • Unified exception type to OSError across both paths.
  • Updated docstring to reflect current behavior.

cc @andy-yang-1 @jason-fxz

@gdevenyi

Copy link
Copy Markdown

Triaged while assembling a merged deployment branch for a 2× RTX 6000 Ada / 2× Xeon Gold 6526Y Linux box serving DeepSeek-V4-Flash with offloaded experts, to benchmark the open PRs together.

FTW truncated-shard error handling. Not merged: this deployment serves HF safetensors directly and never exercises the FTW reader, so I would have been testing nothing. The change reads correctly — turning a bare OSError into a diagnosable message is the right instinct, and it is the same class of problem as the pin failure in #56.

Flagging only so the absence of a report from me is not read as a problem found — I merged and benchmarked #30, #48, #56, #69, #70, #71 and #81, and left this one out deliberately.

@RuixiangMa RuixiangMa changed the title [Checkpoint][Bugfix]: FTW truncated shard raises OSError instead of silent garbage load fix(checkpoint): FTW truncated shard raises OSError instead of silent garbage load Aug 23, 2026
@jason-fxz

Copy link
Copy Markdown
Collaborator

ftw.py looks good! Just drop the tests and ready to merge.

@RuixiangMa

Copy link
Copy Markdown
Contributor Author

ftw.py looks good! Just drop the tests and ready to merge.

Done

@jason-fxz
jason-fxz merged commit bd372b6 into FlashML-org:main Aug 24, 2026
@andy-yang-1

Copy link
Copy Markdown
Collaborator

@RuixiangMa Great job!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants