Skip to content

fix(utils): preserve streamed download bytes - #9857

Open
RerankerGuo wants to merge 1 commit into
modelscope:mainfrom
RerankerGuo:fix/preserve-streamed-download-bytes
Open

fix(utils): preserve streamed download bytes#9857
RerankerGuo wants to merge 1 commit into
modelscope:mainfrom
RerankerGuo:fix/preserve-streamed-download-bytes

Conversation

@RerankerGuo

Copy link
Copy Markdown
Contributor

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

download_ms_file streamed responses through iter_lines(). That API removes
line delimiters, so downloaded binary or text artifacts containing newline
bytes were silently changed on disk. Both streamed download helpers also left
their HTTP responses open after completion or failure.

This PR:

  • streams ModelScope files with iter_content() to preserve the response bytes;
  • validates the HTTP status before writing;
  • closes responses reliably through their context manager;
  • adds CPU-only regression coverage for byte preservation and response cleanup.

The change is limited to the two download helpers in swift.utils.hub_utils.
It does not change cache paths or retry behavior.

Experiment results

Before the fix:

expected: b'first line\n\x00second line\r\n'
actual:   b'first line\x00second line'
2 tests failed

After the fix:

.venv/bin/python tests/run.py --pattern test_hub_utils.py
SUCCESS (Runs=2, success=2)

.venv/bin/pre-commit run --all-files
All hooks passed

Use raw response chunks for ModelScope file downloads so newline and binary bytes are not discarded. Close streamed responses reliably and validate HTTP status for both download helpers.

Test: .venv/bin/python tests/run.py --pattern test_hub_utils.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant