Large dataset file upload (≥100 GB) hangs with ERR_CONNECTION_TIMED_OUT and shows "Upload failed. Please retry."

[eugenegujing]

What happened?

After raising dataset.single_file_upload_max_size_mib to 204800 (200 GiB) via the Admin Settings page so that the frontend size check accepts large files, uploading a multi-100GB .h5 file to a dataset fails.

The dataset page only shows the generic toast: Upload failed. Please retry.

The browser console shows that the multipart-upload requests for many individual parts time out at the TCP level (net::ERR_CONNECTION_TIMED_OUT), e.g.:

  :4200/api/dataset/multipart-upload?...&partNumber=6     net::ERR_CONNECTION_TIMED_OUT
  :4200/api/dataset/multipart-upload?...&partNumber=90    net::ERR_CONNECTION_TIMED_OUT
  :4200/api/dataset/multipart-upload?...&partNumber=115   net::ERR_CONNECTION_TIMED_OUT
  :4200/api/dataset/multipart-upload?...&partNumber=160   net::ERR_CONNECTION_TIMED_OUT

How to reproduce?

1. Run Texera locally (frontend on :4200, file-service backend).
2. As an admin, open Admin → Settings, and set:
   - Max File Size (MiB): 204800 (i.e. 200 GiB)
   - Part Size (MiB): 50 (default)
   Save.
3. Create a new dataset, open the dataset page, click "Browser & Upload Files" (or drag-and-drop).
4. Select a single ≥100 GB file (in my case a 179 GB .h5 file).
5. Observe:
   • Browser DevTools → Network shows many requests to /api/dataset/multipart-upload?...&partNumber=N failing with net::ERR_CONNECTION_TIMED_OUT.
   • The dataset page shows only the toast: Upload failed. Please retry.
   • No new version is created.

Branch

main

Commit Hash (Optional)

No response

What browsers are you seeing the problem on?

No response

Relevant log output

[xuang7]

Thanks for the report. I tried to follow your steps to reproduce, but in my local environment, uploading a 100GB file succeeded. [Edited: my local test used a sparse file, so it may not fully reproduce the same client-side disk I/O behavior as uploading a real 100GB+ data file.] Could you provide a bit more information to help us narrow this down?

1. Does the failure only happen with 100GB+ files, or does it also happen with smaller files?
   Could you try uploading 1GB, 10GB, and 50GB files to the same dataset and let us know whether each one succeeds?

2. Could you also try a few different upload configurations, such as different part sizes, and see whether the issue still occurs?

3. For the timeout errors, do all multipart-upload requests eventually time out, or only some specific parts?

4. Around the time of the failure, could you share the relevant lines from file-service/log/file-service.log, especially anything mentioning multipart-upload, uploadPart, exceptions, or idleTimeout?

Thanks!

[Yicong-Huang]

I did not know we can support or are supporting 200GB files! that's great.

My two cents:

1. does lakefs have this kind of issues reported? if it is a bug from their side, we should not be the first to encounter it.
2. as @xuang7 cannot reproduce it, it might be related to OS/env. For instance, some OS have limited on file IO system calls, and if the 100GB file is cut into small chunks, it may have different threshold depending on the OS.

[eugenegujing]

Thank you @Yicong-Huang and @xuang7 for help! Xuan's Problem 2 really gives me some hints for this problem. Here is my update to this problem:

Uploading the 179 GB .h5 file finally succeeded only after I raised Part Size to 5120 MiB (~36 parts total). 50 / 100 / 200 / 300 / 400 / 500 / 1000 MiB all failed mid-upload.

This is my explanation to the problem. It might be incorrect and welcome to discuss:

It's a statistical failure tied to part count, not a hard "100 GB" threshold. Each XHR for a part has some small chance of hitting a transient timeout. Crucially, the frontend has no per-part retry — in dataset.service.ts, a single failed XHR makes the surrounding mergeMap abort the whole upload. So a 0.5% per-part failure rate for a 179GB file means:

• 50 MiB parts → 3666 parts → ~10⁻⁸ chance of full success → ~always fails
• 1000 MiB parts → 184 parts → ~40% chance of success → sometimes works
• 5120 MiB parts → 36 parts → ~83% chance of success → works for me

Re @xuang7's question 1: it should fail less often with smaller files purely because there are fewer parts. I'd guess 1 GB and 10 GB succeed reliably at 50 MiB, 50 GB starts being flaky, 100 GB+ almost always fails.

Re @xuang7's question 3: from the original screenshot, only specific parts time out (partNumber=6, 90, 115, 160), not all of them.

Re @xuang7's reproduction with a sparse file: that probably hides the issue, because sparse-file reads are nearly free , so the client feeds chunks at a perfectly smooth rate. With a real .h5 file there's actual disk I/O jitter, which makes some XHRs slower and more likely to hit one of the idle timeouts above.

Re @Yicong-Huang's points: I don't think this is a LakeFS bug. LakeFS only sets up the upload session and finalizes the merge, so per-part timeouts can't originate there. And I think OS/env matters: client-side I/O jitter raises per-part failure probability.

Suggested fixes:

1. Add per-part retry with backoff in dataset.service.ts around the inner observable — this alone would let 50 MiB defaults work reliably even for 200 GB.
2. Dynamic part size based on file size. In dataset-detail.component.ts, treat the admin-configured part size as a floor and auto-scale it up so the part count stays bounded (e.g. ≤ 200 parts)