From 2ba064c50d649bba9898030e2b351d6ef2e46ab7 Mon Sep 17 00:00:00 2001 From: Josh Date: Mon, 3 Aug 2026 09:25:12 -0400 Subject: [PATCH 1/3] docs(webdav): clarify bulk upload protocol Document capability negotiation, multipart framing, checksum and mtime header compatibility, per-file response fields, and error handling. Signed-off-by: Josh --- .../client_apis/WebDAV/bulkupload.rst | 128 +++++++++++++++--- 1 file changed, 107 insertions(+), 21 deletions(-) diff --git a/developer_manual/client_apis/WebDAV/bulkupload.rst b/developer_manual/client_apis/WebDAV/bulkupload.rst index 62a8b37b9c4..63492a165cd 100644 --- a/developer_manual/client_apis/WebDAV/bulkupload.rst +++ b/developer_manual/client_apis/WebDAV/bulkupload.rst @@ -1,11 +1,12 @@ -=================== +================ File bulk upload -=================== +================ .. sectionauthor:: Matthieu Gallien Introduction ------------ + Uploading many small files is often slower than what could be achieved because we do not use the whole network bandwidth. Nextcloud has a bulk upload API where you can upload many small files together in order to optimize the use of network bandwidth. @@ -13,36 +14,120 @@ many small files together in order to optimize the use of network bandwidth. Usage ----- -The API is only available for registered users of your instance. And uses the path: -``/remote.php/dav/bulk``. +The API is available to authenticated users at: + +.. code-block:: HTTP + + POST /remote.php/dav/bulk + +Before using this endpoint, clients should check the capabilities response for +``dav.bulkupload`` version ``1.0``. Administrators can disable the feature, so +clients must fall back to regular uploads when this capability is absent. + +The request +^^^^^^^^^^^ + +A bulk upload is an HTTP ``POST`` request with an outer request header of: + +.. code-block:: HTTP + + Content-Type: multipart/related; boundary= + +The server requires the ``multipart/related`` media type and the +``boundary`` parameter. Multipart boundaries, part headers, and the +header/body separator must use CRLF (``\r\n``). + +Each part starts with ``--\r\n``. After its headers, send an empty +line (``\r\n``), followed by exactly the number of payload bytes declared in +that part's ``Content-Length`` header, then ``\r\n``. Terminate the request +with ``----\r\n``. + +The ``Content-Length`` is the size in bytes of the file payload only. It +does not include multipart boundaries, part headers, or CRLF delimiters. The +server reads exactly the specified number of bytes, so file payloads may contain +newlines and arbitrary binary data. + +Each file is sent as one part. + +Each file part needs the following headers: + +* ``Content-Length: `` +* ``X-File-Path: `` +* A checksum header; one or both of: + + * ``OC-Checksum: :`` + * ``X-File-MD5: `` + +The following per-part headers are optional: -Starting a bulk upload -^^^^^^^^^^^^^^^^^^^^^^^^^ +* ``Content-Type: `` +* ``X-OC-Mtime: `` -A bulk upload is simply using a request structured as HTTP multipart with related mime type. +The destination file path is within the authenticated user's files root. +All intermediate folders must already exist. -Each file is then sent as one HTTP part. +At least one checksum header is required. ``OC-Checksum`` is the preferred +checksum header. For example: -Each file inside an HTTP Part will need the following headers: +.. code-block:: HTTP -* Content-Length: -* Content-Type: -* X-File-MD5: -* X-File-Mtime: -* X-File-Path: + OC-Checksum: SHA1:3b5d5c3712955042212316173ccf37be800c4f6e -The reply is a json document with the following structure: +If compatibility with Nextcloud Server versions before 32 is required, clients +should also send the legacy ``X-File-MD5`` header: + +* ``X-File-MD5: `` + +When both checksum headers are present, the server uses the checksum specified +in the ``OC-Checksum`` header for validation and ignore any provided +``X-File-MD5`` header value. + +The server also accepts ``X-File-Mtime`` as an alternative to +``X-OC-Mtime``. When both headers are supplied, ``X-File-Mtime`` takes +precedence. The desktop client currently sends ``Content-Type: +application/octet-stream`` and ``X-File-Mtime``. + +The example below uses ``OC-Checksum`` only. Clients that need compatibility +with Nextcloud Server versions before 32 must also calculate and send +``X-File-MD5`` for every file part. + +The response +^^^^^^^^^^^^ + +The reply is a JSON document keyed by the ``X-File-Path`` value. A successful +file upload has the following structure: .. code-block:: JSON { "/small file.txt": { "error": false, - "etag": "adb9aa24cbfa8e372c88431d1d99629a" + "etag": "adb9aa24cbfa8e372c88431d1d99629a", + "fileid": "123", + "permissions": "RGDNVCK" } } -Example of code to upload some test files with bulk upload protocol +For a file-level error, the corresponding entry contains ``"error": true`` +and an error message: + +.. code-block:: JSON + + { + "/unwritable/file.txt": { + "error": true, + "message": "..." + } + } + +The request can return HTTP ``200 OK`` even if one or more individual file +uploads failed. Clients must inspect the response entry for every submitted +file. Malformed multipart requests return HTTP ``400 Bad Request``. + +Example +------- + +Example code to upload some test files using the bulk upload protocol using *curl*: .. code-block:: BASH @@ -65,19 +150,20 @@ Example of code to upload some test files with bulk upload protocol file_remote_path="$REMOTE_FOLDER/$file_name.txt" head -c "$SIZE" /dev/urandom > "$file_local_path" file_mtime=$(stat -c %Y "$file_local_path") - file_hash=$(md5sum "$file_local_path" | awk '{ print $1 }') + file_hash=$(sha1sum "$file_local_path" | awk '{ print $1 }') file_size=$(du -sb "$file_local_path" | awk '{ print $1 }') { echo -en "--$BOUNDARY\r\n" echo -en "X-File-Path: $file_remote_path\r\n" echo -en "X-OC-Mtime: $file_mtime\r\n" - echo -en "X-File-Md5: $file_hash\r\n" + echo -en "Content-Type: application/octet-stream\r\n" + echo -en "OC-Checksum: SHA1:$file_hash\r\n" echo -en "Content-Length: $file_size\r\n" - echo -en "\r\n" >> "$UPLOAD_PATH" + echo -en "\r\n" cat "$file_local_path" - echo -en "\r\n" >> "$UPLOAD_PATH" + echo -en "\r\n" } >> "$UPLOAD_PATH" done From f376b59aac88fdad6595588407afe37c422b33a0 Mon Sep 17 00:00:00 2001 From: Josh Date: Mon, 3 Aug 2026 10:25:22 -0400 Subject: [PATCH 2/3] docs(webdav): improve bulk upload protocol docs Clarify capability discovery, multipart request framing, checksum and mtime compatibility, response handling, retry considerations, and update GNU/Linux curl example. Adjust headings and update introduction for additional clarity. Signed-off-by: Josh --- .../client_apis/WebDAV/bulkupload.rst | 196 +++++++++++------- 1 file changed, 123 insertions(+), 73 deletions(-) diff --git a/developer_manual/client_apis/WebDAV/bulkupload.rst b/developer_manual/client_apis/WebDAV/bulkupload.rst index 63492a165cd..087593e2f8c 100644 --- a/developer_manual/client_apis/WebDAV/bulkupload.rst +++ b/developer_manual/client_apis/WebDAV/bulkupload.rst @@ -1,18 +1,25 @@ -================ -File bulk upload -================ +=========== +Bulk upload +=========== .. sectionauthor:: Matthieu Gallien Introduction ------------ -Uploading many small files is often slower than what could be achieved because we do not -use the whole network bandwidth. Nextcloud has a bulk upload API where you can upload -many small files together in order to optimize the use of network bandwidth. +Uploading many small files individually adds request overhead and can reduce effective +throughput. The bulk upload API lets a client upload multiple files in one HTTP request, +reducing that overhead and improving overall throughput. -Usage ------ +The API is intended for workloads containing many independent files whose individual +payloads are small relative to the overhead of separate HTTP requests. It does not define +a fixed per-file size threshold or a minimum number of files. Clients should choose a +batching policy appropriate for their environment, limit the total multipart request size +to the limits of the Nextcloud instance and any reverse proxy, and use chunked upload for +large individual files. + +Endpoint and capability discovery +--------------------------------- The API is available to authenticated users at: @@ -20,14 +27,16 @@ The API is available to authenticated users at: POST /remote.php/dav/bulk -Before using this endpoint, clients should check the capabilities response for -``dav.bulkupload`` version ``1.0``. Administrators can disable the feature, so -clients must fall back to regular uploads when this capability is absent. +Before using this endpoint, clients should check for ``dav.bulkupload`` version +``1.0`` in the capabilities response. The feature can be disabled by +administrators, so clients must fall back to regular uploads when the capability +is absent. -The request -^^^^^^^^^^^ +Request format +^^^^^^^^^^^^^^ -A bulk upload is an HTTP ``POST`` request with an outer request header of: +A bulk upload is an HTTP ``POST`` request whose outer ``Content-Type`` header +is: .. code-block:: HTTP @@ -42,7 +51,7 @@ line (``\r\n``), followed by exactly the number of payload bytes declared in that part's ``Content-Length`` header, then ``\r\n``. Terminate the request with ``----\r\n``. -The ``Content-Length`` is the size in bytes of the file payload only. It +Each part's ``Content-Length`` is the size in bytes of the file payload only. It does not include multipart boundaries, part headers, or CRLF delimiters. The server reads exactly the specified number of bytes, so file payloads may contain newlines and arbitrary binary data. @@ -74,13 +83,13 @@ checksum header. For example: OC-Checksum: SHA1:3b5d5c3712955042212316173ccf37be800c4f6e If compatibility with Nextcloud Server versions before 32 is required, clients -should also send the legacy ``X-File-MD5`` header: +should also send the ``X-File-MD5`` header: * ``X-File-MD5: `` -When both checksum headers are present, the server uses the checksum specified -in the ``OC-Checksum`` header for validation and ignore any provided -``X-File-MD5`` header value. +On Nextcloud Server 32 and later, when both checksum headers are present, the +server uses the checksum specified in the ``OC-Checksum`` header for validation +and ignores any provided ``X-File-MD5`` header value. The server also accepts ``X-File-Mtime`` as an alternative to ``X-OC-Mtime``. When both headers are supplied, ``X-File-Mtime`` takes @@ -91,8 +100,8 @@ The example below uses ``OC-Checksum`` only. Clients that need compatibility with Nextcloud Server versions before 32 must also calculate and send ``X-File-MD5`` for every file part. -The response -^^^^^^^^^^^^ +Response format +^^^^^^^^^^^^^^^ The reply is a JSON document keyed by the ``X-File-Path`` value. A successful file upload has the following structure: @@ -124,64 +133,105 @@ The request can return HTTP ``200 OK`` even if one or more individual file uploads failed. Clients must inspect the response entry for every submitted file. Malformed multipart requests return HTTP ``400 Bad Request``. -Example -------- +Retrying a failed bulk-upload request may require retransmitting +all files in that request. -Example code to upload some test files using the bulk upload protocol using *curl*: +GNU/Linux Bash + curl example +----------------------------- -.. code-block:: BASH +The following Bash script example targets GNU/Linux systems with ``curl`` +7.76.0 or newer. Older curl versions can replace ``--fail-with-body`` with +``--fail`` with approximate behavior. The script also requires ``openssl`` and +GNU core utilities. It uploads some test files using the bulk upload protocol. - #!/bin/bash +This example uses the modern ``OC-Checksum`` header and assumes that bulk +upload is enabled. It does not work with Nextcloud Server versions before 32 +as-is, but can be easily modified to do so. Production clients should confirm +that ``dav.bulkupload`` is advertised via the capabilities API before +attempting a bulk upload transaction. - NB=$1 - SIZE=$2 +The server can return HTTP ``200 OK`` while individual parts (files) generate +error response entries. The script displays but does not validate the per-file +JSON results. Production clients must parse the JSON response and inspect the +``error`` value for every submitted destination path. - USER="admin" - PASS="admin" - SERVER="nextcloud.local" - UPLOAD_PATH="/tmp/bulk_upload_request_$(openssl rand --hex 8).txt" - BOUNDARY="boundary_$(openssl rand --hex 8)" - REMOTE_FOLDER="/test" - - for ((i=1; i<="$NB"; i++)) - do - file_name=$(openssl rand --hex 8) - file_local_path="./$file_name.txt" - file_remote_path="$REMOTE_FOLDER/$file_name.txt" - head -c "$SIZE" /dev/urandom > "$file_local_path" - file_mtime=$(stat -c %Y "$file_local_path") - file_hash=$(sha1sum "$file_local_path" | awk '{ print $1 }') - file_size=$(du -sb "$file_local_path" | awk '{ print $1 }') - - { - echo -en "--$BOUNDARY\r\n" - echo -en "X-File-Path: $file_remote_path\r\n" - echo -en "X-OC-Mtime: $file_mtime\r\n" - echo -en "Content-Type: application/octet-stream\r\n" - echo -en "OC-Checksum: SHA1:$file_hash\r\n" - echo -en "Content-Length: $file_size\r\n" - echo -en "\r\n" - - cat "$file_local_path" - echo -en "\r\n" - } >> "$UPLOAD_PATH" - done +The number of test files and bytes per test file should be specified on the +command line. The target Nextcloud instance URL (``BASE_URL``) and credentials +(``NC_USER`` and ``NC_PASSWORD``) can be provided as environment variables or, +alternatively, hardcoded in the script itself. - echo -en "--$BOUNDARY--\r\n" >> "$UPLOAD_PATH" +.. code-block:: BASH - echo "Creating folder /test" + #!/usr/bin/env bash + set -euo pipefail + + if (( $# != 2 )); then + echo "Usage: $0 " >&2 + exit 1 + fi + + NB=$1 + SIZE=$2 + + BASE_URL="${BASE_URL:-https://nextcloud.local}" + BASE_URL="${BASE_URL%/}" + NC_USER="${NC_USER:-admin}" + NC_PASSWORD="${NC_PASSWORD:-admin}" + + REQUEST_ID="$(openssl rand -hex 8)" + BOUNDARY="boundary_${REQUEST_ID}" + REMOTE_FOLDER="bulk-upload-${REQUEST_ID}" + WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/nextcloud-bulk-upload.XXXXXX")" + UPLOAD_PATH="$WORK_DIR/request.multipart" + + cleanup() { + rm -rf "$WORK_DIR" + } + trap cleanup EXIT + + for ((i = 1; i <= NB; i++)); do + file_name="$(openssl rand -hex 8).bin" + file_local_path="$WORK_DIR/$file_name" + file_remote_path="/$REMOTE_FOLDER/$file_name" + + head -c "$SIZE" /dev/urandom > "$file_local_path" + + file_mtime="$(stat -c %Y "$file_local_path")" + file_checksum="$(sha1sum "$file_local_path" | awk '{print $1}')" + file_size="$(wc -c < "$file_local_path")" + + { + printf -- '--%s\r\n' "$BOUNDARY" + printf 'X-File-Path: %s\r\n' "$file_remote_path" + printf 'X-OC-Mtime: %s\r\n' "$file_mtime" + printf 'Content-Type: application/octet-stream\r\n' + printf 'OC-Checksum: SHA1:%s\r\n' "$file_checksum" + printf 'Content-Length: %s\r\n' "$file_size" + printf '\r\n' + cat "$file_local_path" + printf '\r\n' + } >> "$UPLOAD_PATH" + done + + printf -- '--%s--\r\n' "$BOUNDARY" >> "$UPLOAD_PATH" + + echo "Creating /$REMOTE_FOLDER" curl \ - -X MKCOL \ - -k \ - "https://$USER:$PASS@$SERVER/remote.php/dav/files/$USER/test" > /dev/null - - echo "Uploading $NB files with total size: $(du -sh "$UPLOAD_PATH" | cut -d ' ' -f1)" - echo "Local file is: $UPLOAD_PATH" + --fail-with-body \ + --silent \ + --show-error \ + --user "$NC_USER:$NC_PASSWORD" \ + --request MKCOL \ + "$BASE_URL/remote.php/dav/files/$NC_USER/$REMOTE_FOLDER" + + echo "Uploading $NB files; request body size: $(wc -c < "$UPLOAD_PATH") bytes" curl \ - -X POST \ - -k \ - --progress-bar \ - --cookie "XDEBUG_PROFILE=true;path=/;" \ - -H "Content-Type: multipart/related; boundary=$BOUNDARY" \ - --data-binary "@$UPLOAD_PATH" \ - "https://$USER:$PASS@$SERVER/remote.php/dav/bulk" + --fail-with-body \ + --show-error \ + --progress-bar \ + --user "$NC_USER:$NC_PASSWORD" \ + --header "Content-Type: multipart/related; boundary=$BOUNDARY" \ + --data-binary "@$UPLOAD_PATH" \ + "$BASE_URL/remote.php/dav/bulk" + + printf '\n' From 89c77ec39c045f3f01567db3fb7028c26c337e49 Mon Sep 17 00:00:00 2001 From: Josh Date: Mon, 3 Aug 2026 10:54:29 -0400 Subject: [PATCH 3/3] chore: cleanup trailing whitespace in bulkupload.rst for lint happiness Signed-off-by: Josh --- .../client_apis/WebDAV/bulkupload.rst | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/developer_manual/client_apis/WebDAV/bulkupload.rst b/developer_manual/client_apis/WebDAV/bulkupload.rst index 087593e2f8c..e043d14307f 100644 --- a/developer_manual/client_apis/WebDAV/bulkupload.rst +++ b/developer_manual/client_apis/WebDAV/bulkupload.rst @@ -164,42 +164,42 @@ alternatively, hardcoded in the script itself. #!/usr/bin/env bash set -euo pipefail - + if (( $# != 2 )); then echo "Usage: $0 " >&2 exit 1 fi - + NB=$1 SIZE=$2 - + BASE_URL="${BASE_URL:-https://nextcloud.local}" BASE_URL="${BASE_URL%/}" NC_USER="${NC_USER:-admin}" NC_PASSWORD="${NC_PASSWORD:-admin}" - + REQUEST_ID="$(openssl rand -hex 8)" BOUNDARY="boundary_${REQUEST_ID}" REMOTE_FOLDER="bulk-upload-${REQUEST_ID}" WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/nextcloud-bulk-upload.XXXXXX")" UPLOAD_PATH="$WORK_DIR/request.multipart" - + cleanup() { rm -rf "$WORK_DIR" } trap cleanup EXIT - + for ((i = 1; i <= NB; i++)); do file_name="$(openssl rand -hex 8).bin" file_local_path="$WORK_DIR/$file_name" file_remote_path="/$REMOTE_FOLDER/$file_name" - + head -c "$SIZE" /dev/urandom > "$file_local_path" - + file_mtime="$(stat -c %Y "$file_local_path")" file_checksum="$(sha1sum "$file_local_path" | awk '{print $1}')" file_size="$(wc -c < "$file_local_path")" - + { printf -- '--%s\r\n' "$BOUNDARY" printf 'X-File-Path: %s\r\n' "$file_remote_path" @@ -212,9 +212,9 @@ alternatively, hardcoded in the script itself. printf '\r\n' } >> "$UPLOAD_PATH" done - + printf -- '--%s--\r\n' "$BOUNDARY" >> "$UPLOAD_PATH" - + echo "Creating /$REMOTE_FOLDER" curl \ --fail-with-body \ @@ -223,7 +223,7 @@ alternatively, hardcoded in the script itself. --user "$NC_USER:$NC_PASSWORD" \ --request MKCOL \ "$BASE_URL/remote.php/dav/files/$NC_USER/$REMOTE_FOLDER" - + echo "Uploading $NB files; request body size: $(wc -c < "$UPLOAD_PATH") bytes" curl \ --fail-with-body \ @@ -233,5 +233,5 @@ alternatively, hardcoded in the script itself. --header "Content-Type: multipart/related; boundary=$BOUNDARY" \ --data-binary "@$UPLOAD_PATH" \ "$BASE_URL/remote.php/dav/bulk" - + printf '\n'