Skip to content

Commit 6fb1776

Browse files
feat(uploads): unify signed upload sessions
1 parent 48b9b0b commit 6fb1776

136 files changed

Lines changed: 7465 additions & 8194 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

apps/docs/content/docs/en/platform/self-hosting/object-storage.mdx

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -237,6 +237,26 @@ AZURE_STORAGE_OG_IMAGES_CONTAINER_NAME=og-images
237237
AZURE_STORAGE_WORKSPACE_LOGOS_CONTAINER_NAME=workspace-logos
238238
```
239239

240+
Direct browser uploads require a Blob service CORS rule on the storage account. Allow your exact
241+
Sim origin, `GET` and `PUT`, the `Content-Type` header, and the `x-ms-*` prefix used by signed blob
242+
and metadata headers:
243+
244+
```bash
245+
az storage cors add \
246+
--services b \
247+
--methods GET PUT \
248+
--origins https://sim.yourdomain.com \
249+
--allowed-headers content-type 'x-ms-*' \
250+
--exposed-headers ETag \
251+
--max-age 3600 \
252+
--account-name mystorageaccount \
253+
--account-key '<account-key>'
254+
```
255+
256+
If you authenticate with a connection string, replace the last two options with
257+
`--connection-string "$AZURE_CONNECTION_STRING"`. CORS is configured once for the account's Blob
258+
service and applies to all of its containers.
259+
240260
A full Helm example lives at `helm/sim/examples/values-azure.yaml`.
241261

242262
## Set up Google Cloud Storage
@@ -276,11 +296,13 @@ cat > /tmp/cors.json <<'EOF'
276296
"responseHeader": [
277297
"Content-Type",
278298
"ETag",
299+
"x-goog-meta-uploadid",
279300
"x-goog-meta-originalname",
280301
"x-goog-meta-uploadedat",
281302
"x-goog-meta-purpose",
282303
"x-goog-meta-userid",
283304
"x-goog-meta-workspaceid",
305+
"x-goog-meta-knowledgebaseid",
284306
"x-goog-meta-folderid",
285307
"x-goog-meta-workflowid",
286308
"x-goog-meta-executionid"
@@ -445,6 +467,27 @@ The same browser-reachability and CORS requirements apply.
445467
</Tab>
446468
</Tabs>
447469

470+
## Configure temporary upload cleanup
471+
472+
Sim stages every direct upload under the `upload-sessions/` prefix before promoting it to its final,
473+
immutable object key. Apply the cleanup policy to **every** purpose-specific bucket or container
474+
configured above:
475+
476+
- On AWS S3 and Google Cloud Storage, expire objects under `upload-sessions/` after two days and
477+
abort incomplete multipart uploads after two days.
478+
- On Azure Blob, expire committed blobs under `upload-sessions/` after two days. Azure automatically
479+
removes uncommitted blocks after seven days.
480+
- For an S3-compatible provider, configure both rules when its lifecycle implementation supports
481+
them. Check the provider's documentation because lifecycle feature support varies.
482+
483+
The two-day window exceeds the 24-hour upload-token lifetime and leaves time to retry completion.
484+
Do not apply this prefix rule to final objects outside `upload-sessions/`.
485+
486+
<Callout type="warning">
487+
Configure both expiration and incomplete-multipart cleanup where available. Expiring staged
488+
objects alone does not necessarily remove abandoned multipart parts.
489+
</Callout>
490+
448491
## Verify it works
449492

450493
After restarting with the new configuration:

apps/docs/openapi-v2-files-audit.json

Lines changed: 3 additions & 148 deletions
Original file line numberDiff line numberDiff line change
@@ -168,158 +168,13 @@
168168
"$ref": "#/components/responses/InternalError"
169169
}
170170
}
171-
},
172-
"x-removed-buffered-post": {
173-
"operationId": "uploadFile",
174-
"summary": "Upload File",
175-
"description": "Upload a file to a workspace as `multipart/form-data` with a single `file` field. The workspace — and the optional target `folderId` — are supplied as query parameters (not form fields) so authorization runs before the request body is buffered. Maximum file size is 100MB. A name already taken in the destination folder is **not** an error: the name is auto-suffixed (`data.csv` -> `data (1).csv`), matching the in-app uploader, so a `201` can come back with a `name` different from the one you sent — always read `name` from the response rather than assuming it. `409` is returned only if a unique name cannot be allocated after several attempts. Use `PATCH /api/v2/files/{fileId}` if you need a specific name to be exact-or-fail. Returns `201 Created`.\n\nPresigned upload is not part of the public API: it debits the storage quota only in a separate register step, so a caller that never registers would leave unaccounted bytes in storage. This buffered path debits inside the upload transaction.",
176-
"tags": ["Files"],
177-
"x-codeSamples": [
178-
{
179-
"id": "curl",
180-
"label": "cURL",
181-
"lang": "bash",
182-
"source": "curl -X POST \\\n \"https://www.sim.ai/api/v2/files?workspaceId=YOUR_WORKSPACE_ID\" \\\n -H \"X-API-Key: YOUR_API_KEY\" \\\n -F \"file=@/path/to/file.csv\""
183-
}
184-
],
185-
"parameters": [
186-
{
187-
"$ref": "#/components/parameters/WorkspaceIdQuery"
188-
},
189-
{
190-
"name": "folderId",
191-
"in": "query",
192-
"required": false,
193-
"description": "Target file folder. Omit to upload to the workspace root. Supplied as a query parameter, like `workspaceId`, so authorization runs before the multipart body is buffered.",
194-
"schema": {
195-
"type": "string",
196-
"example": "fold_9Kq2mZ7pR4tLxWc0Ye3Nu"
197-
}
198-
}
199-
],
200-
"requestBody": {
201-
"required": true,
202-
"description": "The file to upload, sent as multipart/form-data.",
203-
"content": {
204-
"multipart/form-data": {
205-
"schema": {
206-
"type": "object",
207-
"required": ["file"],
208-
"properties": {
209-
"file": {
210-
"type": "string",
211-
"format": "binary",
212-
"description": "The file to upload. Maximum size is 100MB."
213-
}
214-
}
215-
}
216-
}
217-
}
218-
},
219-
"responses": {
220-
"201": {
221-
"description": "The file was uploaded successfully.",
222-
"headers": {
223-
"X-RateLimit-Limit": {
224-
"$ref": "#/components/headers/X-RateLimit-Limit"
225-
},
226-
"X-RateLimit-Remaining": {
227-
"$ref": "#/components/headers/X-RateLimit-Remaining"
228-
},
229-
"X-RateLimit-Reset": {
230-
"$ref": "#/components/headers/X-RateLimit-Reset"
231-
}
232-
},
233-
"content": {
234-
"application/json": {
235-
"schema": {
236-
"$ref": "#/components/schemas/V2FileResponse"
237-
},
238-
"example": {
239-
"data": {
240-
"id": "wf_V1StGXR8z5jdHi6BmyT91",
241-
"name": "data.csv",
242-
"size": 1024,
243-
"type": "text/csv",
244-
"key": "workspace/a91c4b2e-6d3f-4e8a-b5c7-0d9e2f1a8c64/1709571234-xyz-data.csv",
245-
"folderId": "fold_9Kq2mZ7pR4tLxWc0Ye3Nu",
246-
"folderPath": "Reports/Q1",
247-
"uploadedBy": "user_abc123",
248-
"uploadedAt": "2026-01-15T10:30:00Z",
249-
"updatedAt": "2026-01-15T10:30:00Z"
250-
}
251-
}
252-
}
253-
}
254-
},
255-
"400": {
256-
"description": "The request was malformed: an invalid `workspaceId` query parameter, a body that is not valid multipart form data, or a missing `file` form field.",
257-
"content": {
258-
"application/json": {
259-
"schema": {
260-
"$ref": "#/components/schemas/V2Error"
261-
},
262-
"example": {
263-
"error": {
264-
"code": "BAD_REQUEST",
265-
"message": "file form field is required"
266-
}
267-
}
268-
}
269-
}
270-
},
271-
"401": {
272-
"$ref": "#/components/responses/Unauthorized"
273-
},
274-
"403": {
275-
"$ref": "#/components/responses/Forbidden"
276-
},
277-
"409": {
278-
"description": "A unique filename could not be allocated in the destination folder after several attempts. An ordinary name collision is auto-suffixed instead, not rejected.",
279-
"content": {
280-
"application/json": {
281-
"schema": {
282-
"$ref": "#/components/schemas/V2Error"
283-
},
284-
"example": {
285-
"error": {
286-
"code": "CONFLICT",
287-
"message": "A file named \"data.csv\" already exists in this workspace"
288-
}
289-
}
290-
}
291-
}
292-
},
293-
"413": {
294-
"description": "The upload exceeds the 100MB file size limit, or the workspace storage limit would be exceeded.",
295-
"content": {
296-
"application/json": {
297-
"schema": {
298-
"$ref": "#/components/schemas/V2Error"
299-
},
300-
"example": {
301-
"error": {
302-
"code": "PAYLOAD_TOO_LARGE",
303-
"message": "File size exceeds 100MB limit (142.30MB)"
304-
}
305-
}
306-
}
307-
}
308-
},
309-
"429": {
310-
"$ref": "#/components/responses/RateLimited"
311-
},
312-
"500": {
313-
"$ref": "#/components/responses/InternalError"
314-
}
315-
}
316171
}
317172
},
318173
"/api/v2/files/uploads": {
319174
"post": {
320175
"operationId": "createFileUpload",
321176
"summary": "Create File Upload",
322-
"description": "Create a stateless multipart upload session and signed upload token. Every file uses this flow; a small file is a single part. The maximum file size is 5 GB.",
177+
"description": "Create an upload session and signed control token. Files up to and including 50 MiB receive a single signed PUT URL; larger files receive multipart transfer instructions. The maximum file size is 5 GB.",
323178
"tags": ["Files"],
324179
"requestBody": {
325180
"required": true,
@@ -405,7 +260,7 @@
405260
"post": {
406261
"operationId": "completeFileUpload",
407262
"summary": "Complete File Upload",
408-
"description": "Verify every part, assemble the object, and atomically register the workspace file.",
263+
"description": "Verify the single PUT or assemble every multipart part, then atomically register the workspace file.",
409264
"tags": ["Files"],
410265
"parameters": [
411266
{
@@ -1681,7 +1536,7 @@
16811536
"name": "upload-token",
16821537
"in": "header",
16831538
"required": true,
1684-
"description": "The signed token returned when the multipart upload was created.",
1539+
"description": "The signed control token returned when the upload session was created.",
16851540
"schema": { "type": "string", "minLength": 1 }
16861541
},
16871542
"FileIdPath": {

0 commit comments

Comments
 (0)