docs: add go docs for payload storage by lennessyy · Pull Request #4424 · temporalio/documentation

lennessyy · 2026-04-09T19:01:28Z

What does this PR do?

Add go docs for payload storage

Notes to reviewers

┆Attachments: EDU-6196 docs: add go docs for payload storage

vercel · 2026-04-09T19:01:37Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
temporal-documentation	Ready	Preview, Comment	Apr 10, 2026 8:00pm

github-actions · 2026-04-09T19:02:31Z

📖 Docs PR preview links

Develop
- Go SDK
  - Best Practices
  - Data handling
- Python
  - Data handling
    - External Storage
Evaluate
- Development Production Features
  - Data encryption
External Storage

…mentation into large-payload-go

flippedcoder

Just a couple of small things to consider!

flippedcoder · 2026-04-10T15:02:12Z

docs/develop/go/best-practices/data-handling/index.mdx

+layers that handle different concerns:
+
+<CaptionedImage
+  src="/diagrams/data-converter-flow-with-external-storage.svg"


The arrows in and out of the container looked a little weird to me in dark mode. It could be fine, just worth another look.

switched to themed images. i think maybe after this PR i will just update our captioned images component to have a dark/light theme

flippedcoder · 2026-04-10T15:13:06Z

docs/develop/go/best-practices/data-handling/large-payload-storage.mdx

@@ -0,0 +1,315 @@
+---
+id: large-payload-storage


Could the id be changed to external-storage to match the page title? Maybe external-storage-aws if the other cloud platforms are coming later.

docs/develop/go/best-practices/data-handling/large-payload-storage.mdx

…mentation into large-payload-go

jmaeagle99 · 2026-04-10T18:38:57Z

docs/develop/go/best-practices/data-handling/external-storage.mdx

+   ```go
+   import (
+       "github.com/aws/aws-sdk-go-v2/config"
+       awss3 "github.com/aws/aws-sdk-go-v2/service/s3"


Any particular reason why you aliased the import to awss3 instead of just using the default and calling s3.NewFromConfig?

jmaeagle99 · 2026-04-10T18:40:15Z

docs/develop/go/best-practices/data-handling/external-storage.mdx

+   <!--SNIPEND-->
+
+   By default, payloads larger than 256 KiB are offloaded to external storage. You can adjust this with the
+   `PayloadSizeThreshold` option, even setting it to 0 to externalize all payloads regardless of size. Refer to


Suggested change

`PayloadSizeThreshold` option, even setting it to 0 to externalize all payloads regardless of size. Refer to

`PayloadSizeThreshold` option, even setting it to 1 to externalize all payloads regardless of size. Refer to

The minimum value for externalizing all payloads in the Go SDK is 1. That is because the construction of the ExternalStorage struct already has this field set to 0. We interpret 0 to mean the default value.

jmaeagle99 · 2026-04-10T18:44:09Z

docs/develop/go/best-practices/data-handling/external-storage.mdx

+
+- `Name()` returns a unique string that identifies the driver instance. The SDK stores this name in the claim check
+  reference so it can route retrieval requests to the correct driver. Changing the name after payloads have been stored
+  breaks retrieval. For example, two S3 drivers could be named `"s3-us-east"` and `"s3-eu-west"`.


Not sure that drivers would actually be named after the region that they might be targeting. Maybe they'd be named for why they are stored differently.

jmaeagle99 · 2026-04-10T18:45:50Z

docs/develop/go/best-practices/data-handling/external-storage.mdx

+  reference so it can route retrieval requests to the correct driver. Changing the name after payloads have been stored
+  breaks retrieval. For example, two S3 drivers could be named `"s3-us-east"` and `"s3-eu-west"`.
+- `Type()` returns a string that identifies the driver implementation. Unlike `Name()`, this must be the same across all
+  instances of the same driver type regardless of configuration. Both S3 drivers in the example above would return


Both S3 drivers in the example above would return"aws.s3driver" as their type.

Only the example at the top of the page would return this. The sample driver you wrote just above this would return local-disk.

was referring the example in text s3-us-east etc, but will clarify here

jmaeagle99 · 2026-04-10T18:49:57Z

docs/develop/go/best-practices/data-handling/external-storage.mdx

+	storeDir string
+}
+
+func NewLocalDiskStorageDriver(storeDir string) *LocalDiskStorageDriver {


Probably better to return StorageDriver instead of *LocalDiskStorageDriver.

jmaeagle99 · 2026-04-10T18:50:36Z

docs/develop/go/best-practices/data-handling/external-storage.mdx

+## Configure payload size threshold
+
+You can configure the payload size threshold that triggers external storage. By default, payloads larger than 256 KiB
+are offloaded to external storage. You can adjust this with the `PayloadSizeThreshold` option, or set it to 0 to


Same comment about how 1 is the minimum for externalizing all payloads in Go.

ahh that's right

jmaeagle99 · 2026-04-10T18:50:53Z

docs/develop/go/best-practices/data-handling/external-storage.mdx

+c, err := client.Dial(client.Options{
+    ExternalStorage: converter.ExternalStorage{
+        Drivers:              []converter.StorageDriver{driver},
+        PayloadSizeThreshold: 0,


Ditto https://github.com/temporalio/documentation/pull/4424/changes#r3066173956

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

drewhoskins-temporal

approved but please look at my comments either here or in a followup PR.

drewhoskins-temporal · 2026-04-10T23:37:23Z

docs/develop/go/best-practices/data-handling/external-storage.mdx

+
+### 4. Configure the Client
+
+Pass an `ExternalStorage` struct with your driver in the Client options:


Call out/link out to plugins here as well.

drewhoskins-temporal · 2026-04-10T23:40:52Z

docs/develop/go/best-practices/data-handling/external-storage.mdx

+  - Go SDK
+  - Temporal SDKs
+  - Data Converters


did you intentionally include fewer tags/keywords here than in the encyclopedia?

drewhoskins-temporal · 2026-04-10T23:47:04Z

docs/develop/go/best-practices/data-handling/external-storage.mdx

+## Use multiple storage drivers
+
+When you register multiple drivers, you must provide a `DriverSelector` that implements the `StorageDriverSelector`
+interface. The SDK returns an error at client creation if multiple drivers are registered without a selector. The


nit: There's no need to point out the error, especially at a moment when we're just trying to figure out what selectors do.

drewhoskins-temporal · 2026-04-10T23:55:33Z

docs/develop/go/best-practices/data-handling/external-storage.mdx

+- Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the
+  one you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads.
+  The old driver remains available for retrieving existing claims.
+- Latency vs. durability tradeoffs. Some Workflow types may benefit from a faster storage backend


I'm having second thoughts about this. I'm not sure we should recommend a non-durable solution. In fact, it contradicts above where we recommend "In production, use a durable storage system"
There are indeed a few customers who do this, but it feels like very "at your own risk" rather than something we should recommend.

For an alternative, let's do multi-cloud. S3 if you're on AWS , GCS if on gcloud, etc.

drewhoskins-temporal · 2026-04-10T23:56:59Z

docs/encyclopedia/data-conversion/external-storage.mdx

 slug: /external-storage
 toc_max_heading_level: 4
 keywords:
  - external-storage


I searched for "Large payloads" and this doc didn't come up. Does it need to be a tag for that to happen?
The main thing that made docs come up was the "#large-payloads" slack channel mention.

docs: add go docs for payload storage

152c3c6

Merge branch 'main' into large-payload-go

88906a4

vercel bot deployed to Preview April 9, 2026 19:02 View deployment

lennessyy added 2 commits April 9, 2026 12:26

custom driver addition

10bcdbb

Merge branch 'large-payload-go' of https://github.com/temporalio/docu…

58e16ee

…mentation into large-payload-go

lennessyy marked this pull request as ready for review April 9, 2026 20:15

lennessyy requested a review from a team as a code owner April 9, 2026 20:15

custom driver addition

c73b244

vercel bot deployed to Preview April 9, 2026 20:35 View deployment

address feedback

cdd350d

vercel bot deployed to Preview April 9, 2026 22:11 View deployment

flippedcoder approved these changes Apr 10, 2026

View reviewed changes

lennessyy commented Apr 10, 2026

View reviewed changes

docs/develop/go/best-practices/data-handling/large-payload-storage.mdx Outdated Show resolved Hide resolved

Apply suggestion from @lennessyy

a3dbb8c

vercel bot had a problem deploying to Preview April 10, 2026 16:48 Failure

lennessyy added 2 commits April 10, 2026 09:58

page name change

f1045bf

update page name

fd978fb

vercel bot had a problem deploying to Preview April 10, 2026 16:58 Failure

vercel bot had a problem deploying to Preview April 10, 2026 16:59 Failure

Merge branch 'main' into large-payload-go

0da2106

vercel bot had a problem deploying to Preview April 10, 2026 17:00 Failure

lennessyy added 2 commits April 10, 2026 10:01

sidebar change

a6662b5

Merge branch 'large-payload-go' of https://github.com/temporalio/docu…

e22f1ce

…mentation into large-payload-go

vercel bot deployed to Preview April 10, 2026 17:03 View deployment

use themed images

6edd970

vercel bot deployed to Preview April 10, 2026 18:38 View deployment

add border to dark image

29a4c3b

vercel bot deployed to Preview April 10, 2026 18:45 View deployment

jmaeagle99 reviewed Apr 10, 2026

View reviewed changes

address PR feedback

4a3a76d

vercel bot deployed to Preview April 10, 2026 19:01 View deployment

clarify wording

c1a215f

vercel bot deployed to Preview April 10, 2026 19:07 View deployment

add StorageDriverActivityInfo handling to custom driver example

19c48c4

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

vercel bot deployed to Preview April 10, 2026 19:54 View deployment

add comment clarifying StorageDriverActivityInfo usage

a3e5370

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

vercel bot deployed to Preview April 10, 2026 20:00 View deployment

lennessyy merged commit 88b97f2 into main Apr 10, 2026
10 checks passed

lennessyy deleted the large-payload-go branch April 10, 2026 21:06

drewhoskins-temporal approved these changes Apr 10, 2026

View reviewed changes

	`PayloadSizeThreshold` option, even setting it to 0 to externalize all payloads regardless of size. Refer to
	`PayloadSizeThreshold` option, even setting it to 1 to externalize all payloads regardless of size. Refer to


		### 4. Configure the Client

		Pass an `ExternalStorage` struct with your driver in the Client options:

Conversation

lennessyy commented Apr 9, 2026 • edited by sync-by-unito bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Notes to reviewers

Uh oh!

vercel bot commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📖 Docs PR preview links

Uh oh!

flippedcoder left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lennessyy Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

drewhoskins-temporal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lennessyy commented Apr 9, 2026 •

edited by sync-by-unito bot

Loading

vercel bot commented Apr 9, 2026 •

edited

Loading

github-actions bot commented Apr 9, 2026 •

edited

Loading

lennessyy Apr 10, 2026 •

edited

Loading