Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/snippets/storage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ export const PyStorageTableTimeout = "table = db.create_table(\n \"table\",\n

export const PyStorageTigrisConnect = "db = lancedb.connect(\n \"s3://your-bucket/path\",\n storage_options={\n \"endpoint\": \"https://t3.storage.dev\",\n \"region\": \"auto\",\n },\n)\n";

export const PyStorageCosConnect = "db = lancedb.connect(\n \"cos://my-bucket/my-database\",\n storage_options={\n \"secret_id\": \"<tencent-cloud-secret-id>\",\n \"secret_key\": \"<tencent-cloud-secret-key>\",\n \"region\": \"ap-guangzhou\",\n },\n)\n";

export const PyStorageGoosefsConnect = "db = lancedb.connect(\"goosefs://my-namespace/my-database\")\n";

export const TsStorageAzureAccount = "async function storageAzureAccount() {\n const db = await lancedb.connect(\n \"az://my-container/my-database\",\n {\n storageOptions: {\n accountName: \"some-account\",\n accountKey: \"some-key\",\n },\n },\n );\n return db;\n}\n";

export const TsStorageAzureSas = "async function storageAzureSas() {\n const db = await lancedb.connect(\n \"az://my-container/my-database\",\n {\n storageOptions: {\n azureStorageAccountName: \"some-account\",\n azureStorageSasToken: \"<sas-token>\",\n },\n },\n );\n return db;\n}\n";
Expand Down Expand Up @@ -56,3 +60,5 @@ export const TsStorageTableTimeout = "async function storageTableTimeout() {\n

export const TsStorageTigrisConnect = "async function storageTigrisConnect() {\n const db = await lancedb.connect(\n \"s3://your-bucket/path\",\n {\n storageOptions: {\n endpoint: \"https://t3.storage.dev\",\n region: \"auto\",\n },\n },\n );\n return db;\n}\n";

export const TsStorageGoosefsConnect = "async function storageGoosefsConnect() {\n const db = await lancedb.connect(\"goosefs://my-namespace/my-database\");\n return db;\n}\n";

43 changes: 43 additions & 0 deletions docs/storage/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ import {
PyStorageS3SseKms,
PyStorageTableTimeout,
PyStorageTigrisConnect,
PyStorageCosConnect,
PyStorageGoosefsConnect,
TsStorageAzureAccount,
TsStorageAzureSas,
TsStorageConnectAzure,
Expand All @@ -29,6 +31,7 @@ import {
TsStorageS3SseKms,
TsStorageTableTimeout,
TsStorageTigrisConnect,
TsStorageGoosefsConnect,
} from '/snippets/storage.mdx';

When using LanceDB OSS, you can choose where to store your data. The tradeoffs between storage options are covered in the [storage architecture guide](/storage). This page shows how to configure each backend.
Expand Down Expand Up @@ -296,3 +299,43 @@ Tigris exposes an S3-compatible API. Configure the endpoint and region:
</CodeGroup>

Environment variables `AWS_ENDPOINT=https://t3.storage.dev` and `AWS_DEFAULT_REGION=auto` achieve the same configuration.

## Tencent COS

[Tencent Cloud Object Storage (COS)](https://www.tencentcloud.com/products/cos) is the primary object store for workloads running in the China region. Use the `cos://` URI scheme to connect directly to a COS bucket.

<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyStorageCosConnect}
</CodeBlock>
</CodeGroup>

Supported keys include `secret_id`, `secret_key`, `region`, and `endpoint`. You can also authenticate via the `TENCENTCLOUD_SECRET_ID` and `TENCENTCLOUD_SECRET_KEY` environment variables.

<Note>
**Availability**

COS is bundled in the Python wheel by default. To use it from Rust or the Node binding, build with the `cos` Cargo feature enabled.
</Note>

## GooseFS

[GooseFS](https://www.tencentcloud.com/document/product/1424) is Tencent Cloud's distributed cache acceleration layer for COS and S3. It is a common choice when the same hot dataset is read repeatedly, such as vector search and AI training workloads. Connect using the `goosefs://` URI scheme.

<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyStorageGoosefsConnect}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsStorageGoosefsConnect}
</CodeBlock>
</CodeGroup>

GooseFS reads credentials and endpoint configuration from the GooseFS client environment. See the [GooseFS documentation](https://www.tencentcloud.com/document/product/1424) for cluster setup.

<Note>
**Availability**

GooseFS is bundled by default in the Python wheel and the Node binding. To use it from Rust, build with the `goosefs` Cargo feature enabled.
</Note>

2 changes: 1 addition & 1 deletion docs/storage/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: "Understand LanceDB storage backends, tradeoffs, and how to pick th
icon: "database"
---

LanceDB is one of the few vector databases built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API.
LanceDB is one of the few vector databases built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API. It also supports region-specific backends such as [Tencent COS](/storage/configuration#tencent-cos) and cache-acceleration layers such as [GooseFS](/storage/configuration#goosefs).

Choosing a backend is a balance between latency, scalability, cost, and operational complexity. Use this guide to pick the right fit for your workload.

Expand Down
Loading