diff --git a/docs/client-api/operations/content/_what-are-operations-csharp.mdx b/docs/client-api/operations/content/_what-are-operations-csharp.mdx index 4f88c4b59e..f0df915415 100644 --- a/docs/client-api/operations/content/_what-are-operations-csharp.mdx +++ b/docs/client-api/operations/content/_what-are-operations-csharp.mdx @@ -303,6 +303,10 @@ Task SendAsync(IMaintenanceOperation operation,        UpdateEtlOperation        [ResetEtlOperation](../../../client-api/operations/maintenance/etl/reset-etl.mdx) +* **Sink tasks**: +        [AddCdcSinkOperation](../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task.mdx#create-a-cdc-sink-task-via-client-api) +        [UpdateCdcSinkOperation](../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/update-task.mdx#update-a-cdc-sink-task-via-client-api) + * **AI tasks**:        [AddEmbeddingsGenerationOperation](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configuring-an-embeddings-generation-task---from-the-client-api) diff --git a/docs/integrations/connection-strings/overview.mdx b/docs/integrations/connection-strings/overview.mdx index b48c252864..0f2078e68f 100644 --- a/docs/integrations/connection-strings/overview.mdx +++ b/docs/integrations/connection-strings/overview.mdx @@ -54,7 +54,7 @@ RavenDB supports the following connection string types: | Type | Connects to | Used by | | - | - | - | | **RavenDB** | Another RavenDB database | [RavenDB ETL](../../server/ongoing-tasks/etl/raven.mdx), [External Replication](../../server/ongoing-tasks/external-replication.mdx),
and [Hub/Sink Replication](../../server/ongoing-tasks/hub-sink-replication.mdx) | -| **SQL** | A relational database | [SQL ETL](../../server/ongoing-tasks/etl/sql.mdx) and CDC Sink tasks | +| **SQL** | A relational database | [SQL ETL](../../server/ongoing-tasks/etl/sql.mdx) and [CDC Sink](../../server/ongoing-tasks/cdc-sink/overview.mdx) | | **Snowflake** | A Snowflake warehouse | [Snowflake ETL](../../server/ongoing-tasks/etl/snowflake.mdx) | | **OLAP** | Local or cloud storage | [OLAP ETL](../../server/ongoing-tasks/etl/olap.mdx) | | **Elasticsearch** | An Elasticsearch cluster | [Elasticsearch ETL](../../server/ongoing-tasks/etl/elasticsearch.mdx) | diff --git a/docs/integrations/connection-strings/per-database/content/_add-or-update-connection-string-csharp.mdx b/docs/integrations/connection-strings/per-database/content/_add-or-update-connection-string-csharp.mdx index f2ea8c151c..1c6ee9544d 100644 --- a/docs/integrations/connection-strings/per-database/content/_add-or-update-connection-string-csharp.mdx +++ b/docs/integrations/connection-strings/per-database/content/_add-or-update-connection-string-csharp.mdx @@ -199,7 +199,8 @@ The specific fields and complete examples are documented in the relevant task or and [Hub/Sink Replication](../../../../server/ongoing-tasks/hub-sink-replication.mdx). See [Add a RavenDB connection string](../../../../server/ongoing-tasks/etl/raven.mdx#add-a-ravendb-connection-string). -* **SQL** - used by [SQL ETL](../../../../server/ongoing-tasks/etl/sql.mdx) and CDC Sink tasks. +* **SQL** - used by [SQL ETL](../../../../server/ongoing-tasks/etl/sql.mdx) + and [CDC Sink](../../../../server/ongoing-tasks/cdc-sink/overview.mdx). See [Add an SQL connection string](../../../../server/ongoing-tasks/etl/sql.mdx#add-an-sql-connection-string). * **Snowflake** - used by [Snowflake ETL](../../../../server/ongoing-tasks/etl/snowflake.mdx). diff --git a/docs/querying/rql/what-is-rql.mdx b/docs/querying/rql/what-is-rql.mdx index 2f114c4c84..368ae2241a 100644 --- a/docs/querying/rql/what-is-rql.mdx +++ b/docs/querying/rql/what-is-rql.mdx @@ -138,7 +138,7 @@ The following keywords and methods are available in RQL: - [long](../../querying/sorting-query-results/sort-query-results.mdx) - [double](../../querying/sorting-query-results/sort-query-results.mdx) - [alphaNumeric](../../querying/sorting-query-results/sort-query-results.mdx) - - [NULLS FIRST | NULLS LAST](../../querying/sorting-query-results/sort-query-results.mdx) // TODO... + - [NULLS FIRST | NULLS LAST](../../querying/sorting-query-results/sort-query-results.mdx#control-position-of-null-values) - [random()](../../querying/sorting-query-results/sort-query-results.mdx#order-by-random) - [score()](../../querying/sorting-query-results/sort-query-results.mdx#order-by-score) - [spatial.distance()](../../client-api/session/querying/how-to-make-a-spatial-query.mdx#spatial-sorting) diff --git a/docs/server/ongoing-tasks/cdc-sink/_category_.json b/docs/server/ongoing-tasks/cdc-sink/_category_.json new file mode 100644 index 0000000000..fece99c61d --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/_category_.json @@ -0,0 +1,4 @@ +{ + "position": 2, + "label": "CDC Sink" +} diff --git a/docs/server/ongoing-tasks/cdc-sink/assets/snagit/the-state-document.snagx b/docs/server/ongoing-tasks/cdc-sink/assets/snagit/the-state-document.snagx new file mode 100644 index 0000000000..4b1d13b5d9 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/assets/snagit/the-state-document.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/assets/the-state-document.png b/docs/server/ongoing-tasks/cdc-sink/assets/the-state-document.png new file mode 100644 index 0000000000..03e98dc5d4 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/assets/the-state-document.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/content/_handling-deletes-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/content/_handling-deletes-csharp.mdx new file mode 100644 index 0000000000..abf9899408 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/content/_handling-deletes-csharp.mdx @@ -0,0 +1,290 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; +import ContentFrame from '@site/src/components/ContentFrame'; + + + +* CDC Sink lets you configure how DELETE events are handled for root tables and embedded tables + by setting the `OnDelete` property (`CdcSinkOnDeleteConfig`) on the table configuration. + +* By default, a DELETE event deletes the corresponding RavenDB document. + For embedded tables, it removes the item from an array or map, or sets a `Value`-type property to `null`. + The `OnDelete` configuration changes this behavior. + +* In this article: + * [Default DELETE behavior](#default-delete-behavior) + * [OnDelete configuration](#ondelete-configuration) + * [OnDelete behavior summary](#ondelete-behavior-summary) + * [Common patterns](#common-patterns) + * [Archive](#archive) + * [Audit trail (root document)](#audit-trail-root-document) + * [Silent ignore](#silent-ignore) + * [OnDelete for embedded tables](#ondelete-for-embedded-tables) + * [DELETE routing and REPLICA IDENTITY](#delete-routing-and-replica-identity) + + + + + +When `OnDelete` is not configured, CDC Sink applies the delete automatically: + +* **Root table DELETE** - Deletes the corresponding RavenDB document. +* **Embedded table DELETE (Array/Map)** - Removes the item from the parent document's array or map. +* **Embedded table DELETE (Value)** - Sets the `Value`-type property on the parent document to `null`. + + + + + +`CdcSinkOnDeleteConfig` has two properties: + + + +```csharp +public class CdcSinkOnDeleteConfig +{ + // JavaScript patch that runs when a DELETE event arrives + // Available variables: this, $row, $old + public string Patch { get; set; } + + // When true, CDC Sink does not apply its automatic delete/removal + public bool IgnoreDeletes { get; set; } +} +``` + + + +* If `Patch` is set, it runs before CDC Sink applies or skips its automatic delete handling. +* If `IgnoreDeletes = true`, CDC Sink skips the automatic delete/removal after the patch runs. +* If `IgnoreDeletes = false` (default), CDC Sink applies the automatic delete/removal after the patch runs. + +A DELETE event is discarded outright only when `IgnoreDeletes = true` **and** `Patch = null` +(that is, `OnDelete.IgnoreDeletes = true` with no `OnDelete.Patch`). +If a delete patch is configured, the event is still routed so the patch can run - even when `IgnoreDeletes = true`. + + +When `IgnoreDeletes = true`, CDC Sink skips its automatic delete/removal but still saves changes made to `this` by the patch. +For root tables, `this` is the document being handled. For embedded tables, `this` is the parent document. + + + + + + +| IgnoreDeletes | Patch | Behavior | +|---------------|-------|----------| +| `false` | null | CDC Sink applies the normal delete/removal (default). | +| `false` | set | Patch runs, and CDC Sink still applies the normal delete/removal. | +| `true` | null | DELETE event is discarded silently; no patch runs. | +| `true` | set | Patch runs, and CDC Sink skips its automatic delete/removal. | + +When CDC Sink applies its automatic delete/removal (that is, when `IgnoreDeletes = false`): + +* For root tables, the delete patch runs before the document is deleted. +* For embedded tables, CDC Sink removes the item (or sets a `Value`-type property to `null`) before running the patch, + and `$old` contains the removed item. + +When `IgnoreDeletes = true`, CDC Sink skips its automatic delete/removal. +If a patch is configured, its changes to `this` are saved unless the patch explicitly calls `del()` or `put()`. + + + + + + + +### Archive + +Keep the document in RavenDB but mark it as archived. +The patch sets archive fields, and `IgnoreDeletes = true` prevents CDC Sink's automatic deletion: + + + +```csharp +OnDelete = new CdcSinkOnDeleteConfig +{ + IgnoreDeletes = true, + Patch = @" + this.Archived = true; + this.ArchivedAt = new Date().toISOString(); + " +} +``` + + + +The document remains in RavenDB with `Archived = true`. +Queries can filter on `Archived` to exclude archived records. + + + + + +### Audit trail (root document) + +When a root document is deleted, use `OnDelete.Patch` to write an audit document with `put()` before CDC Sink applies the delete. + +Because `IgnoreDeletes` is `false` by default, CDC Sink deletes `this` after the patch runs. +Changes made only to `this` do not remain; use `put()` to create a separate document that survives the delete: + + + +```csharp +OnDelete = new CdcSinkOnDeleteConfig +{ + // IgnoreDeletes defaults to false - CDC Sink deletes the RavenDB document after the patch + Patch = @" + put('DeletedOrders/' + id(this), { + OriginalId: id(this), + Customer: this.Customer, + Total: this.Total, + DeletedAt: new Date().toISOString(), + '@metadata': { '@collection': 'DeletedOrders' } + }); + " +} +``` + + + +The patch creates a document in the `DeletedOrders` collection before `this` is deleted. +That audit document is not removed by this delete operation. + + + + + +### Silent ignore + +Set `IgnoreDeletes = true` and leave `Patch` unset to discard DELETE events without running a patch. +Use this for append-only data where deletes should not remove the RavenDB document or embedded item: + + + +```csharp +OnDelete = new CdcSinkOnDeleteConfig +{ + // No patch - DELETE events are silently discarded + IgnoreDeletes = true +} +``` + + + + + + + + + +The same `OnDelete` configuration works on embedded tables. +For embedded tables: + +* `Patch` + runs on the parent document, not on the embedded item. +* `$old` + contains the embedded item's last known state before deletion. +* `IgnoreDeletes = true` + prevents CDC Sink from automatically removing the array/map item or setting a `Value`-type property to `null`. + +--- + +**Example: Keep deleted items in an audit array** + +Rather than removing a deleted line item from the array, move it to a separate `DeletedLines` property: + + + +```csharp +new CdcSinkEmbeddedTableConfig +{ + SourceTableName = "order_lines", + PropertyName = "Lines", + // ... + OnDelete = new CdcSinkOnDeleteConfig + { + IgnoreDeletes = true, + Patch = @" + // Remove from active Lines + this.Lines = (this.Lines || []) + .filter(l => l.LineId !== $old?.LineId); + + // Add to DeletedLines audit array + this.DeletedLines = this.DeletedLines || []; + this.DeletedLines.push({ + LineId: $old?.LineId, + Product: $old?.Product, + Quantity: $old?.Quantity, + DeletedAt: new Date().toISOString() + }); + " + } +} +``` + + + +With `IgnoreDeletes = true`, CDC Sink does not automatically remove the item - +the patch takes full control of both the `Lines` array and the `DeletedLines` audit trail. + +--- + +**Example: Run a patch but still remove the item** + +To run side-effect logic on DELETE while still removing the item from the array, +leave `IgnoreDeletes` as `false` (the default). CDC Sink removes the item, and the patch can use `$old` to inspect the removed item: + + + +```csharp +new CdcSinkEmbeddedTableConfig +{ + SourceTableName = "order_lines", + PropertyName = "Lines", + // ... + OnDelete = new CdcSinkOnDeleteConfig + { + // IgnoreDeletes = false (default) - CDC Sink removes the item + Patch = @" + // Log total for the line being removed + this.RemovedTotal = (this.RemovedTotal || 0) + + ($old?.UnitPrice || 0) * ($old?.Quantity || 0); + " + } +} +``` + + + +The `Lines` item is removed by CDC Sink. The patch only handles the side-effect logic. + + + + + +For embedded-table DELETE events, CDC Sink must know which parent document and embedded item to update. +The DELETE event must include the embedded table's primary key columns and the join columns that identify the parent. + +For PostgreSQL, the default `REPLICA IDENTITY` includes only primary key columns in DELETE events. +If an embedded table's join columns are not part of its primary key, configure `REPLICA IDENTITY` so DELETE events include those join columns. + + +This requirement is PostgreSQL-specific. +Other databases may include the needed DELETE-event columns by default or use a different mechanism. +See [REPLICA IDENTITY](../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx) (PostgreSQL). + + +--- + +**Skipping DELETE routing for embedded tables**: + +* To discard embedded-table DELETE events entirely, set `OnDelete.IgnoreDeletes = true` and leave `OnDelete.Patch` unset. + In that configuration, CDC Sink does not need to route the DELETE event to a parent document, so PostgreSQL `REPLICA IDENTITY` changes are not needed for that delete behavior. + +* This skip applies only when no patch is set. + If `OnDelete.Patch` is configured, the DELETE event is still routed and processed so the patch can run - + so the join columns (and `REPLICA IDENTITY`, if the join columns aren't in the primary key) are still required. + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/content/_monitoring-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/content/_monitoring-csharp.mdx new file mode 100644 index 0000000000..19cbccbca6 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/content/_monitoring-csharp.mdx @@ -0,0 +1,117 @@ +import Admonition from '@theme/Admonition'; +import Panel from '@site/src/components/Panel'; + + + +* This article explains the runtime information exposed for a CDC Sink task - + including its running state, progress, fallback behavior, error handling, notifications, and performance statistics. + +* In this article: + * [Task status and statistics](#task-status-and-statistics) + * [Fallback mode](#fallback-mode) + * [Notifications](#notifications) + * [Error handling](#error-handling) + * [Performance endpoints](#performance-endpoints) + + + + + +Retrieve the current task info with `GetOngoingTaskInfoOperation` and cast the result to `OngoingTaskCdcSink`. +Two standard ongoing-task fields report the task's enabled state and current connection status: + +* `TaskState` - whether the task is enabled, disabled, or partially enabled. +* `TaskConnectionStatus` - whether the task is active, reconnecting, inactive, or running on another node. + +The following runtime fields report how a running task is progressing: + +* **Progress and lag** + `LastBatchTime` and `SecondsSinceLastBatch` show when the task last completed a batch, + while `LastActivityTime` and `SecondsSinceLastActivity` show the last activity from the source. + Recent activity with an old `LastBatchTime` indicates that the source connection is alive but there are no changes to apply. +* **Health** + `HealthIssue` is `null` when healthy and carries a diagnostic message when the task detects a problem, + such as fallback mode or a stale connection. +* **Position** + `LastCheckpoint` reports the last persisted CDC checkpoint (an LSN or GTID). + + + +For the call syntax and the full list of returned fields with their possible values, +see [Get task info via Client API](../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/get-task-info.mdx#get-cdc-sink-task-info-via-client-api). +Task info is also available in the **Ongoing Tasks** view in the Management Studio - +see [Get task info via Studio](../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/get-task-info.mdx#get-cdc-sink-task-info-via-studio). + + + + + + + +When a CDC Sink task encounters a retryable failure, it enters **fallback mode** and retries after a backoff delay. +While the task is in fallback mode, `TaskConnectionStatus` is reported as `Reconnect`. + +In fallback mode: +* The task waits for the current backoff delay, then retries +* No changes are applied while the task is waiting +* If the next attempt succeeds, processing resumes automatically + +The first fallback delay is 5 seconds. +On subsequent failures, the base delay grows to twice the time elapsed since the last error, up to the cap set by `CdcSink.MaxFallbackTimeInSec` (default 15 minutes). +RavenDB then adds up to 10% random jitter to avoid synchronized retries across processes. +See [Server Configuration](../../../../server/ongoing-tasks/cdc-sink/server-configuration.mdx). + + + + + +CDC Sink raises RavenDB alerts for task-level failures and provider-specific warning conditions. +A retryable process failure also moves the task into fallback mode. + +CDC Sink alerts are visible in the **Notification Center** in the Management Studio (bell icon). + +The alert message identifies the affected task and describes the failure or warning. +Depending on the case, it may also include provider-specific context, such as a table, publication, checkpoint, or exception details. + + + + + +CDC Sink can tolerate deterministic per-document failures, such as mapping or patch-script errors, while the overall error ratio remains acceptable. +The failed document is logged and recorded, and CDC Sink continues processing the remaining documents in the batch. + +The CDC checkpoint advances only when the current batch execution completes without errors or successfully processes at least one document. +If all documents in the execution fail, the checkpoint is not advanced. + +**Error threshold:** +When cumulative per-document processing errors reach **100** and also exceed the number of successful operations, +CDC Sink stops the current batch. The failure is reported as an alert, and the task enters fallback mode. + +Non-document-scoped process failures are not treated as tolerable per-document errors. +They fail the current attempt and move the task into fallback mode. + +**Patch errors:** +Patch-script errors, including scripts that exceed `Patching.MaxStepsForScript`, +fail only the affected document while the script-error ratio remains acceptable. +Other documents in the same batch can still be processed. + +For fallback retry timing, see [Fallback mode](#fallback-mode) and [Server Configuration](../../../../server/ongoing-tasks/cdc-sink/server-configuration.mdx). + + + + + +CDC Sink exposes detailed performance statistics through dedicated endpoints: + +| Method | Endpoint | Auth | Description | +|--------|-------------------------------------------------------|-------------|-------------------------------------------------| +| `GET` | `/databases/{databaseName}/cdc-sink/performance` | `ValidUser` | Get the latest CDC Sink performance statistics | +| `GET` | `/databases/{databaseName}/cdc-sink/performance/live` | `ValidUser` | Open a WebSocket stream of live performance statistics | + +Both endpoints can be filtered with the optional `name` query parameter. + +The response is grouped by CDC Sink task and process. +Each performance entry includes timing information, read and processed message counts, +script-processing and read-error counts, memory allocation, batch stop reason, processing status, and nested operation timings. + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/content/_patching-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/content/_patching-csharp.mdx new file mode 100644 index 0000000000..4b9b9ab24b --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/content/_patching-csharp.mdx @@ -0,0 +1,561 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; +import ContentFrame from '@site/src/components/ContentFrame'; + + + +* A **patch** is a JavaScript script that customizes how CDC Sink applies a source-table change to RavenDB. + +* Use `Patch` for INSERT and UPDATE events. + It runs after column mapping and can transform the mapped RavenDB document before CDC Sink saves it. + +* Use `OnDelete.Patch` for DELETE events. + It lets you add delete-handling logic, such as writing audit records, archiving documents, or updating aggregates. + +* Patches are configured per table, for both root-table and embedded-table mappings. + +* In this article: + * [When patches run](#when-patches-run) + * [The patch context](#the-patch-context) + * [Comparing $row, $old, and this](#comparing-row-old-and-this) + * [$row column types](#row-column-types) + * [Patch scope: root vs embedded](#patch-scope-root-vs-embedded) + * [Capabilities and limitations](#capabilities-and-limitations) + * [Idempotency and failover](#idempotency-and-failover) + * [Step limit](#step-limit) + * [Test a patch](#test-a-patch) + * [Common scenarios](#common-scenarios) + * [Column transformation](#column-transformation) + * [Aggregation with embedded tables](#aggregation-with-embedded-tables) + * [Delta computations](#delta-computations) + * [Computed fields](#computed-fields) + * [Document metadata](#document-metadata) + * [Loading related documents](#loading-related-documents) + + + + + +| Event | Configure the script in | When it runs | +|-----------------------------|-------------------------|--------------| +| INSERT | `Patch` | After column mapping | +| UPDATE | `Patch` | After column mapping,
on the merged document | +| DELETE
(root table) | `OnDelete.Patch` | Before the document is deleted | +| DELETE
(embedded table) | `OnDelete.Patch` | On the parent document,
after the embedded item is removed | + + +If `OnDelete.IgnoreDeletes` is enabled, the patch still runs, +but CDC Sink keeps the document or embedded item unless the script deletes it explicitly. + + + + + + +Every patch script runs with three values in scope - `this`, `$row`, and `$old` - +and can use the `load(id)` helper function to read another RavenDB document from the current database. +Understanding what each one holds - and how they differ - is the key to writing correct patches. + +| Object | Type | What it holds | +|------------|----------------|---------------| +| `this` | object | The RavenDB document currently being patched (the root document, or the **parent** document for an embedded table). Write changes to `this` when you want to modify that document. | +| `$row` | object | The **incoming** SQL row from the CDC event,
with all columns (mapped and unmapped). Read-only input. | +| `$old` | object \| null | The **previous** stored document or embedded item, as it was *before* this change.
`null` when there is no previous value (a new document or embedded item).
Read-only snapshot. | +| `load(id)` | function | Loads a RavenDB document by ID from the current database.
See [loading related documents](#loading-related-documents). | + +The [next section](#comparing-row-old-and-this) compares these values side by side, since `$row`, `$old`, and `this` are easy to confuse. + +Additional document functions - `put(id, document)` and `del(id)` - are also available. +See [capabilities and limitations](#capabilities-and-limitations) for details. + + + + + +| Aspect | `$row` | `$old` | `this` | +|--------------------|------------------------------------------------------------|-----------------------------------------------------------------|--------------------------------------------------------------| +| **Represents** | Incoming SQL row from the CDC event | Previous stored document or embedded item, *before* this change | RavenDB document currently being patched | +| **Property names** | Original SQL column names (`customer_name`) | RavenDB document property names (`CustomerName`) | RavenDB document property names (`CustomerName`) | +| **Value types** | Raw SQL types (see [$row column types](#row-column-types)) | Stored JSON types (whatever was last written to RavenDB) | JSON values in the document being patched | +| **On INSERT** | The row being inserted | `null` - no prior state exists | The newly built document | +| **On UPDATE** | The row's new values | The document/item as it was before this update | The document after column mapping or embedded update | +| **On DELETE** | DELETE event data (see note below) | The document/item as it was before deletion | The document about to be deleted, or the parent document for an embedded delete | +| **Mutable?** | No - read-only input | No - read-only snapshot | Yes - write changes here to modify the current document | + + + +* **`$row` vs `this` - source row vs RavenDB document**: + + `$row` is the raw SQL input: original column names, raw source values, including unmapped columns. + `this` is the RavenDB document currently being patched, using RavenDB property names and JSON values. + Use `$row` to read values from the source event, and write document changes to `this`. + +* **`$old` vs `this` - previous state vs current patch target**: + + `$old` is the stored document or embedded item before this change; + `this` is the document currently being patched. + Use `$old` to detect what changed or to reverse a previous contribution (see [delta computations](#delta-computations)). + + + +Given this column mapping: + +```csharp +Columns = +[ + new CdcColumnMapping() { Column = "customer_name", Name = "CustomerName" }, + new CdcColumnMapping() { Column = "amount", Name = "Amount" }, +] +``` +
+ +In the patch script: + + + +```javascript +// $row - SQL column names, raw SQL types (the incoming row) +$row.customer_name // string +$row.amount // number (from numeric/decimal column) + +// $old - mapped property names, stored JSON types (the previous state) +$old?.CustomerName // string, or undefined when there is no previous value +$old?.Amount // number, or undefined when there is no previous value + +// this - mapped property names, after column mapping for this event (the new state) +this.CustomerName // string (just set from $row.customer_name) +this.Amount // number (just set from $row.amount) +``` + + + +--- + +* Unmapped columns - those not listed in `Columns` - appear in `$row` but are not written by column mapping. + They appear in `this` or later in `$old` only if a patch stores them in the document. + +* For root-table DELETE patches, changes made to `this` are saved only when the delete is skipped, + for example with `OnDelete.IgnoreDeletes`. + +* `$row` on a PostgreSQL DELETE + With PostgreSQL's default `REPLICA IDENTITY`, `$row` contains only the primary-key columns on DELETE events. + If a DELETE patch needs values from the deleted row, prefer `$old`, which holds the previous RavenDB document or embedded item. + + + + + +`$row` exposes all SQL columns using their **original column names** (not the mapped RavenDB property names). +The JavaScript type of each value depends on the source database type. + +The table below covers the **PostgreSQL** source provider. +Provider-specific type mappings can differ for SQL Server and MySQL/MariaDB. + +### PostgreSQL + +| PostgreSQL Type | JavaScript Type in `$row` | Notes | +|-----------------------------------------|---------------------------|--------------------------------------------------------------------| +| `integer` / `bigint` | `number` | | +| `real` / `double precision` | `number` | | +| `numeric` | `number` | | +| `boolean` | `boolean` | | +| `date` / `timestamp` / `timestamptz` | `string` | Date/time string, not a JavaScript `Date` | +| `uuid` | `string` | | +| `varchar` / `text` / `char` | `string` | | +| `bytea` | `string` | Base64-encoded | +| `json` / `jsonb` | `string` | In `$row`, JSON is raw text.
Call `JSON.parse()` to access it as an object / array | +| `text[]` / `varchar[]` / `int[]` / etc. | `Array` | Elements are always strings, even for numeric array types | +| `vector` (pgvector) | `Array` | Elements are JavaScript numbers (floats) | + +**JSON/JSONB columns require explicit parsing:** + + + +```javascript +var meta = JSON.parse($row.metadata); +this.Priority = meta.priority; +this.Label = meta.label; +``` + + + +**Array columns:** + + + +```javascript +this.TagCount = $row.tags.length; +this.FirstTag = $row.tags[0]; +this.Tags = $row.tags; + +// INT[] elements are strings - parse explicitly if numeric arithmetic is needed +var first = parseInt($row.scores[0], 10); +``` + + + +**pgvector `vector` columns (elements are numbers, not strings):** + + + +```javascript +this.EmbeddingSum = $row.embedding.reduce(function(a, b) { return a + b; }, 0); +this.Embedding = $row.embedding; +``` + + + + + + + + + +**Root table patches** operate on the root document: + +```csharp +Patch = "this.Status = $row.is_active ? 'Active' : 'Inactive';" +``` +
+ +`this` is the Orders (or whatever root collection) document. + + + + + +**Embedded table patches** operate on the **parent** document, not the embedded item: + +```csharp +// Patch on order_lines embedded table +Patch = "this.TotalQuantity = (this.Lines || []).reduce((s, l) => s + l.Quantity, 0);" +``` +
+ +`this` is the parent Orders document. +This lets you recompute parent-level aggregates when an embedded item is inserted, updated, or deleted. + + +Embedded table patches run with `this` set to the owning RavenDB document that contains the embedded item, +not to the embedded item itself. `$row` is the incoming embedded-table row, and `$old` is the previous embedded item. + + + + + + + + +**What patches can do:** + +* Read `this`, `$row`, and `$old` +* Load documents from the current database with `load(id)` +* Create or replace documents in the current database with `put(id, document)` +* Delete documents from the current database with `del(id)` +* Compute and transform property values +* Set and modify document metadata +* Use JavaScript logic such as conditionals, loops, and array methods +* Use built-in JavaScript objects such as `Date`, `Math`, `JSON`, and `Array` + +**What patches cannot do:** + +* Make external HTTP calls +* Access the file system +* Load documents from other RavenDB databases + + + + + +After a failover, CDC Sink resumes from the last replicated `@cdc-states` checkpoint. +If the previous mentor node processed changes that had not yet reached that checkpoint on the new node, those source changes may be read again. + +A patch is **idempotent** when applying the same source change more than once leaves the document in the same final state. + + + +```javascript +// NOT idempotent - increments again on re-read +this.ViewCount = (this.ViewCount || 0) + 1; + +// Idempotent - absolute value from SQL source +this.ViewCount = $row.view_count; + +// Idempotent - delta via $old +const oldVal = $old?.Amount || 0; +const newVal = $row.amount || 0; +this.RunningTotal = (this.RunningTotal || 0) + (newVal - oldVal); +``` + + + +Column mapping itself is idempotent - writing the same mapped values again is safe. + + + + + +Patch scripts are bounded by a **step quota** rather than a time limit. +Each operation in the script (assignment, loop iteration, function call) consumes steps. + +The limit is controlled by the `Patching.MaxStepsForScript` configuration setting. +See [Patching Configuration](../../../../server/configuration/patching-configuration.mdx). + +If a patch exceeds the limit, the CDC operation fails and is retried. + +Keep patches focused and efficient - avoid unnecessary loops, repeated document loads, and large in-script computations. + + + + + +Validate a patch script against rows fetched from the configured source table before saving the task. +Testing runs CDC Sink mapping and the relevant `Patch` or `OnDelete.Patch`, +then returns the preview result without persisting any documents. + +**From Studio**: +Use the **Test** tool in the CDC Sink task view. +This is the recommended way and requires no code. +See [Test mapping](../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task.mdx#test-mapping) for a step-by-step walkthrough. + +**Over HTTP**: +The Test tool is backed by a REST endpoint you can call directly: + +| Method | Endpoint | Auth | +|--------|----------|------| +| `POST` | `/databases/{databaseName}/admin/cdc-sink/test` | `DatabaseAdmin` | + + +Test mode previews the selected root table mapping and the relevant patch script. +Linked and embedded tables configured under that table are **not** exercised. + + + + + + +The following examples show common patching tasks: + +* [Column transformation](#column-transformation) +* [Aggregation with embedded tables](#aggregation-with-embedded-tables) +* [Delta computations](#delta-computations) +* [Computed fields](#computed-fields) +* [Document metadata](#document-metadata) +* [Loading related documents](#loading-related-documents) + +--- + + + +### Column transformation + +Transform or combine source columns into RavenDB document properties: + + + +```javascript +this.FullName = ($row.first_name + ' ' + $row.last_name).trim(); +this.Status = $row.is_active ? 'Active' : 'Inactive'; +this.CreatedAt = new Date($row.created_unix * 1000).toISOString(); +``` + + + + + + + +### Aggregation with embedded tables + +Recompute parent-level totals whenever an embedded item is inserted, updated, or deleted. + + +If you use a `Patch` to maintain an aggregate, you **must** also provide an `OnDelete.Patch` that updates the aggregate when an item is deleted. +Without it, deletes will leave the aggregate in an incorrect state. + + + + +```csharp +new CdcSinkEmbeddedTableConfig +{ + SourceTableName = "order_lines", + PropertyName = "Lines", + PrimaryKeyColumns = ["line_id"], + JoinColumns = ["order_id"], + Columns = + [ + new CdcColumnMapping() { Column = "line_id", Name = "LineId" }, + new CdcColumnMapping() { Column = "quantity", Name = "Quantity" }, + ], + + // Runs on INSERT and UPDATE - recomputes total from current Lines array + Patch = @" + this.TotalQuantity = (this.Lines || []) + .reduce((sum, line) => sum + (line.Quantity || 0), 0); + ", + + // REQUIRED: Runs on DELETE - recomputes after item is removed from array + OnDelete = new CdcSinkOnDeleteConfig + { + Patch = @" + this.TotalQuantity = (this.Lines || []) + .reduce((sum, line) => sum + (line.Quantity || 0), 0); + " + } +} +``` + + + +With the default delete behavior, the `OnDelete.Patch` runs after the item has already been removed from the array, +so re-summing `this.Lines` gives the correct post-deletion total. + + + + + +### Delta computations + +Use `$old` to compute the difference between the previous embedded item value and the new source-row value, +so you can keep a running total without recomputing it from scratch. + + +Running-total patches that use `$old` must also provide an `OnDelete.Patch` that subtracts the deleted item's value. +Without it, deletes leave the running total incorrect. + + + + +```csharp +new CdcSinkEmbeddedTableConfig +{ + SourceTableName = "invoice_lines", + PropertyName = "Lines", + PrimaryKeyColumns = ["line_id"], + JoinColumns = ["invoice_id"], + Columns = + [ + new CdcColumnMapping() { Column = "line_id", Name = "LineId" }, + new CdcColumnMapping() { Column = "amount", Name = "Amount" }, + ], + + // INSERT: $old is null, so delta = new amount (0 → new) + // UPDATE: $old has previous Amount, delta = new - old + Patch = @" + const oldAmount = $old?.Amount || 0; + const newAmount = $row.amount || 0; + this.RunningTotal = (this.RunningTotal || 0) + (newAmount - oldAmount); + ", + + // REQUIRED: Subtract the deleted item's amount using $old + OnDelete = new CdcSinkOnDeleteConfig + { + Patch = @" + const deletedAmount = $old?.Amount || 0; + this.RunningTotal = (this.RunningTotal || 0) - deletedAmount; + " + } +} +``` + + + +In `OnDelete.Patch`, `$old` contains the embedded item's last known state before deletion. + + + + + +### Computed fields + +Compute document fields from unmapped source columns that you don't want to store directly: + + + +```csharp +Columns = +[ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "name", Name = "Name" }, +], +// base_price and tax_rate are NOT in Columns +Patch = @" + this.FinalPrice = $row.base_price * (1 + $row.tax_rate); + this.Discount = $row.is_vip ? $row.base_price * 0.1 : 0; +" +``` + + + + + + + +### Document metadata + +Patches can set or clear RavenDB document metadata, including expiration: + + + +```javascript +this['@metadata'] = this['@metadata'] || {}; + +if ($row.expires_at) { + this['@metadata']['@expires'] = new Date($row.expires_at).toISOString(); +} else { + delete this['@metadata']['@expires']; +} + +this['@metadata']['SourceTable'] = 'orders'; +``` + + + + + + + +### Loading related documents + +Use `load()` to load a related RavenDB document and denormalize its data: + + + +```javascript +const customer = load('Customers/' + $row.customer_id); + +if (customer) { + this.CustomerName = customer.Name; + this.CustomerEmail = customer.Email; + this.CustomerTier = customer.Tier; +} else { + // Related document does not exist yet, has not been synced yet, or was deleted + this.CustomerName = null; + this.CustomerEmail = null; + this.CustomerTier = null; +} +``` + + + +Use `load()` when you want to copy related document values into the current document at patch time. +The copied values are denormalized; later changes to the loaded document do not automatically update this document. + +`load()` returns `null` if the document does not exist or has not yet been created by CDC Sink +(race condition when multiple tables are loading in parallel). +Always check for null before accessing properties. + +**When to use `load()`:** + +* Denormalizing slowly-changing reference data (customer name, category, region) +* Capturing a snapshot of related data at insert time + +**Prefer linked tables** when you only need to store a simple foreign-key reference by document ID. +They avoid denormalized-copy drift and the null-handling complexity of `load()`. + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/document-modeling/_category_.json b/docs/server/ongoing-tasks/cdc-sink/document-modeling/_category_.json new file mode 100644 index 0000000000..4c5404ed18 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/document-modeling/_category_.json @@ -0,0 +1,4 @@ +{ + "position": 4, + "label": "Document Modeling" +} diff --git a/docs/server/ongoing-tasks/cdc-sink/document-modeling/attachment-handling.mdx b/docs/server/ongoing-tasks/cdc-sink/document-modeling/attachment-handling.mdx new file mode 100644 index 0000000000..8dba1841c4 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/document-modeling/attachment-handling.mdx @@ -0,0 +1,35 @@ +--- +title: "CDC Sink: Attachment Handling" +sidebar_label: Attachment Handling +description: "Explains how to store binary or text SQL columns as RavenDB attachments using CdcColumnType.Attachment on root and embedded table configurations." +sidebar_position: 5 +supported_languages: ["csharp"] +see_also: + - title: "Column Mapping" + link: "server/ongoing-tasks/cdc-sink/document-modeling/column-mapping" + source: "docs" + path: "Server > CDC Sink" + - title: "Embedded Tables" + link: "server/ongoing-tasks/cdc-sink/document-modeling/embedded-tables" + source: "docs" + path: "Server > CDC Sink" + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" + - title: "Attachments Overview" + link: "document-extensions/attachments/overview" + source: "docs" + path: "Document Extensions > Attachments" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import AttachmentHandlingCsharp from './content/_attachment-handling-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/document-modeling/column-mapping.mdx b/docs/server/ongoing-tasks/cdc-sink/document-modeling/column-mapping.mdx new file mode 100644 index 0000000000..b38c4b9a9c --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/document-modeling/column-mapping.mdx @@ -0,0 +1,39 @@ +--- +title: "CDC Sink: Column Mapping" +sidebar_label: Column Mapping +description: "Explains how to control which SQL columns appear in RavenDB documents and under what property names using the Columns list of CdcColumnMapping entries." +sidebar_position: 4 +supported_languages: ["csharp"] +see_also: + - title: "Schema Design" + link: "server/ongoing-tasks/cdc-sink/document-modeling/schema-design" + source: "docs" + path: "Server > CDC Sink" + - title: "Patching" + link: "server/ongoing-tasks/cdc-sink/patching" + source: "docs" + path: "Server > CDC Sink" + - title: "Attachment Handling" + link: "server/ongoing-tasks/cdc-sink/document-modeling/attachment-handling" + source: "docs" + path: "Server > CDC Sink" + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" + - title: "Patching: $row Column Types" + link: "server/ongoing-tasks/cdc-sink/patching#row-column-types" + source: "docs" + path: "Server > CDC Sink" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import ColumnMappingCsharp from './content/_column-mapping-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_attachment-handling-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_attachment-handling-csharp.mdx new file mode 100644 index 0000000000..149c1ee2c6 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_attachment-handling-csharp.mdx @@ -0,0 +1,203 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; +import ContentFrame from '@site/src/components/ContentFrame'; + + + +* To store a source column as a RavenDB attachment instead of a document property, + set `Type = CdcColumnType.Attachment` on the column mapping entry. + +* Attachment mappings are typically used for source columns with binary, text, or vector content, + such as `bytea`, `varbinary`, `text`, `nvarchar`, or pgvector `vector` columns. + +* Attachment mappings can be used on root tables and embedded tables: + * **Root-table attachments** are stored on the mapped document. + * **Embedded-table attachments** are stored on the parent document with a generated + `{PropertyName}/{primary-key-values}/` name prefix. + +* In this article: + * [Root table attachments](#root-table-attachments) + * [Embedded table attachments](#embedded-table-attachments) + * [Attachment naming](#attachment-naming) + * [Attachment lifecycle](#attachment-lifecycle) + + + + + +Given a root SQL table with a binary column: + + + +```sql +CREATE TABLE files ( + id SERIAL PRIMARY KEY, + filename TEXT NOT NULL, + mime_type TEXT NOT NULL, + content BYTEA NOT NULL -- binary attachment +); +``` + + + +Set `Type = CdcColumnType.Attachment` on the `content` column entry: + + + +```csharp +new CdcSinkTableConfig +{ + CollectionName = "Files", + SourceTableName = "files", + PrimaryKeyColumns = ["id"], + Columns = + [ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "filename", Name = "Filename" }, + new CdcColumnMapping() { Column = "mime_type", Name = "MimeType" }, + new CdcColumnMapping() + { + Column = "content", Name = "file", + // store this column as a RavenDB attachment + Type = CdcColumnType.Attachment + }, + ] +} +``` + + + +The `content` column is stored as an attachment named `"file"` on the document. +The `Name` value on the attachment mapping becomes the attachment name. + +The `filename` and `mime_type` columns are stored as regular document properties. +They are not used to set the RavenDB attachment name or content type. +The attachment content type is determined by the **runtime value type**: +`application/octet-stream` for binary values, `text/plain; charset=utf-8` for text values. + + + + + +Columns mapped as attachments on embedded tables are stored as attachments on the **parent** document. +The attachment name is prefixed with the embedded property name and primary key values. + +Given a parent `products` table and a child `photos` table: + + + +```sql +CREATE TABLE products ( + product_id SERIAL PRIMARY KEY, + name TEXT NOT NULL +); + +CREATE TABLE photos ( + product_id INT NOT NULL REFERENCES products(product_id), + photo_num INT NOT NULL, + caption TEXT, + thumbnail BYTEA, -- binary attachment + PRIMARY KEY (product_id, photo_num) +); +``` + + + +Configure the embedded table with `Type = CdcColumnType.Attachment` on the attachment column: + + + +```csharp +new CdcSinkEmbeddedTableConfig +{ + SourceTableName = "photos", + PropertyName = "Photos", + PrimaryKeyColumns = ["photo_num"], + JoinColumns = ["product_id"], + Columns = + [ + new CdcColumnMapping() { Column = "photo_num", Name = "PhotoNum" }, + new CdcColumnMapping() { Column = "caption", Name = "Caption" }, + new CdcColumnMapping() + { + Column = "thumbnail", Name = "thumb", + Type = CdcColumnType.Attachment + }, + ] +} +``` + + + +A photo with `photo_num = 1` creates an attachment named `"Photos/1/thumb"` on the parent document. +The prefix `"Photos/1/"` is generated from the `PropertyName` and the primary key value. + + + + + + + +### Root table attachments + +The attachment name is exactly the `Name` value on the mapping entry with `Type = CdcColumnType.Attachment`. + +```text +Column = "content", Name = "file", Type = CdcColumnType.Attachment +→ Attachment name: "file" +``` + + + + + +### Embedded table attachments + +The attachment name is built from the embedded `PropertyName`, the embedded row's primary key values, +and the mapping `Name`. +The generated attachment name has this format: + +```text +{PropertyName}/{primary-key-value-1}/.../{Name} +``` +
+ +where `{Name}` is the `Name` value from the attachment column mapping. + +``` +PropertyName = "Photos" +PrimaryKeyColumns = ["photo_num"] → photo_num = 1 +Column = "thumbnail", Name = "thumb", Type = CdcColumnType.Attachment +→ Attachment name: "Photos/1/thumb" +``` +
+ +For composite primary keys, the key values are added in `PrimaryKeyColumns` order and separated with `/`: + +```text +PrimaryKeyColumns = ["date", "seq"] → date='2024-01', seq=3 +→ Attachment name: "Photos/2024-01/3/thumb" +``` + + + + + + + +* **INSERT** + A non-null attachment value creates an attachment on the document. + +* **UPDATE** + A non-null attachment value replaces the existing attachment. + If the new value is `NULL`, the attachment update is skipped and the existing attachment is left unchanged. + +* **DELETE (embedded item)** + When the embedded delete is processed, attachments generated from that embedded item are removed from the parent document. + +* **DELETE (root document)** + When the root document is deleted, the document and its attachments are deleted. + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_column-mapping-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_column-mapping-csharp.mdx new file mode 100644 index 0000000000..31883c700b --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_column-mapping-csharp.mdx @@ -0,0 +1,242 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; +import ContentFrame from '@site/src/components/ContentFrame'; + + + +* Column mapping defines which SQL columns are written to RavenDB, what document property or attachment names they use, and how each value is stored. + It uses a `Columns` list of `CdcColumnMapping` objects, not a dictionary. + +* In this article: + * [Mapping columns to properties](#mapping-columns-to-properties) + * [Column types](#column-types) + * [Unmapped columns](#unmapped-columns) + * [JSON columns](#json-columns) + * [Validation rules](#validation-rules) + * [Schema (source table schema)](#schema-source-table-schema) + + + + + +`Columns` is a `List` where each entry maps a SQL column to a RavenDB document property attachment: + + + +```csharp +Columns = +[ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "customer_name", Name = "CustomerName" }, + new CdcColumnMapping() { Column = "order_date", Name = "OrderDate" }, + new CdcColumnMapping() { Column = "total_amount", Name = "TotalAmount" }, +] +``` + + + +* **`Column`:** SQL column name (case-insensitive match against the column names in CDC events) +* **`Name`:** Property name in the RavenDB document, or attachment name when `Type = CdcColumnType.Attachment` + + + +Every column in `PrimaryKeyColumns` must also be mapped in `Columns` as a regular document property. +CDC Sink stores the value in the document and reads it back to match the correct embedded array/map item on subsequent UPDATE/DELETE events. +The document ID itself is built from the primary key's raw source-row value. + + + + + + + +Each `CdcColumnMapping` entry has an optional `Type` property of type `CdcColumnType` that controls how the SQL value is stored in RavenDB: + + + +```csharp +Columns = +[ + new CdcColumnMapping() { Column = "order_id", Name = "OrderId" }, + // Parsed as a native JSON value + new CdcColumnMapping() + { + Column = "metadata", Name = "Metadata", + Type = CdcColumnType.Json + }, + // Stored as an attachment + new CdcColumnMapping() + { + Column = "receipt", Name = "receipt.pdf", + Type = CdcColumnType.Attachment + }, +] +``` + + + + + +### `CdcColumnType.Default` + +Standard SQL-to-JSON type conversion. +This is the default when `Type` is omitted. Type conversions are source-database-specific. + +For PostgreSQL: + + * `smallint` / `integer` / `oid` → `long` + * `bigint` → `long` + * `real` → `float` + * `double precision` → `double` + * `numeric` / `decimal` → `decimal` + * `boolean` → `bool` + * `date` → `DateOnly` + * `timestamp` / `timestamptz` → `DateTime` + * `uuid` → `string` + * `varchar` / `text` / `char` → `string` + * `json` / `jsonb` → `string` (raw JSON text, not parsed) + * SQL arrays → JSON arrays + * `vector` (pgvector) → `float[]` + +For the JavaScript values exposed in patch scripts, see [Patching: $row Column Types](../../../../../server/ongoing-tasks/cdc-sink/patching.mdx#row-column-types). + + + + +### `CdcColumnType.Json` + +Parses the string value as a native JSON value (object, array, string, number, boolean, etc.) in the document. +Use for PostgreSQL `json`/`jsonb`, MySQL/MariaDB `json`, or SQL Server `nvarchar(max)` with JSON content. + + + + +### `CdcColumnType.Attachment` + +Stores the raw column value as a RavenDB attachment. +`byte[]` → binary (`application/octet-stream`), `string` → UTF-8 text (`text/plain`), +`float[]`/`double[]` → raw vector data. The `Name` field becomes the attachment name. + +Learn more in [CDC Sink: Attachment handling](../../../../../server/ongoing-tasks/cdc-sink/document-modeling/attachment-handling.mdx). + + + + + + + +Columns not listed in `Columns` are **not stored automatically** in the document, +but they are available in patch scripts via the `$row` variable. + +This allows you to use data for computations without permanently storing raw SQL values. + +In the following example, `base_price` and `tax_rate` are available during the patch but not stored as document properties. +Only the computed `FinalPrice` is stored. + + + +```csharp +Columns = +[ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "name", Name = "Name" }, + // base_price and tax_rate are NOT mapped - won't appear in document +], +Patch = "this.FinalPrice = $row.base_price * (1 + $row.tax_rate);" +``` + + + + + +### Patch script variables + +### `$this` +The current RavenDB document is available as `this`. +Mapped columns are written using their `Name` values, so the `name` column above becomes `this.Name`. +Patch scripts can also add computed properties, such as `this.FinalPrice`. + +### `$row` +Source column names are available through `$row`, including mapped and unmapped columns. +For example, `$row.base_price` reads the current row's `base_price` value. + +### `$old` +`$old` contains the previous stored value before the current change: +for root-table patches, the previous document; for embedded-table patches, the previous embedded item. +It uses stored document property names, not SQL column names. +Unmapped columns do not appear in `$old` unless a patch previously stored them. + +See [Patching: $row and $old](../../../../../server/ongoing-tasks/cdc-sink/patching.mdx#row-and-old-names-and-types) for more details. + + + + + + + +SQL columns that contain JSON data, such as PostgreSQL `json`/`jsonb`, MySQL/MariaDB `json`, or SQL Server `nvarchar(max)` with JSON content, +can be parsed into native JSON values using `CdcColumnType.Json`. +This handles JSON objects, arrays, and scalar values (strings, numbers, booleans, and `null`): + + + +```csharp +Columns = +[ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "name", Name = "Name" }, + new CdcColumnMapping() + { + Column = "metadata", Name = "Metadata", + Type = CdcColumnType.Json + }, +] +``` + + + +Without `CdcColumnType.Json`, the JSON column value would be stored as a raw string. +With it, the value is parsed and stored as a nested object, array, or scalar value in the document. + +The same `Type = CdcColumnType.Json` mapping is used regardless of the source provider. +For SQL Server, set this explicitly on string columns that contain JSON, since the source column type is still `nvarchar`. + + + + + +The following rules apply to both root table and embedded table column mappings: + +- Each `Column` name must be unique within the table. The check is case-insensitive. +- Each `Name` must be unique within the table and must not duplicate any embedded or linked table `PropertyName` on the same parent. The check is case-insensitive. +- Both `Column` and `Name` are required and must be non-empty strings. +- At least one entry is required - an empty `Columns` list is a validation error. +- Every column in `PrimaryKeyColumns` must also appear in `Columns`. + Map primary key columns as regular document properties, because CDC Sink reads the stored values to match the correct embedded array/map item on subsequent UPDATE/DELETE events. + The document ID itself is built from the primary key's raw source-row value. + + + + + +`SourceTableSchema` specifies the source schema that contains the table. +If omitted, CDC Sink uses the provider default: `public` for PostgreSQL, `dbo` for SQL Server, and the database selected in the connection string for MySQL/MariaDB. + + + +```csharp +new CdcSinkTableConfig +{ + SourceTableSchema = "sales", // Table is in the "sales" schema + SourceTableName = "orders", + // ... +} +``` + + + +When the table is in the provider's default schema, this can be omitted. + + diff --git a/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_embedded-tables-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_embedded-tables-csharp.mdx new file mode 100644 index 0000000000..97aa309f67 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_embedded-tables-csharp.mdx @@ -0,0 +1,307 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; +import ContentFrame from '@site/src/components/ContentFrame'; + + + +* Embedded tables let CDC Sink store related source table rows inside a parent RavenDB document as arrays, maps, + or a single object. + +* This article covers embedded table configuration, key constraints, and how embedded items are matched, updated, and deleted. + +* In this article: + * [Basic configuration](#basic-configuration) + * [Join columns and primary key interaction](#join-columns-and-primary-key-interaction) + * [Matching items on update and delete](#matching-items-on-update-and-delete) + * [Deep nesting](#deep-nesting) + * [Attachments on embedded items](#attachments-on-embedded-items) + + + + + +* Each embedded table is configured with a `CdcSinkEmbeddedTableConfig` object in its parent table configuration's `EmbeddedTables` list. + +* This configuration defines the related source table, the parent document property that will hold the embedded data, + the join columns used to find the parent document, the embedded table's primary key columns, and the columns to map for each embedded item. + +* `CdcSinkEmbeddedTableConfig` is placed inside a root table's `EmbeddedTables` list: + + + + ```csharp + new CdcSinkEmbeddedTableConfig + { + SourceTableSchema = "public", // Source schema (default: "public" PostgreSQL, "dbo" SQL Server, db name for MySQL/MariaDB) + SourceTableName = "order_lines", // SQL table name + PropertyName = "Lines", // Property name in parent document + Type = CdcSinkRelationType.Array, // Array, Map, or Value + JoinColumns = ["order_id"], // FK to parent's PrimaryKeyColumns + PrimaryKeyColumns = ["line_id"], // Used to match items on UPDATE/DELETE + Columns = + [ + new CdcColumnMapping() { Column = "line_id", Name = "LineId" }, + new CdcColumnMapping() { Column = "product", Name = "Product" }, + new CdcColumnMapping() { Column = "quantity", Name = "Quantity" }, + ] + } + ``` + + + + + + + + + +### Purpose of JoinColumns + +* `JoinColumns` lists the embedded table columns whose values correspond to the direct parent table's `PrimaryKeyColumns`. +* CDC Sink pairs the two lists by position. The column names can differ, but the order and count must match: + + + + ```csharp + // Parent root table: + PrimaryKeyColumns = ["id"] + + // Embedded table - CORRECT: + JoinColumns = ["order_id"] // FK that references the parent's PK ("id"); + // the name can differ, but it must correspond + // to the parent's PK in the same order/count + + // Embedded table - INCORRECT: + JoinColumns = ["order_id", "line_no"] // Column count does not match the parent's + // single-column PrimaryKeyColumns + ``` + + + + + + +### DELETE events + +For embedded-table DELETE events, CDC Sink needs the embedded table's primary key values to find the item to remove, +and the join column values to route the event to the correct parent document or embedded parent item. + +Whether the source includes the join columns in DELETE events depends on the database: + +* **PostgreSQL:** + The default `REPLICA IDENTITY` includes only primary key columns in DELETE events. + If the join columns are not part of the embedded table's primary key, additional PostgreSQL configuration is required. + See [REPLICA IDENTITY](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx) + for how CDC Sink handles this automatically when it has sufficient permissions, + or how a DBA can configure it manually. + +* **SQL Server:** + CDC change rows always include all tracked columns, so DELETE events already carry the join columns. + No extra configuration is required. + +* **MySQL / MariaDB:** + DELETE events carry the full row only when `binlog_row_image = FULL`, which CDC Sink already requires. + With that in place, the join columns are included. + + + +#### Skipping embedded deletes + +Regardless of source, if deletes on an embedded table don't need to be processed, +set `OnDelete.IgnoreDeletes = true` and leave `OnDelete.Patch` unset. + +In that case, CDC Sink discards delete events for the embedded table and does not need to route them to the parent. +For PostgreSQL, this also avoids any REPLICA IDENTITY changes. + +If an `OnDelete.Patch` is configured, CDC Sink still routes the delete to the parent document so the patch can run. +See [Handling Deletes](../../../../../server/ongoing-tasks/cdc-sink/handling-deletes.mdx). + + + + + +#### Avoiding PostgreSQL REPLICA IDENTITY changes + +If embedded deletes do need to be processed, +a schema-level way to avoid PostgreSQL REPLICA IDENTITY changes is to include the join column in the embedded table's primary key: + + + +```sql +-- SQL schema: +CREATE TABLE order_lines ( + order_id INT NOT NULL REFERENCES orders(id), + line_id INT NOT NULL, + product VARCHAR(200), + PRIMARY KEY (order_id, line_id) -- order_id in PK means DELETE events include it +); +``` + + + + + +```csharp +// Configuration: +PrimaryKeyColumns = ["order_id", "line_id"] +JoinColumns = ["order_id"] +``` + + + +With `order_id` in the primary key, PostgreSQL DELETE events include it by default and no additional REPLICA IDENTITY configuration is needed. + + + + + + + + + +When an INSERT, UPDATE, or DELETE arrives for an embedded row, +CDC Sink identifies the embedded item by the values in the columns listed in `PrimaryKeyColumns`. + +For `Array` and `Map` embedded tables, CDC Sink uses the full `PrimaryKeyColumns` composite key: + +* **INSERT** + Appends the item to the array or adds it to the map. +* **UPDATE** + Finds the item by PK match and merges the mapped column values into it; + if no item is found, the row is added as a new item. +* **DELETE** + Finds the item by PK match and removes it from the array/map, unless `OnDelete` changes the behavior. + +**Composite PKs** work the same way - all PK columns must match: + + + +```csharp +PrimaryKeyColumns = ["invoice_date", "invoice_seq"] + +// UPDATE event for (invoice_date='2024-01-15', invoice_seq=3) +// → Finds and updates the item where both columns match +``` + + + +**Case sensitivity:** +By default, string PK values are matched case-insensitively. +Set `CaseSensitiveKeys = true` on `CdcSinkEmbeddedTableConfig` if your string keys are case-sensitive. + +**Value embedded tables:** +The matching behavior above applies to `Array` and `Map` embedded tables. +For `Value` embedded tables, there is only one embedded object at the configured property. +Updates merge into that object, and deletes set the property to `null` unless `OnDelete` changes the behavior. + + + + + +Embedded tables can contain their own `EmbeddedTables`, creating hierarchies with multiple levels. + +**Key constraint:** +Every descendant table must include the column values that identify the root document. +In practice, this means carrying the root-level join columns: +the columns from the first embedded table's `JoinColumns`, which correspond to the root table's `PrimaryKeyColumns`. + +CDC Sink uses those values to route each row to its root document in a single pass. + +--- + +**Example: employees nested under departments** + +The `employees` table must have `company_id`, the root-level join column that identifies the owning company, +even though it only directly joins to `departments`: + + + +```sql +CREATE TABLE employees ( + company_id INT NOT NULL, -- Denormalized root PK + dept_id INT NOT NULL, + emp_id INT NOT NULL, + PRIMARY KEY (company_id, dept_id, emp_id) +); +``` + + + + + +```csharp +// Configuration for the employees embedded table: +JoinColumns = ["dept_id"] // FK to the direct parent (departments) PK +PrimaryKeyColumns = ["emp_id"] // Local item key within each department +``` + + + +Note that `PrimaryKeyColumns` here lists only `emp_id`, not the full physical `PRIMARY KEY (company_id, dept_id, emp_id)` from the SQL schema. +At each nesting level, `PrimaryKeyColumns` identifies the item **within its direct parent's** array or map, +so only the local key column (`emp_id`) is needed. + +The root PK (`company_id`) still has to be present as a denormalized column on the row (see the key constraint above) +so CDC Sink can locate the root document, but it is not part of the embedded table's `PrimaryKeyColumns`. + + + +**Why is the root PK required?** + +When a CDC Sink event arrives for an `employees` row, CDC Sink needs to: + +1. Find the root document using `company_id` +2. Navigate to the department item whose key matches `dept_id` +3. Add or update the employee item whose key matches `emp_id` + +Without `company_id` in the event, CDC Sink cannot identify the root document without an additional lookup, +which is not supported. + + + + + + + +Source columns from embedded tables can be stored as RavenDB attachments by adding a mapping with` Type = CdcColumnType.Attachment`. +When that type is set, the mapping's `Name` value is used as the attachment name, +not as a property on the embedded item. + +For example: + + + +```csharp +new CdcSinkEmbeddedTableConfig +{ + SourceTableName = "photos", + PropertyName = "Photos", + PrimaryKeyColumns = ["photo_num"], + JoinColumns = ["product_id"], + Columns = + [ + new CdcColumnMapping() { Column = "photo_num", Name = "PhotoNum" }, + new CdcColumnMapping() { Column = "caption", Name = "Caption" }, + new CdcColumnMapping() + { + Column = "thumbnail", Name = "thumb", + Type = CdcColumnType.Attachment + }, + ] +} +``` + + + +In the example above, +`Column = "thumbnail", Name = "thumb", Type = CdcColumnType.Attachment` stores the `thumbnail` column as an attachment named `thumb`; +it does not add a `thumb` field to the embedded item. + +A photo with `photo_num = 1` creates an attachment named `"Photos/1/thumb"` on the parent document. +The attachment name is prefixed with the embedded table's `PropertyName` and primary key values. + +When the embedded item is removed, attachments with that embedded-item prefix are automatically removed. + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_linked-tables-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_linked-tables-csharp.mdx new file mode 100644 index 0000000000..a4d8e82098 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_linked-tables-csharp.mdx @@ -0,0 +1,130 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* A **linked table** writes a RavenDB document ID reference instead of embedding related data. + CDC Sink builds the ID from `LinkedCollectionName` and the configured `JoinColumns`. + +* `CdcSinkLinkedTableConfig` does **not** create, load, or maintain the referenced document. + The referenced document is maintained by CDC Sink only when that source table is also configured as a root table in the task. + +* In this article: + * [Basic configuration](#basic-configuration) + * [Composite foreign keys](#composite-foreign-keys) + * [Linked vs embedded](#linked-vs-embedded) + + + + + +`CdcSinkLinkedTableConfig` is added to the `LinkedTables` list of a root or embedded table configuration: + + + +```csharp +new CdcSinkTableConfig +{ + CollectionName = "Orders", + SourceTableName = "orders", + PrimaryKeyColumns = ["id"], + Columns = + [ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + ], + LinkedTables = + [ + new CdcSinkLinkedTableConfig + { + SourceTableName = "customers", + PropertyName = "Customer", // Property name in document + LinkedCollectionName = "Customers", // Target collection for ID + JoinColumns = ["customer_id"] // FK used to build the document ID + } + ] +} +``` + + + +With `customer_id = 42`, the Orders document gets: + + + +```json +{ + "Id": 1, + "Customer": "Customers/42" +} +``` + + + +If all join column values are `null`, CDC Sink sets the link property to `null`. + +The `Customer` property is a RavenDB document ID built from `LinkedCollectionName` and the source-row values in `JoinColumns`. +The join column values are appended to `LinkedCollectionName` in the order listed. +Use the same order as the referenced table's `PrimaryKeyColumns`, so the generated ID matches the referenced document ID. + +This configuration only writes the `"Customers/42"` reference into the Orders document. +The `Customers/42` document itself exists only if the `customers` table is also configured as a root table. + +When loading the parent document, use [includes](../../../../../client-api/session/loading-entities.mdx#load-with-includes) +to fetch the referenced document without a second network call. + + + + + +For a reference to a table with a composite primary key, +list the matching foreign-key columns in `JoinColumns` in the same order: + + + +```csharp +new CdcSinkLinkedTableConfig +{ + SourceTableName = "customers", + PropertyName = "Customer", + LinkedCollectionName = "Customers", + + // Must match the referenced table's PK order + JoinColumns = ["customer_region", "customer_id"] +} +``` + + + +With `customer_region = 'US'` and `customer_id = 42`, the document gets: + +``` +"Customer": "Customers/US/42" +``` + + + + + +| Consideration | Embedded | Linked | +|---------------------|-------------------------------------------------|--------| +| Data stored | Full nested object/array inside document | Document ID reference only | +| Load pattern | Single document load | Load parent, then include/load referenced documents | +| Document size | Grows with embedded items | Parent document stays small | +| Related-row updates | Update the embedded data in the parent document | Update the referenced document only when that table is also configured as a root table | +| Relationship model | Related data is owned by the parent | Referenced entity exists independently | +| Typical use | Orders own LineItems | Orders reference Customers | + +--- + + + +* **Choose embedded** when the child data belongs to the parent and is always read together with it. + +* **Choose linked** when the referenced entity is independently meaningful and shared across many parents, + and you want to avoid duplicating data. + + + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_schema-design-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_schema-design-csharp.mdx new file mode 100644 index 0000000000..6b5d3bceb4 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/document-modeling/content/_schema-design-csharp.mdx @@ -0,0 +1,380 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; +import ContentFrame from '@site/src/components/ContentFrame'; + + + +* CDC Sink maps a relational schema to a RavenDB document model through configuration. + +* This article explains the three building blocks: + root tables, embedded tables, and linked tables - and how to combine them. + +* In this article: + * [Root tables](#root-tables) + * [Embedded tables](#embedded-tables) + * [Linked tables](#linked-tables) + * [Primary key and join column requirements](#primary-key-and-join-column-requirements) + * [Multi-level nesting](#multi-level-nesting) + * [Relation types](#relation-types) + * [Choosing between embedded and linked](#choosing-between-embedded-and-linked) + + + + + +A **root table** maps a source table to a RavenDB collection. Each row in the source table becomes one document. + +Each root table is configured with a `CdcSinkTableConfig` object. +This configuration defines the source table to read from, the target RavenDB collection to write to, +the primary key columns used to build document IDs, and the columns to map into each document. + + + +```csharp +new CdcSinkTableConfig +{ + CollectionName = "Orders", // RavenDB collection name + SourceTableSchema = "public", // Source schema (optional; default is provider-specific) + SourceTableName = "orders", // Source table name + PrimaryKeyColumns = ["id"], // Used for document ID generation + Columns = + [ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "customer_name", Name = "CustomerName" }, + new CdcColumnMapping() { Column = "total", Name = "Total" }, + ] +} +``` + + + + + +The default `SourceTableSchema` is provider-specific: PostgreSQL uses `public`, SQL Server uses `dbo`, +and MySQL/MariaDB use the database name from the connection string. + + + +**Document ID generation:** +CDC Sink generates document IDs in this format: +`{CollectionName}/{primary-key-value-1}/{primary-key-value-2}/...` + +The braces indicate placeholders, not literal text. +For example, a row with `id = 42` and `CollectionName = "Orders"` becomes `Orders/42`. +A composite primary key `(region, id)` with values `(US, 42)` becomes `Orders/US/42`. + +CDC Sink uses the `CollectionName` verbatim as the document ID prefix. + +**Column mapping:** +Only columns listed in `Columns` are stored as direct mapped fields. + +Embedded and linked table configurations can add additional document properties, +and unmapped source columns can still be used for joins, links, and patch scripts. + + + + + +An **embedded table** creates nested data within a parent document. +For example, a `order_lines` table becomes an array inside each `Orders` document. + +Embedded tables are configured inside the root table's `CdcSinkTableConfig`. +The example below configures an `Orders` root document with a `Lines` embedded array from the `order_lines` table. + + + +```csharp +new CdcSinkTableConfig +{ + CollectionName = "Orders", + SourceTableName = "orders", + PrimaryKeyColumns = ["id"], + Columns = + [ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "customer_name", Name = "CustomerName" }, + ], + EmbeddedTables = + [ + new CdcSinkEmbeddedTableConfig + { + SourceTableName = "order_lines", + PropertyName = "Lines", // Property in parent document + Type = CdcSinkRelationType.Array, // Array of items + JoinColumns = ["order_id"], // FK referencing parent's PK + PrimaryKeyColumns = ["line_id"], // Used to match items on update/delete + Columns = + [ + new CdcColumnMapping() { Column = "line_id", Name = "LineId" }, + new CdcColumnMapping() { Column = "product", Name = "Product" }, + new CdcColumnMapping() { Column = "quantity", Name = "Quantity" }, + ] + } + ] +} +``` + + + +This produces documents like: + + + +```json +{ + "Id": 1, + "CustomerName": "Alice", + "Lines": [ + { "LineId": 1, "Product": "Apples", "Quantity": 5 }, + { "LineId": 2, "Product": "Bananas", "Quantity": 3 } + ], + "@metadata": { "@collection": "Orders" } +} +``` + + + + + + + +A **linked table** creates a document ID reference rather than embedding data. +A foreign key in the source row becomes a RavenDB document ID. + + + +```csharp +new CdcSinkTableConfig +{ + CollectionName = "Orders", + SourceTableName = "orders", + PrimaryKeyColumns = ["id"], + Columns = + [ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "customer_id", Name = "CustomerId" }, + ], + LinkedTables = + [ + new CdcSinkLinkedTableConfig + { + SourceTableName = "customers", + PropertyName = "Customer", // Property in parent document + LinkedCollectionName = "Customers", // Target collection for ID + JoinColumns = ["customer_id"] // FK used to build the ID + } + ] +} +``` + + + +With `customer_id = 42`, the document gets `"Customer": "Customers/42"`. + +For composite references, list `JoinColumns` in the same order as the referenced table's `PrimaryKeyColumns`, +so the generated ID matches the referenced document ID. + +A linked table does NOT load or maintain the linked document by itself. +It only writes a document ID reference such as `Customers/42` into the current document. +The referenced `Customers/42` document is maintained only if `customers` is also configured as a root table. + + + + + + + +### Root tables + +The `PrimaryKeyColumns` list defines which source columns are used to generate the document ID. + +All PK columns must be present in every INSERT, UPDATE, and DELETE event. +Every column listed in `PrimaryKeyColumns` must also appear in the `Columns` list. + +In the samples, `Columns` is the list of `CdcColumnMapping` entries, +where `Column` is the source SQL column name and `Name` is the RavenDB document field name. + + + + +### Embedded tables (one level) + +An embedded table needs: + +* **PrimaryKeyColumns** - Used to match items within the parent's array or map for UPDATE and DELETE +* **JoinColumns** - Foreign key referencing the parent's `PrimaryKeyColumns` + +The `JoinColumns` must correspond 1:1 (same count and order) to the parent's `PrimaryKeyColumns`: + +| Parent PK | Required JoinColumns | Valid? | +|--------------|-----------------------------------------------|--------| +| `[id]` | `[order_id]` where `order_id` = parent's `id` | ✓ | +| `[id, year]` | `[order_id, order_year]` | ✓ | +| `[id]` | `[customer_id, order_id]` | ✗ Extra column not from parent PK | +| `[id, year]` | `[order_id]` | ✗ Missing `order_year` | + + + + +### DELETE events and REPLICA IDENTITY (PostgreSQL) + +For DELETE events, the source database must include the join column values so CDC Sink can route the delete to the correct parent document. + +If the join column is not part of the source table's primary key, +the source database may need additional configuration to include it in DELETE events. +For PostgreSQL, this is controlled by REPLICA IDENTITY. + +See [REPLICA IDENTITY](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx) for +the PostgreSQL-specific requirement and how CDC Sink handles it automatically. + + + + + + + +Embedded tables can themselves have embedded tables, creating arbitrarily deep hierarchies. + +**Example: Multi-level nesting with companies, departments, and employees** + + + +```csharp +new CdcSinkTableConfig +{ + CollectionName = "Companies", + SourceTableName = "companies", + PrimaryKeyColumns = ["company_id"], + Columns = + [ + new CdcColumnMapping() { Column = "company_id", Name = "CompanyId" }, + new CdcColumnMapping() { Column = "name", Name = "Name" }, + ], + EmbeddedTables = + [ + new CdcSinkEmbeddedTableConfig + { + SourceTableName = "departments", + PropertyName = "Departments", + Type = CdcSinkRelationType.Array, + JoinColumns = ["company_id"], // Root FK + PrimaryKeyColumns = ["dept_id"], + Columns = + [ + new CdcColumnMapping() { Column = "dept_id", Name = "DeptId" }, + new CdcColumnMapping() { Column = "dept_name", Name = "DeptName" }, + ], + EmbeddedTables = + [ + new CdcSinkEmbeddedTableConfig + { + SourceTableName = "employees", + PropertyName = "Employees", + Type = CdcSinkRelationType.Array, + JoinColumns = ["dept_id"], // FK to the direct parent (departments) PK + PrimaryKeyColumns = ["emp_id"], + Columns = + [ + new CdcColumnMapping() { Column = "emp_id", Name = "EmpId" }, + new CdcColumnMapping() { Column = "emp_name", Name = "EmpName" }, + ] + } + ] + } + ] +} +``` + + + +**Critical requirement for deep nesting:** +All descendant tables must carry the root-level join columns as denormalized columns. +These are the columns from the first embedded table's `JoinColumns`, which correspond to the root table's `PrimaryKeyColumns`. + +In this example, `departments.JoinColumns = ["company_id"]`, +so the `employees` table must also include `company_id` even though it joins directly to `departments` via `dept_id`. + +CDC Sink needs these values to route each descendant row to the correct root document in a single pass, +without additional lookups. + +**Source schema to support this:** + + + + +```sql +CREATE TABLE employees ( + company_id INT NOT NULL, -- Denormalized root FK + dept_id INT NOT NULL, -- Parent FK + emp_id INT NOT NULL, -- Local PK + emp_name VARCHAR(200), + PRIMARY KEY (company_id, dept_id, emp_id) +); +``` + + + + +The denormalized root-level join column values must be present in the source row, but they do not need to be listed in the nested table's `PrimaryKeyColumns`. + +For embedded tables, `PrimaryKeyColumns` identifies the item within its direct parent. + +For PostgreSQL, including all routing columns in the source table's primary key also avoids the need for REPLICA IDENTITY configuration - +the default DELETE events include all source primary-key columns. + + + + + +When defining a CDC Sink task, you choose how related SQL tables are represented in RavenDB: +embedded inside the parent document, or linked as references to separate documents. + +The `Type` property on embedded tables controls the document structure: + +| Type | Use Case | Document Structure | +|---------|----------------------------------------|--------------------| +| `Array` | One-to-many: parent has many children | `"Lines": [{ ... }, { ... }]` | +| `Map` | One-to-many with direct key lookup | `"Lines": { "1": { ... }, "2": { ... } }` | +| `Value` | One-to-one: parent owns a single child | `"ShippingInfo": { ... }` | + +**Array** +Items are matched by `PrimaryKeyColumns` for UPDATE and DELETE. +Use when you need to iterate over all items. + +**Map** +Items are stored as a JSON object keyed by the primary key value(s). +Use when you need fast direct-key access within the document. + +**Value** +Stores a single embedded object. +Use for a single child object owned by the parent, such as an order's shipping info or billing address. + +--- + +The following table summarizes how embedded and linked tables compare: + +| Consideration | Embedded | Linked | +|----------------|------------------------------------------|--------| +| Data location | Stored inside parent document | Stored in a separate document | +| Access pattern | Read parent to get all data | Load parent, then load referenced doc | +| Updates | Automatic via CDC | Only if the referenced table is also configured as a root table | +| Document size | Grows with embedded items | Parent stays small | +| Use case | Parent owns child
(orders own lines) | Independent entities (orders reference customers) | + + + + + +**Use embedded tables** when: +* The child entity has no meaning outside the parent (order lines without an order) +* You always read the parent and child together +* You want a single-document read + +**Use linked tables** when: +* The referenced entity is independently meaningful (customers exist without orders) +* The referenced entity is shared by many parents +* You want to load referenced documents with RavenDB includes + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/document-modeling/embedded-tables.mdx b/docs/server/ongoing-tasks/cdc-sink/document-modeling/embedded-tables.mdx new file mode 100644 index 0000000000..736dfbd288 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/document-modeling/embedded-tables.mdx @@ -0,0 +1,47 @@ +--- +title: "CDC Sink: Embedded Tables" +sidebar_label: Embedded Tables +description: "Covers configuration options, nesting constraints, and how embedded items are updated and deleted in CDC Sink." +sidebar_position: 2 +supported_languages: ["csharp"] +see_also: + - title: "Schema Design" + link: "server/ongoing-tasks/cdc-sink/document-modeling/schema-design" + source: "docs" + path: "Server > CDC Sink" + - title: "Linked Tables" + link: "server/ongoing-tasks/cdc-sink/document-modeling/linked-tables" + source: "docs" + path: "Server > CDC Sink" + - title: "Handling Deletes" + link: "server/ongoing-tasks/cdc-sink/handling-deletes" + source: "docs" + path: "Server > CDC Sink" + - title: "Attachment Handling" + link: "server/ongoing-tasks/cdc-sink/document-modeling/attachment-handling" + source: "docs" + path: "Server > CDC Sink" + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" + - title: "REPLICA IDENTITY" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "REPLICA IDENTITY Manual Setup" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity-manual-setup" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import EmbeddedTablesCsharp from './content/_embedded-tables-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/document-modeling/linked-tables.mdx b/docs/server/ongoing-tasks/cdc-sink/document-modeling/linked-tables.mdx new file mode 100644 index 0000000000..b34ae07445 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/document-modeling/linked-tables.mdx @@ -0,0 +1,35 @@ +--- +title: "CDC Sink: Linked Tables" +sidebar_label: Linked Tables +description: "Explains how linked tables create document ID references in parent documents by converting foreign key values into RavenDB document IDs." +sidebar_position: 3 +supported_languages: ["csharp"] +see_also: + - title: "Schema Design" + link: "server/ongoing-tasks/cdc-sink/document-modeling/schema-design" + source: "docs" + path: "Server > CDC Sink" + - title: "Embedded Tables" + link: "server/ongoing-tasks/cdc-sink/document-modeling/embedded-tables" + source: "docs" + path: "Server > CDC Sink" + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" + - title: "Load with Includes" + link: "client-api/session/loading-entities#load-with-includes" + source: "docs" + path: "Client API > Session" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import LinkedTablesCsharp from './content/_linked-tables-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/document-modeling/schema-design.mdx b/docs/server/ongoing-tasks/cdc-sink/document-modeling/schema-design.mdx new file mode 100644 index 0000000000..7fe969872c --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/document-modeling/schema-design.mdx @@ -0,0 +1,39 @@ +--- +title: "CDC Sink: Schema Design" +sidebar_label: Schema Design +description: "Explains how CDC Sink maps a relational schema to a RavenDB document model using root tables, embedded tables, and linked tables." +sidebar_position: 1 +supported_languages: ["csharp"] +see_also: + - title: "Embedded Tables" + link: "server/ongoing-tasks/cdc-sink/document-modeling/embedded-tables" + source: "docs" + path: "Server > CDC Sink" + - title: "Linked Tables" + link: "server/ongoing-tasks/cdc-sink/document-modeling/linked-tables" + source: "docs" + path: "Server > CDC Sink" + - title: "Column Mapping" + link: "server/ongoing-tasks/cdc-sink/document-modeling/column-mapping" + source: "docs" + path: "Server > CDC Sink" + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" + - title: "REPLICA IDENTITY" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import SchemaDesignCsharp from './content/_schema-design-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/_category_.json b/docs/server/ongoing-tasks/cdc-sink/examples/_category_.json new file mode 100644 index 0000000000..786b939e4c --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/examples/_category_.json @@ -0,0 +1,4 @@ +{ + "position": 13, + "label": "Examples" +} diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-complex-nesting-0.png b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-complex-nesting-0.png new file mode 100644 index 0000000000..d8fc28b416 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-complex-nesting-0.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-complex-nesting-1.png b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-complex-nesting-1.png new file mode 100644 index 0000000000..64bf40ab11 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-complex-nesting-1.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-complex-nesting-2.png b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-complex-nesting-2.png new file mode 100644 index 0000000000..4d6038788b Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-complex-nesting-2.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-denormalization-0.png b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-denormalization-0.png new file mode 100644 index 0000000000..632915e91a Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-denormalization-0.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-denormalization-1.png b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-denormalization-1.png new file mode 100644 index 0000000000..f767808372 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-denormalization-1.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-denormalization-2.png b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-denormalization-2.png new file mode 100644 index 0000000000..8c183931ec Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-denormalization-2.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-running-balance-0.png b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-running-balance-0.png new file mode 100644 index 0000000000..53109c4718 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-running-balance-0.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-running-balance-1.png b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-running-balance-1.png new file mode 100644 index 0000000000..30e1dd4f01 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-running-balance-1.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-running-balance-2.png b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-running-balance-2.png new file mode 100644 index 0000000000..9c9cee6757 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-running-balance-2.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-running-balance-3.png b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-running-balance-3.png new file mode 100644 index 0000000000..e87791d87d Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-running-balance-3.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-simple-migration-1.png b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-simple-migration-1.png new file mode 100644 index 0000000000..5cb38cf488 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-simple-migration-1.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-simple-migration-2.png b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-simple-migration-2.png new file mode 100644 index 0000000000..16eec2b7cf Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/example-simple-migration-2.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-complex-nesting-0.snagx b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-complex-nesting-0.snagx new file mode 100644 index 0000000000..decfc17eb4 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-complex-nesting-0.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-complex-nesting-1.snagx b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-complex-nesting-1.snagx new file mode 100644 index 0000000000..adb335f745 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-complex-nesting-1.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-complex-nesting-2.snagx b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-complex-nesting-2.snagx new file mode 100644 index 0000000000..79bf80c676 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-complex-nesting-2.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-denormaliztion-0.snagx b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-denormaliztion-0.snagx new file mode 100644 index 0000000000..7360fcbd13 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-denormaliztion-0.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-denormaliztion-1.snagx b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-denormaliztion-1.snagx new file mode 100644 index 0000000000..24183a4abf Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-denormaliztion-1.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-denormaliztion-2.snagx b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-denormaliztion-2.snagx new file mode 100644 index 0000000000..87a2a0c08e Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-denormaliztion-2.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-running-balance-0.snagx b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-running-balance-0.snagx new file mode 100644 index 0000000000..de24225e4a Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-running-balance-0.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-running-balance-1.snagx b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-running-balance-1.snagx new file mode 100644 index 0000000000..3a5e5e6133 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-running-balance-1.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-running-balance-2.snagx b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-running-balance-2.snagx new file mode 100644 index 0000000000..4af14266e1 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-running-balance-2.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-running-balance-3.snagx b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-running-balance-3.snagx new file mode 100644 index 0000000000..b344c3c5d1 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-running-balance-3.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-simple-migration-1.snagx b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-simple-migration-1.snagx new file mode 100644 index 0000000000..83c25acc68 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-simple-migration-1.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-simple-migration-2.snagx b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-simple-migration-2.snagx new file mode 100644 index 0000000000..c97985bdc6 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/examples/assets/snagit/example-simple-migration-2.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/content/_example-complex-nesting-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/examples/content/_example-complex-nesting-csharp.mdx new file mode 100644 index 0000000000..a7f34e22a0 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/examples/content/_example-complex-nesting-csharp.mdx @@ -0,0 +1,464 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* This example shows how to combine deep **embedded-table nesting** with **linked table references** + in one CDC Sink task. + +* It models a product catalog as RavenDB documents, with `products` as the root table, + nested `product_variants` and `variant_attributes` arrays, and a linked `categories` document reference. + +* For detailed instructions on creating a CDC Sink task with the Client API or Studio, + see [Create a CDC Sink task](../../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task.mdx). + +* In this article: + * [Source schema](#source-schema) + * [REPLICA IDENTITY setup](#replica-identity-setup) + * [Task configuration](#task-configuration) + * [Via the Client API](#via-the-client-api) + * [Via Studio](#via-studio) + * [Resulting documents](#resulting-documents) + + + + + +This schema uses `products` as the root table for `Products` documents. + +`categories` is synced as a separate root table and referenced from each product, +while `product_variants` and `variant_attributes` are embedded under the product document. + +`variant_attributes` includes `product_id` even though it joins directly to `product_variants`: +CDC Sink needs the root primary key on descendant embedded rows to locate the root document. + + + +```sql +CREATE TABLE categories ( + category_id SERIAL PRIMARY KEY, + name TEXT NOT NULL +); + +CREATE TABLE products ( + product_id SERIAL PRIMARY KEY, + name TEXT NOT NULL, + category_id INT REFERENCES categories(category_id) +); + +CREATE TABLE product_variants ( + variant_id SERIAL PRIMARY KEY, + product_id INT NOT NULL REFERENCES products(product_id), + sku TEXT NOT NULL, + price NUMERIC(10,2) +); + +CREATE TABLE variant_attributes ( + attr_id SERIAL PRIMARY KEY, + -- Denormalized root primary key, required for deep nesting + product_id INT NOT NULL REFERENCES products(product_id), + variant_id INT NOT NULL REFERENCES product_variants(variant_id), + attr_name TEXT NOT NULL, + attr_value TEXT NOT NULL +); +``` + + + + + + + +* For delete events, CDC Sink must receive the columns it uses to route the change and identify the embedded item. + +* In this schema, both embedded tables use single-column primary keys that do not include their routing columns, + so PostgreSQL's default primary-key-only replica identity is not enough. + +* Instead of `REPLICA IDENTITY FULL`, which includes all columns, + this example uses targeted unique indexes that cover only the required routing and primary-key columns: + + + + ```sql + -- product_variants: + -- product_id locates the root Products document, variant_id identifies the embedded item + CREATE UNIQUE INDEX product_variants_replica_idx + ON product_variants (product_id, variant_id); + ALTER TABLE product_variants + REPLICA IDENTITY USING INDEX product_variants_replica_idx; + + -- variant_attributes: + -- product_id locates the root Products document, + -- variant_id locates the parent variant, + -- attr_id identifies the embedded attribute item + CREATE UNIQUE INDEX variant_attributes_replica_idx + ON variant_attributes (product_id, variant_id, attr_id); + ALTER TABLE variant_attributes + REPLICA IDENTITY USING INDEX variant_attributes_replica_idx; + ``` + + + + Learn more in [REPLICA IDENTITY](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx). + + + + + +Multi-level embedding follows these routing rules: + +* Each embedded table's `JoinColumns` points to the primary key of its direct parent. +* Every embedded row must also contain the root primary key (`product_id`) so CDC Sink can locate the root `Products` document. +* For deeper embedded tables, `JoinColumns` still names the direct-parent key. + Here, `variant_attributes.JoinColumns` is `variant_id`; + `product_id` is present only for root-document routing and does not need to be mapped as an embedded property. + +--- + +### Via the Client API + +Create the task with the Client API: + + + +```csharp +var config = new CdcSinkConfiguration +{ + Name = "ProductCatalog-CDC-Sink", + ConnectionStringName = "ConnectionStringToPostgreSQL", + Tables = + [ + new CdcSinkTableConfig + { + CollectionName = "Products", + SourceTableSchema = "public", + SourceTableName = "products", + PrimaryKeyColumns = ["product_id"], + Columns = + [ + new CdcColumnMapping() { Column = "product_id", Name = "ProductId" }, + new CdcColumnMapping() { Column = "name", Name = "Name" }, + ], + // Linked table: category_id FK → document ID in Categories collection + LinkedTables = + [ + new CdcSinkLinkedTableConfig + { + SourceTableSchema = "public", + SourceTableName = "categories", + PropertyName = "Category", + LinkedCollectionName = "Categories", + JoinColumns = ["category_id"] + } + ], + EmbeddedTables = + [ + new CdcSinkEmbeddedTableConfig + { + SourceTableSchema = "public", + SourceTableName = "product_variants", + PropertyName = "Variants", + Type = CdcSinkRelationType.Array, + JoinColumns = ["product_id"], + PrimaryKeyColumns = ["variant_id"], + Columns = + [ + new CdcColumnMapping() { Column = "variant_id", Name = "VariantId" }, + new CdcColumnMapping() { Column = "sku", Name = "Sku" }, + new CdcColumnMapping() { Column = "price", Name = "Price" }, + ], + // Deep-nested: attributes within each variant + EmbeddedTables = + [ + new CdcSinkEmbeddedTableConfig + { + SourceTableSchema = "public", + SourceTableName = "variant_attributes", + PropertyName = "Attributes", + Type = CdcSinkRelationType.Array, + // FK to the direct parent (product_variants) PK. + // The root PK (product_id) must exist as a denormalized column + // but is NOT listed here. + JoinColumns = ["variant_id"], + PrimaryKeyColumns = ["attr_id"], + Columns = + [ + new CdcColumnMapping() { Column = "attr_id", Name = "AttrId" }, + new CdcColumnMapping() { Column = "attr_name", Name = "Name" }, + new CdcColumnMapping() { Column = "attr_value", Name = "Value" }, + ] + } + ] + } + ] + }, + // Sync the linked target as its own root table. + // Without this, this task writes the Category ID string + // but does not create the referenced Categories document. + new CdcSinkTableConfig + { + CollectionName = "Categories", + SourceTableSchema = "public", + SourceTableName = "categories", + PrimaryKeyColumns = ["category_id"], + Columns = + [ + new CdcColumnMapping() { Column = "category_id", Name = "CategoryId" }, + new CdcColumnMapping() { Column = "name", Name = "Name" }, + ] + } + ] +}; + +await store.Maintenance.SendAsync(new AddCdcSinkOperation(config)); +``` + + + +--- + +### Via Studio + +In Studio, configure how the `products` source table maps to the `Products` target collection, +add `categories` as a **linked reference**, and nest `product_variants` and `variant_attributes` as **embedded tables**. + +The left **Tables** tree shows the full structure: the root `products` table, its linked `categories` reference, +the embedded `product_variants` → `variant_attributes` chain, and the standalone `categories` root table. + +--- + +First, configure the root `products` table: + +![Configure tables - products table](../assets/example-complex-nesting-0.png) + +1. Select the root source table to configure. +2. **Source table → target collection** + The `products` table maps to the `Products` collection. + This is a root table, so each row becomes its own document. +3. **Primary key columns** + `product_id` is used to build the document ID for each product. +4. **Field mapping** + Maps each source column to its target property + (`product_id` → `ProductId`, `name` → `Name`). +5. **Linked and embedded children** + The linked `categories` reference and the embedded `product_variants` tree are attached to this root table + and appear as child nodes under `products` in the Tables tree. + +--- + +Then add the linked `categories` reference: + +![Configure tables - linked categories table](../assets/example-complex-nesting-1.png) + +1. **Linked table** + `categories` is added as a **linked reference** on the `products` table, not as an embedded table. + It stores a document ID string on the product document rather than nesting a copy of the category. +2. **Target property** + The reference is written to the `Category` property of each product document. +3. **Linked collection** + `Categories` - the reference resolves to IDs such as `Categories/3`. +4. **Join columns** + `category_id` - the foreign key on the product row used to build the linked document ID. +5. **Referenced table as a root table** + The lower `categories` node is configured separately as a root table, not as part of the linked reference. + This lets CDC Sink create `Categories/` documents that the `Category` link can point to and include. + Learn more in [Linked tables](../../../../../server/ongoing-tasks/cdc-sink/document-modeling/linked-tables.mdx). + +--- + + + +#### Embedded tables + +* The embedded `product_variants` and `variant_attributes` tables (the `{+}` nodes in the tree) are configured by + selecting their node in the Tables tree and setting the relation type, primary key, join columns, and field mapping. + +* For an example that shows the embedded-table fields in Studio, see + [Denormalization with Embedded Tables](../../../../../server/ongoing-tasks/cdc-sink/examples/example-denormalization.mdx). + For the full configuration reference, see [Embedded Tables](../../../../../server/ongoing-tasks/cdc-sink/document-modeling/embedded-tables.mdx). + +#### Deep nesting is configured the same way + +* `variant_attributes` is added as an embedded table *inside* `product_variants`, + just as `product_variants` is added inside `products` - one level deeper. + +* The extra requirement is that descendant embedded rows carry the denormalized root primary key (`product_id`) + so CDC Sink can locate the root document. + + + +--- + +Alternatively, toggle **Raw config** and paste the equivalent JSON: + +![Use raw config](../assets/example-complex-nesting-2.png) + + + +```json +{ + "Name": "ProductCatalog-CDC-Sink", + "ConnectionStringName": "ConnectionStringToPostgreSQL", + "Tables": [ + { + "CollectionName": "Products", + "SourceTableSchema": "public", + "SourceTableName": "products", + "PrimaryKeyColumns": ["product_id"], + "Columns": [ + { "Column": "product_id", "Name": "ProductId" }, + { "Column": "name", "Name": "Name" } + ], + "LinkedTables": [ + { + "SourceTableSchema": "public", + "SourceTableName": "categories", + "PropertyName": "Category", + "LinkedCollectionName": "Categories", + "JoinColumns": ["category_id"] + } + ], + "EmbeddedTables": [ + { + "SourceTableSchema": "public", + "SourceTableName": "product_variants", + "PropertyName": "Variants", + "Type": "Array", + "JoinColumns": ["product_id"], + "PrimaryKeyColumns": ["variant_id"], + "Columns": [ + { "Column": "variant_id", "Name": "VariantId" }, + { "Column": "sku", "Name": "Sku" }, + { "Column": "price", "Name": "Price" } + ], + "EmbeddedTables": [ + { + "SourceTableSchema": "public", + "SourceTableName": "variant_attributes", + "PropertyName": "Attributes", + "Type": "Array", + "JoinColumns": ["variant_id"], + "PrimaryKeyColumns": ["attr_id"], + "Columns": [ + { "Column": "attr_id", "Name": "AttrId" }, + { "Column": "attr_name", "Name": "Name" }, + { "Column": "attr_value", "Name": "Value" } + ] + } + ] + } + ] + }, + { + "CollectionName": "Categories", + "SourceTableSchema": "public", + "SourceTableName": "categories", + "PrimaryKeyColumns": ["category_id"], + "Columns": [ + { "Column": "category_id", "Name": "CategoryId" }, + { "Column": "name", "Name": "Name" } + ] + } + ] +} +``` + + + + + + + +Given these source rows: + +**`categories`** + +| category_id | name | +|-------------|----------| +| 3 | Footwear | + +**`products`** + +| product_id | name | category_id | +|------------|-------------|-------------| +| 42 | Hiking Boot | 3 | + +**`product_variants`** + +| variant_id | product_id | sku | price | +|------------|------------|-----------|-------| +| 101 | 42 | HB-BLK-10 | 89.99 | +| 102 | 42 | HB-BRN-11 | 89.99 | + +**`variant_attributes`** + +| attr_id | product_id | variant_id | attr_name | attr_value | +|---------|------------|------------|-----------|------------| +| 1 | 42 | 101 | Color | Black | +| 2 | 42 | 101 | Size | 10 | +| 3 | 42 | 102 | Color | Brown | +| 4 | 42 | 102 | Size | 11 | + +
+ +CDC Sink creates the root RavenDB document `Products/42`: + + + +```json +{ + "ProductId": 42, + "Name": "Hiking Boot", + "Category": "Categories/3", + "Variants": [ + { + "VariantId": 101, + "Sku": "HB-BLK-10", + "Price": 89.99, + "Attributes": [ + { "AttrId": 1, "Name": "Color", "Value": "Black" }, + { "AttrId": 2, "Name": "Size", "Value": "10" } + ] + }, + { + "VariantId": 102, + "Sku": "HB-BRN-11", + "Price": 89.99, + "Attributes": [ + { "AttrId": 3, "Name": "Color", "Value": "Brown" }, + { "AttrId": 4, "Name": "Size", "Value": "11" } + ] + } + ], + "@metadata": { "@collection": "Products" } +} +``` + + + +The standalone `categories` root table creates `Categories/3`: + + + +```json +{ + "CategoryId": 3, + "Name": "Footwear", + "@metadata": { "@collection": "Categories" } +} +``` + + + + + +* The `Category` property is a document ID reference, not an embedded copy of the category row. + +* Because `categories` is also configured as a root table, CDC Sink creates the referenced `Categories/3` document, + which can be included when loading or querying products. + + + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/content/_example-denormalization-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/examples/content/_example-denormalization-csharp.mdx new file mode 100644 index 0000000000..4109a383ff --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/examples/content/_example-denormalization-csharp.mdx @@ -0,0 +1,309 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; +import ContentFrame from "@site/src/components/ContentFrame"; + + + +* This example shows how to model a normalized PostgreSQL `orders` + `order_lines` schema + as RavenDB `Orders` documents, each with an embedded `Lines` array. + +* For detailed instructions on creating a CDC Sink task with the Client API or Studio, + see [Create a CDC Sink task](../../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task.mdx). + +* In this article: + * [Source schema](#source-schema) + * [REPLICA IDENTITY setup](#replica-identity-setup) + * [Task configuration](#task-configuration) + * [Via the Client API](#via-the-client-api) + * [Via Studio](#via-studio) + * [Resulting documents](#resulting-documents) + * [What happens on change events](#what-happens-on-change-events) + + + + + +The `orders` table is the root table. +The `order_lines` table stores line items; each row belongs to an order through the `order_id` foreign key. +CDC Sink later uses this relationship to embed each order's lines in the `Lines` array. + + + + +```sql +CREATE TABLE orders ( + order_id SERIAL PRIMARY KEY, + customer TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'pending', + created_at TIMESTAMPTZ DEFAULT now() +); + +CREATE TABLE order_lines ( + line_id SERIAL PRIMARY KEY, + order_id INT NOT NULL REFERENCES orders(order_id), + product TEXT NOT NULL, + qty INT NOT NULL, + unit_price NUMERIC(10,2) NOT NULL +); +``` + + + + + + + +* `order_lines` uses `line_id` as its primary key. + The `order_id` column points to the parent `orders` row, but it is not part of the primary key. + +* With PostgreSQL's default replica identity, DELETE events for `order_lines` include only `line_id`. + CDC Sink also needs `order_id` to find the parent `Orders` document before removing the embedded line. + +* Setting `REPLICA IDENTITY FULL` on `order_lines` makes PostgreSQL include the full old row in DELETE events, + including `order_id`. + + + + ```sql + ALTER TABLE order_lines REPLICA IDENTITY FULL; + ``` + + + + Learn more in [REPLICA IDENTITY](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx). + + + + + +### Via the Client API + +Define `orders` as the **root** table. +Then define `order_lines` as an **embedded** table stored in the `Lines` array: + + + +```csharp +var config = new CdcSinkConfiguration +{ + Name = "Orders-CDC-Sink", + ConnectionStringName = "ConnectionStringToPostgreSQL", + Tables = new List + { + new CdcSinkTableConfig + { + CollectionName = "Orders", + SourceTableSchema = "public", + SourceTableName = "orders", + PrimaryKeyColumns = new List { "order_id" }, + Columns = + [ + new CdcColumnMapping() { Column = "order_id", Name = "OrderId" }, + new CdcColumnMapping() { Column = "customer", Name = "Customer" }, + new CdcColumnMapping() { Column = "status", Name = "Status" }, + new CdcColumnMapping() { Column = "created_at", Name = "CreatedAt" }, + ], + EmbeddedTables = new List + { + new CdcSinkEmbeddedTableConfig + { + SourceTableSchema = "public", + SourceTableName = "order_lines", + PropertyName = "Lines", + Type = CdcSinkRelationType.Array, + JoinColumns = new List { "order_id" }, + PrimaryKeyColumns = new List { "line_id" }, + Columns = + [ + new CdcColumnMapping() { Column = "line_id", Name = "LineId" }, + new CdcColumnMapping() { Column = "product", Name = "Product" }, + new CdcColumnMapping() { Column = "qty", Name = "Qty" }, + new CdcColumnMapping() { Column = "unit_price", Name = "UnitPrice" }, + ] + } + } + } + } +}; + +await store.Maintenance.SendAsync(new AddCdcSinkOperation(config)); +``` + + + +--- + +### Via Studio + +In Studio, first configure how the `orders` source table maps to the `Orders` target collection. +Then add `order_lines` as an embedded table inside each `Orders` document. + +--- + +First, configure the root `orders` table: + +![Configure tables - orders table](../assets/example-denormalization-0.png) + +1. Select the root source table to configure. +2. **Source table → target collection** + The `orders` table maps to the `Orders` collection. + This is a root table, so each row becomes its own document. +3. **Primary key columns** + `order_id` is used to build the document ID for each order (`Orders/`). +4. **Field mapping** + Maps each source column to its target property + (`order_id` → `OrderId`, `customer` → `Customer`, `status` → `Status`, `created_at` → `CreatedAt`). + +--- + +Then add the embedded `order_lines` table: + +![Configure tables - order_lines table](../assets/example-denormalization-1.png) + +1. **Embedded table** + `order_lines` is configured as an embedded table inside each `Orders` document, not as its own collection. +2. **Relation type** + Each `Orders` document stores its lines as a JSON array (`Type: Array` in the config). +3. **Primary key columns** + `line_id` identifies each array element, so later updates and deletes affect the correct line. +4. **Join columns** + `order_id` links each line row to its parent `Orders` document. + In this example, PostgreSQL DELETE events need this value to route deleted lines correctly. +5. **Field mapping** + Maps each source column to its target property, matching the `Columns` list in the config. + +--- + +Alternatively, toggle **Raw config** and paste the equivalent JSON: + +![Use raw config](../assets/example-denormalization-2.png) + + + +```json +{ + "Name": "Orders-CDC-Sink", + "ConnectionStringName": "ConnectionStringToPostgreSQL", + "Tables": [ + { + "CollectionName": "Orders", + "SourceTableSchema": "public", + "SourceTableName": "orders", + "PrimaryKeyColumns": ["order_id"], + "Columns": [ + { "Column": "order_id", "Name": "OrderId" }, + { "Column": "customer", "Name": "Customer" }, + { "Column": "status", "Name": "Status" }, + { "Column": "created_at", "Name": "CreatedAt" } + ], + "EmbeddedTables": [ + { + "SourceTableSchema": "public", + "SourceTableName": "order_lines", + "PropertyName": "Lines", + "Type": "Array", + "JoinColumns": ["order_id"], + "PrimaryKeyColumns": ["line_id"], + "Columns": [ + { "Column": "line_id", "Name": "LineId" }, + { "Column": "product", "Name": "Product" }, + { "Column": "qty", "Name": "Qty" }, + { "Column": "unit_price", "Name": "UnitPrice" } + ] + } + ] + } + ] +} +``` + + + + + + + +Assume the source tables contain these rows: + +```plain +orders: order_id=1, customer='Acme Corp', status='pending', created_at='2024-06-01 09:00:00+00' +order_lines: line_id=1, order_id=1, product='Widget A', qty=3, unit_price=9.99 +order_lines: line_id=2, order_id=1, product='Widget B', qty=1, unit_price=24.99 +``` +
+ +CDC Sink creates the RavenDB document `Orders/1`: + + + +```json +{ + "OrderId": 1, + "Customer": "Acme Corp", + "Status": "pending", + "CreatedAt": "2024-06-01T09:00:00+00:00", + "Lines": [ + { + "LineId": 1, + "Product": "Widget A", + "Qty": 3, + "UnitPrice": 9.99 + }, + { + "LineId": 2, + "Product": "Widget B", + "Qty": 1, + "UnitPrice": 24.99 + } + ], + "@metadata": { "@collection": "Orders" } +} +``` + + + + + + + + + +**INSERT into `order_lines`**: +A new item is appended to the `Lines` array. + + + + +**UPDATE to `order_lines`**: +If `order_id` stays the same, +CDC Sink finds the item by `line_id` within the `Lines` array and updates its properties. +If `order_id` changes, +CDC Sink removes the line from the old parent document and adds it to the new parent document. +This also relies on the [`REPLICA IDENTITY FULL` setup](#replica-identity-setup), so CDC Sink receives the old `order_id`. + + + + +**DELETE from `order_lines`**: +CDC Sink uses `order_id` to find the parent `Orders` document, +then uses `line_id` to remove the matching item from the `Lines` array. +This also relies on the [`REPLICA IDENTITY FULL` setup](#replica-identity-setup), so CDC Sink receives `order_id` from the deleted row. + + + + +**UPDATE to `orders`**: +Only the root document properties (`Status`, etc.) are updated. +The `Lines` array is not affected. + + + + +**DELETE from `orders`**: +The entire `Orders/1` document is deleted, including all embedded lines. + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/content/_example-running-balance-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/examples/content/_example-running-balance-csharp.mdx new file mode 100644 index 0000000000..489892685d --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/examples/content/_example-running-balance-csharp.mdx @@ -0,0 +1,316 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* This example shows how to use CDC Sink patches to **maintain a computed aggregate** on a RavenDB document + as individual event rows arrive from PostgreSQL. + +* For detailed instructions on creating a CDC Sink task with the Client API or Studio, + see [Create a CDC Sink task](../../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task.mdx). + +* In this article: + * [Source schema](#source-schema) + * [Goal](#goal) + * [REPLICA IDENTITY setup](#replica-identity-setup) + * [Task configuration](#task-configuration) + * [Via the Client API](#via-the-client-api) + * [Via Studio](#via-studio) + * [Resulting documents](#resulting-documents) + * [Handling deletes](#handling-deletes) + + + + + +An accounts table and a transactions table: + + + +```sql +CREATE TABLE accounts ( + account_id SERIAL PRIMARY KEY, + owner TEXT NOT NULL, + currency TEXT NOT NULL DEFAULT 'USD' +); + +CREATE TABLE transactions ( + txn_id SERIAL PRIMARY KEY, + account_id INT NOT NULL REFERENCES accounts(account_id), + amount NUMERIC(12,2) NOT NULL, + type TEXT NOT NULL CHECK (type IN ('credit', 'debit')), + created_at TIMESTAMPTZ DEFAULT now() +); +``` + + + + + + + +Store each account as a RavenDB document with a `Balance` field that reflects the running total of all transactions. +Transaction rows are embedded as an array for history, and `Balance` is maintained using patch logic. + + + + + +* The `transactions` table uses `txn_id` as its primary key. + The join column `account_id` is not part of that primary key. + +* By default, PostgreSQL `REPLICA IDENTITY` sends only primary-key columns. + Without `REPLICA IDENTITY FULL`, DELETE events for `transactions` rows would not include `account_id`, + and CDC Sink could not route the deleted transaction to the correct account document. + +* For this example to handle deleted transactions, CDC Sink must receive the `account_id` value in PostgreSQL DELETE events. + Set `REPLICA IDENTITY FULL` on the `transactions` table: + + + + ```sql + ALTER TABLE transactions REPLICA IDENTITY FULL; + ``` + + + + Learn more in [REPLICA IDENTITY](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx). + + + + + +### Via the Client API + +Define the task with the client API: + + + +```csharp +var config = new CdcSinkConfiguration +{ + Name = "Accounts-CDC-Sink", + ConnectionStringName = "ConnectionStringToPostgreSQL", + Tables = + [ + new CdcSinkTableConfig + { + CollectionName = "Accounts", + SourceTableSchema = "public", + SourceTableName = "accounts", + PrimaryKeyColumns = ["account_id"], + Columns = + [ + new CdcColumnMapping() { Column = "account_id", Name = "AccountId" }, + new CdcColumnMapping() { Column = "owner", Name = "Owner" }, + new CdcColumnMapping() { Column = "currency", Name = "Currency" }, + ], + EmbeddedTables = + [ + new CdcSinkEmbeddedTableConfig + { + SourceTableSchema = "public", + SourceTableName = "transactions", + PropertyName = "Transactions", + Type = CdcSinkRelationType.Array, + JoinColumns = ["account_id"], + PrimaryKeyColumns = ["txn_id"], + Columns = + [ + new CdcColumnMapping() { Column = "txn_id", Name = "TxnId" }, + new CdcColumnMapping() { Column = "amount", Name = "Amount" }, + new CdcColumnMapping() { Column = "type", Name = "Type" }, + new CdcColumnMapping() { Column = "created_at", Name = "CreatedAt" }, + ], + // Runs after a transaction is inserted or updated; + // adjusts Balance by replacing the previous contribution with the new one. + Patch = """ + const oldAmount = $old?.Amount || 0; + const newAmount = $row.amount || 0; + const sign = $row.type === 'credit' ? 1 : -1; + const oldSign = $old?.Type === 'credit' ? 1 : -1; + this.Balance = (this.Balance || 0) + - (oldSign * oldAmount) + + (sign * newAmount); + """, + // Runs after a transaction is deleted; + // uses $old to subtract the removed transaction's last stored contribution. + OnDelete = new CdcSinkOnDeleteConfig + { + Patch = """ + const deletedAmount = $old?.Amount || 0; + const sign = $old?.Type === 'credit' ? 1 : -1; + this.Balance = (this.Balance || 0) - (sign * deletedAmount); + """ + } + } + ] + } + ] +}; + +await store.Maintenance.SendAsync(new AddCdcSinkOperation(config)); +``` + + + +--- + +### Via Studio + +In Studio, configure how the `accounts` source table maps to the `Accounts` target collection, +then add `transactions` as an embedded table for each account. + +--- + +First, configure the root `accounts` table: + +![Configure tables - accounts table](../assets/example-running-balance-0.png) + +1. **Source table → target collection** + The `accounts` table maps to the `Accounts` collection. + This is a root table, so each row becomes its own document. +2. **Primary key columns** + `account_id` is used to build the document ID for each account. +3. **Field mapping** + Maps each source column to its target property + (`account_id` → `AccountId`, `owner` → `Owner`, `currency` → `Currency`). + +--- + +Then add the embedded `transactions` table: + +![Configure tables - transactions table](../assets/example-running-balance-1.png) + +1. **Embedded table** + `transactions` is configured as an embedded table inside the `accounts` document, not as its own collection. +2. **Relation type** + Each account stores its transactions as a JSON array (`Type: Array` in the config). +3. **Primary key columns** + `txn_id` identifies each array element, so later updates and deletes patch the correct transaction. +4. **Join columns** + `account_id` links each transaction row to its parent account document. + In this example, PostgreSQL DELETE events need this value to route deleted transactions correctly. +5. **Field mapping** + Maps each source column to its target property, matching the `Columns` list in the config. + +--- + +The embedded `transactions` table and its balance patches (`Patch` / `OnDelete.Patch`) are defined under **Advanced settings**. + +![Configure patches](../assets/example-running-balance-2.png) + +1. **Case sensitive keys** + Controls whether primary-key and map-key matching is case sensitive. + It only affects string keys, so it has no effect in this example, where the key is the numeric `txn_id`. +2. **Patch** + The INSERT/UPDATE patch that maintains `Balance`; + runs on the parent account document after a transaction is added or updated. +3. **Skip deletion** + Maps to `IgnoreDeletes`. + Leave this off for this example. + **Off** = CDC Sink removes the deleted transaction item from the array, and the delete patch updates `Balance`. + **On** = CDC Sink skips the automatic removal. If a delete patch is configured, the patch still runs. +4. **Delete patch** + The `OnDelete.Patch` script reverses the deleted transaction's contribution to `Balance`. + +--- + +Alternatively, toggle **Raw config** and paste the equivalent JSON: + +![Use raw config](../assets/example-running-balance-3.png) + + + +```json +{ + "Name": "Accounts-CDC-Sink", + "ConnectionStringName": "ConnectionStringToPostgreSQL", + "Tables": [ + { + "CollectionName": "Accounts", + "SourceTableSchema": "public", + "SourceTableName": "accounts", + "PrimaryKeyColumns": ["account_id"], + "Columns": [ + { "Column": "account_id", "Name": "AccountId" }, + { "Column": "owner", "Name": "Owner" }, + { "Column": "currency", "Name": "Currency" } + ], + "EmbeddedTables": [ + { + "SourceTableSchema": "public", + "SourceTableName": "transactions", + "PropertyName": "Transactions", + "Type": "Array", + "JoinColumns": ["account_id"], + "PrimaryKeyColumns": ["txn_id"], + "Columns": [ + { "Column": "txn_id", "Name": "TxnId" }, + { "Column": "amount", "Name": "Amount" }, + { "Column": "type", "Name": "Type" }, + { "Column": "created_at", "Name": "CreatedAt" } + ], + "Patch": "const oldAmount = $old?.Amount || 0; const newAmount = $row.amount || 0; const sign = $row.type === 'credit' ? 1 : -1; const oldSign = $old?.Type === 'credit' ? 1 : -1; this.Balance = (this.Balance || 0) - (oldSign * oldAmount) + (sign * newAmount);", + "OnDelete": { + "Patch": "const deletedAmount = $old?.Amount || 0; const sign = $old?.Type === 'credit' ? 1 : -1; this.Balance = (this.Balance || 0) - (sign * deletedAmount);" + } + } + ] + } + ] +} +``` + + + + + + + +After three transactions (credit 100, debit 30, credit 50): + + + +```json +{ + "AccountId": 1, + "Owner": "Alice", + "Currency": "USD", + "Balance": 120.00, + "Transactions": [ + { "TxnId": 1, "Amount": 100.00, "Type": "credit", "CreatedAt": "..." }, + { "TxnId": 2, "Amount": 30.00, "Type": "debit", "CreatedAt": "..." }, + { "TxnId": 3, "Amount": 50.00, "Type": "credit", "CreatedAt": "..." } + ], + "@metadata": { "@collection": "Accounts" } +} +``` + + + + + + + +The patches keep `Balance` correct by applying only the change caused by each transaction event: + +* **When a transaction row is inserted**: + There is no previous embedded item, so the patch adds the new transaction's contribution to `Balance`. + +* **When a transaction row is updated**: + The patch subtracts the previous contribution from `$old`, then adds the new contribution from `$row`. + +* **When a transaction row is deleted**: + There is no new row to add. + The `OnDelete.Patch` subtracts the deleted transaction's last stored contribution from `$old`. + +Without `OnDelete.Patch`, deleting a transaction row from SQL would remove it from the `Transactions` array +but leave `Balance` stale. + +This also keeps `Balance` correct if the same change is re-applied after failover. +Learn more in [Failover and consistency](../../../../../server/ongoing-tasks/cdc-sink/failover-and-consistency.mdx). + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/content/_example-simple-migration-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/examples/content/_example-simple-migration-csharp.mdx new file mode 100644 index 0000000000..8f943cb604 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/examples/content/_example-simple-migration-csharp.mdx @@ -0,0 +1,155 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* This example shows the basic CDC Sink configuration to migrate a single SQL table into a RavenDB collection. + +* For detailed instructions on creating a CDC Sink task with the Client API or Studio, + see [Create a CDC Sink task](../../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task.mdx). + +* In this article: + * [Source schema](#source-schema) + * [Task configuration](#task-configuration) + * [Via the Client API](#via-the-client-api) + * [Via Studio](#via-studio) + * [Resulting documents](#resulting-documents) + + + + + +A simple customers table: + + + +```sql +CREATE TABLE customers ( + id SERIAL PRIMARY KEY, + name TEXT NOT NULL, + email TEXT NOT NULL, + created_at TIMESTAMPTZ DEFAULT now() +); +``` + + + + + + + +### Via the Client API + +Define the task with the client API: + + + +```csharp +var config = new CdcSinkConfiguration +{ + Name = "Customers-CDC-Sink", + ConnectionStringName = "ConnectionStringToPostgreSQL", + Tables = new List + { + new CdcSinkTableConfig + { + CollectionName = "Customers", + SourceTableSchema = "public", + SourceTableName = "customers", + PrimaryKeyColumns = new List { "id" }, + // Column = source SQL column, Name = RavenDB document property + Columns = + [ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "name", Name = "Name" }, + new CdcColumnMapping() { Column = "email", Name = "Email" }, + new CdcColumnMapping() { Column = "created_at", Name = "CreatedAt" }, + ] + } + } +}; + +await store.Maintenance.SendAsync(new AddCdcSinkOperation(config)); +``` + + + +--- + +### Via Studio + +In Studio, configure how the selected `customers` source table maps to the `Customers` target collection: +set the source schema/table, primary key column, and field mapping (source column → target property). + +![Configure the customers table](../assets/example-simple-migration-1.png) + +--- + +Alternatively, toggle **Raw config** and paste the equivalent JSON: + +![Use raw config](../assets/example-simple-migration-2.png) + + + +```json +{ + "Name": "Customers-CDC-Sink", + "ConnectionStringName": "ConnectionStringToPostgreSQL", + "Tables": [ + { + "CollectionName": "Customers", + "SourceTableSchema": "public", + "SourceTableName": "customers", + "PrimaryKeyColumns": ["id"], + "Columns": [ + { "Column": "id", "Name": "Id" }, + { "Column": "name", "Name": "Name" }, + { "Column": "email", "Name": "Email" }, + { "Column": "created_at", "Name": "CreatedAt" } + ] + } + ] +} +``` + + + + + + + +Given this SQL row: + +```plain +id=1, name='Alice', email='alice@example.com', created_at='2024-01-15 10:30:00+00' +``` +
+ +CDC Sink generates the document ID from `CollectionName` and the configured primary key column value. +Here, `CollectionName = "Customers"` and `PrimaryKeyColumns = ["id"]`, so the document ID is `Customers/1`. + +The `id` column must also be mapped in `Columns`; otherwise, the task configuration is rejected. +In this example, it is mapped to the `Id` document property. + + +The resulting RavenDB document: + + + +```json +{ + "Id": 1, + "Name": "Alice", + "Email": "alice@example.com", + "CreatedAt": "2024-01-15T10:30:00+00:00", + "@metadata": { + "@collection": "Customers" + } +} +``` + + + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/example-complex-nesting.mdx b/docs/server/ongoing-tasks/cdc-sink/examples/example-complex-nesting.mdx new file mode 100644 index 0000000000..8fbe54a361 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/examples/example-complex-nesting.mdx @@ -0,0 +1,47 @@ +--- +title: "Complex Nesting with Linked Tables" +sidebar_label: "Complex Nesting with Linked Tables" +description: "Multi-level embedded table structure combined with linked table references, representing a product catalog with variants, attributes, and a category reference." +sidebar_position: 3 +supported_languages: ["csharp"] +see_also: + - title: "Simple Migration" + link: "server/ongoing-tasks/cdc-sink/examples/example-simple-migration" + source: "docs" + path: "Server > CDC Sink > PostgreSQL > Examples" + - title: "Denormalization" + link: "server/ongoing-tasks/cdc-sink/examples/example-denormalization" + source: "docs" + path: "Server > CDC Sink > PostgreSQL > Examples" + - title: "Runnning Balance" + link: "server/ongoing-tasks/cdc-sink/examples/example-running-balance" + source: "docs" + path: "Server > CDC Sink > PostgreSQL > Examples" + - title: "Schema Design" + link: "server/ongoing-tasks/cdc-sink/document-modeling/schema-design" + source: "docs" + path: "Server > CDC Sink" + - title: "Embedded Tables" + link: "server/ongoing-tasks/cdc-sink/document-modeling/embedded-tables" + source: "docs" + path: "Server > CDC Sink" + - title: "Linked Tables" + link: "server/ongoing-tasks/cdc-sink/document-modeling/linked-tables" + source: "docs" + path: "Server > CDC Sink" + - title: "REPLICA IDENTITY" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import ExampleComplexNestingCsharp from './content/_example-complex-nesting-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/example-denormalization.mdx b/docs/server/ongoing-tasks/cdc-sink/examples/example-denormalization.mdx new file mode 100644 index 0000000000..bdb94e4b63 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/examples/example-denormalization.mdx @@ -0,0 +1,39 @@ +--- +title: "Denormalization with Embedded Tables" +sidebar_label: "Denormalization with Embedded Tables" +description: "How to merge a normalized SQL schema (orders + order_lines) into denormalized RavenDB documents with embedded arrays using CDC Sink." +sidebar_position: 1 +supported_languages: ["csharp"] +see_also: + - title: "Simple Migration" + link: "server/ongoing-tasks/cdc-sink/examples/example-simple-migration" + source: "docs" + path: "Server > CDC Sink > PostgreSQL > Examples" + - title: "Complex Nesting" + link: "server/ongoing-tasks/cdc-sink/examples/example-complex-nesting" + source: "docs" + path: "Server > CDC Sink > PostgreSQL > Examples" + - title: "Embedded Tables" + link: "server/ongoing-tasks/cdc-sink/document-modeling/embedded-tables" + source: "docs" + path: "Server > CDC Sink" + - title: "REPLICA IDENTITY" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Handling Deletes" + link: "server/ongoing-tasks/cdc-sink/handling-deletes" + source: "docs" + path: "Server > CDC Sink" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import ExampleDenormalizationCsharp from './content/_example-denormalization-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/example-running-balance.mdx b/docs/server/ongoing-tasks/cdc-sink/examples/example-running-balance.mdx new file mode 100644 index 0000000000..197ff0126f --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/examples/example-running-balance.mdx @@ -0,0 +1,43 @@ +--- +title: "Maintaining a Running Balance" +sidebar_label: "Running Balance" +description: "How to use CDC Sink patches to incrementally update an account balance as transaction rows arrive from PostgreSQL" +sidebar_position: 2 +supported_languages: ["csharp"] +see_also: + - title: "Simple Migration" + link: "server/ongoing-tasks/cdc-sink/examples/example-simple-migration" + source: "docs" + path: "Server > CDC Sink > PostgreSQL > Examples" + - title: "Denormalization" + link: "server/ongoing-tasks/cdc-sink/examples/example-denormalization" + source: "docs" + path: "Server > CDC Sink > PostgreSQL > Examples" + - title: "Complex Nesting" + link: "server/ongoing-tasks/cdc-sink/examples/example-complex-nesting" + source: "docs" + path: "Server > CDC Sink > PostgreSQL > Examples" + - title: "Patching" + link: "server/ongoing-tasks/cdc-sink/patching" + source: "docs" + path: "Server > CDC Sink" + - title: "Handling Deletes" + link: "server/ongoing-tasks/cdc-sink/handling-deletes" + source: "docs" + path: "Server > CDC Sink" + - title: "Failover and Consistency" + link: "server/ongoing-tasks/cdc-sink/failover-and-consistency" + source: "docs" + path: "Server > CDC Sink" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import ExampleRunningBalanceCsharp from './content/_example-running-balance-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/examples/example-simple-migration.mdx b/docs/server/ongoing-tasks/cdc-sink/examples/example-simple-migration.mdx new file mode 100644 index 0000000000..82ab968a3c --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/examples/example-simple-migration.mdx @@ -0,0 +1,39 @@ +--- +title: "Simple Table Migration" +sidebar_label: "Simple Table Migration" +description: "Minimal setup to replicate a single SQL table into a RavenDB collection using CDC Sink." +sidebar_position: 0 +supported_languages: ["csharp"] +see_also: + - title: "Denormalization" + link: "server/ongoing-tasks/cdc-sink/examples/example-denormalization" + source: "docs" + path: "Server > CDC Sink > PostgreSQL > Examples" + - title: "Event Sourcing" + link: "server/ongoing-tasks/cdc-sink/examples/example-running-balance" + source: "docs" + path: "Server > CDC Sink > PostgreSQL > Examples" + - title: "Complex Nesting" + link: "server/ongoing-tasks/cdc-sink/examples/example-complex-nesting" + source: "docs" + path: "Server > CDC Sink > PostgreSQL > Examples" + - title: "Schema Design" + link: "server/ongoing-tasks/cdc-sink/document-modeling/schema-design" + source: "docs" + path: "Server > CDC Sink" + - title: "Column Mapping" + link: "server/ongoing-tasks/cdc-sink/document-modeling/column-mapping" + source: "docs" + path: "Server > CDC Sink" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import ExampleSimpleMigrationCsharp from './content/_example-simple-migration-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/failover-and-consistency.mdx b/docs/server/ongoing-tasks/cdc-sink/failover-and-consistency.mdx new file mode 100644 index 0000000000..1704946040 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/failover-and-consistency.mdx @@ -0,0 +1,198 @@ +--- +title: "CDC Sink: Failover and Consistency" +sidebar_label: Failover and Consistency +description: "Explains how CDC Sink handles node failover, what consistency guarantees it provides, and why patches must be designed to be idempotent." +sidebar_position: 9 +see_also: + - title: "How It Works" + link: "server/ongoing-tasks/cdc-sink/how-it-works" + source: "docs" + path: "Server > CDC Sink" + - title: "Patching" + link: "server/ongoing-tasks/cdc-sink/patching" + source: "docs" + path: "Server > CDC Sink" + - title: "Monitoring" + link: "server/ongoing-tasks/cdc-sink/monitoring" + source: "docs" + path: "Server > CDC Sink" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* This article explains how CDC Sink handles node failover and consistency. + After failover, the task resumes from the last replicated state, so no changes are lost, + but some source changes may be re-read and applied more than once. + +* Custom patches must be idempotent under this at-least-once delivery model. + +* In this article: + * [State storage](#state-storage) + * [Failover behavior](#failover-behavior) + * [Re-reading changes](#re-reading-changes) + * [Consistency guarantee](#consistency-guarantee) + * [Idempotency requirements](#idempotency-requirements) + + + + + + + +CDC Sink stores processing progress in a RavenDB state document in the `@cdc-states` system collection. +For the full structure, see [The state document](../../../server/ongoing-tasks/cdc-sink/how-it-works.mdx#the-state-document). + +This state document records: + +* The last processed source-database change position (`LastLsn`) +* Per-table initial-load progress, including whether each table's initial load completed + and the last primary-key values read + +Like any RavenDB document, this state document is replicated between cluster nodes. +Different nodes may temporarily have different versions of the document. + + + + + +When the cluster elects a new mentor node for a CDC Sink task, the new mentor reads the state document that was replicated to it. +This state may be older than the previous mentor's actual progress. + +**Example:** + + + +```plain +Before failure: +├─ Node A (Mentor) - Reached position X; latest state update has not replicated +├─ Node B - State document shows position X-100 +└─ Node C - State document shows position X-100 + +Node A crashes. +Node B is elected new mentor. +Node B reads its state document: position X-100. +Node B resumes CDC Sink from position X-100. +``` + + + +Changes between positions X-100 and X may already be in RavenDB, or they may not be if Node A crashed before writing them. +Node B re-reads and applies those changes either way. + + + + + +CDC Sink guarantees that no source changes are lost, +but changes after the last replicated checkpoint may be read again. + +When the new mentor resumes from an older state document, it reprocesses source changes after that saved position: + +* If the previous mentor's RavenDB write already replicated, the same change is applied again. +* If the write did not replicate, the new mentor applies the change for the first time on that node. +* Changes after the previous mentor's progress are processed normally. + +Re-reading is normal and expected. For regular column mapping, re-applying the same +INSERT or UPDATE is safe because mapped properties are overwritten with the same values. + + +Patches that are not idempotent can produce incorrect results when a change is re-read after a failover. +See [Idempotency requirements](#idempotency-requirements) below. + + + + + + +CDC Sink provides **at-least-once delivery** with **eventual consistency**. + +This means: + +* Source changes are not skipped because of failover. +* Changes are processed in source transaction order. +* After failover, the task resumes from the last replicated state document. +* Changes after that saved position may be read and applied more than once. + +CDC Sink does not provide exactly-once delivery. Regular column mapping is safe to re-apply, +but custom patches must be idempotent if they can run more than once for the same source change. + + +This guarantee assumes the source database can still provide changes from the saved position. +For PostgreSQL backup/restore cases where the replication slot no longer retains the saved LSN, +see [Backup/Restore Task](../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/backup-restore-task.mdx#restore-considerations). + + + + + + +A patch is **idempotent** if applying it multiple times produces the same result as applying it once. + +**Column mapping is always idempotent** - overwriting a property with the same value is a no-op. + +**Patches that use absolute values are idempotent:** + + + +```javascript +// Idempotent - always sets to the current SQL value +this.Status = $row.is_active ? 'Active' : 'Inactive'; +this.ViewCount = $row.view_count; +``` + + + +**Patches that accumulate are NOT idempotent:** + +Any patch that reads the document's current value and adds to it will double-count +when a change is re-read after a failover. + + + +```javascript +// NOT idempotent - increments again on re-read +this.ViewCount = (this.ViewCount || 0) + 1; +``` + + + +**Use `$old` when a patch needs to apply a delta safely.** +`$old` contains the previous RavenDB document or embedded item value before the current source row is applied. +When the same source change is re-read after the RavenDB write has already replicated, `$old` reflects the already-applied value, so the computed delta is `0`. + + + +```javascript +// Idempotent - applies only the change between the previous and current values +const oldAmount = $old?.Amount || 0; +const newAmount = $row.amount || 0; +this.RunningTotal = (this.RunningTotal || 0) + (newAmount - oldAmount); +``` + + + +For example, suppose the source row's `amount` is updated from 40 to 55, the document already stores that +previous value as `Amount` (40), and `RunningTotal` currently holds 100. On the first apply, the delta is `55 - 40 = 15`, +so `RunningTotal` moves from 100 to 115 and `Amount` is updated to 55. + +If the same change is re-read after failover and the document write already replicated, `$old.Amount` is already 55, +so the delta is `55 - 55 = 0` and `RunningTotal` remains 115. + +This pattern is idempotent only when the value used for the delta is stored on the +RavenDB document or embedded item and updated on each apply. + +`$old` is available when CDC Sink finds a previous root document or embedded item; it is `null` for new documents or embedded items. +It is also useful for update and delete logic, such as reparenting embedded items. + + +A patch is still not idempotent if it blindly increments, creates a new side-effect document, +or otherwise cannot detect that the same source change was already applied. Prefer absolute +assignments from `$row` when possible. When you need a delta, compute it from `$old` and the current `$row` value. + + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/handling-deletes.mdx b/docs/server/ongoing-tasks/cdc-sink/handling-deletes.mdx new file mode 100644 index 0000000000..28da8683af --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/handling-deletes.mdx @@ -0,0 +1,35 @@ +--- +title: "CDC Sink: Handling Deletes" +sidebar_label: Handling Deletes +description: "Documents configurable behavior for DELETE events in CDC Sink, including archive, audit trail, and silent ignore patterns." +sidebar_position: 7 +supported_languages: ["csharp"] +see_also: + - title: "Patching" + link: "server/ongoing-tasks/cdc-sink/patching" + source: "docs" + path: "Server > CDC Sink" + - title: "Embedded Tables" + link: "server/ongoing-tasks/cdc-sink/document-modeling/embedded-tables" + source: "docs" + path: "Server > CDC Sink" + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" + - title: "REPLICA IDENTITY" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import HandlingDeletesCsharp from './content/_handling-deletes-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/how-it-works.mdx b/docs/server/ongoing-tasks/cdc-sink/how-it-works.mdx new file mode 100644 index 0000000000..78e35e5f83 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/how-it-works.mdx @@ -0,0 +1,345 @@ +--- +title: "CDC Sink: How It Works" +sidebar_label: How It Works +description: "Describes the internal operation of CDC Sink - how it connects to the source database, loads initial data, streams changes, and handles failover." +sidebar_position: 1 +see_also: + - title: "Overview" + link: "server/ongoing-tasks/cdc-sink/overview" + source: "docs" + path: "Server > CDC Sink" + - title: "Schema Design" + link: "server/ongoing-tasks/cdc-sink/document-modeling/schema-design" + source: "docs" + path: "Server > CDC Sink" + - title: "Patching" + link: "server/ongoing-tasks/cdc-sink/patching" + source: "docs" + path: "Server > CDC Sink" + - title: "Property Retention" + link: "server/ongoing-tasks/cdc-sink/property-retention" + source: "docs" + path: "Server > CDC Sink" + - title: "Failover and Consistency" + link: "server/ongoing-tasks/cdc-sink/failover-and-consistency" + source: "docs" + path: "Server > CDC Sink" + - title: "PostgreSQL Prerequisites Checklist" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/prerequisites-checklist" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "PostgreSQL Initial Setup" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; +import ContentFrame from '@site/src/components/ContentFrame'; + + + +* This article describes the internal operation of CDC Sink - how it verifies the source + database, loads initial data, streams changes, preserves transaction order, and handles failover. + +* Understanding these mechanics is important when designing patches, planning for failover, + and updating task configuration. + +* In this article: + * [Startup and verification](#startup-and-verification) + * [Initial load](#initial-load) + * [Streaming changes](#streaming-changes) + * [Transaction ordering](#transaction-ordering) + * [State persistence](#state-persistence) + * [Failover behavior](#failover-behavior) + * [Updating the task configuration](#updating-the-task-configuration) + * [Embedded row before root row](#embedded-row-before-root-row) + + + + + +When a CDC Sink task starts, it first verifies that the source database is ready for change capture. +The specific checks depend on the source database provider. + +If a check fails, CDC Sink reports the problem and the command an administrator can run to fix it. +The task does not continue until all checks pass. + +After verification, CDC Sink creates the required change-tracking infrastructure in the source database, +then begins the initial load. + + + +### PostgreSQL + +CDC Sink checks: + +* WAL level is set to `logical` +* The connecting user has sufficient privileges +* REPLICA IDENTITY is configured correctly for embedded tables that need delete routing + +See the [PostgreSQL prerequisites checklist](../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/prerequisites-checklist.mdx) +for the full list of requirements. + + + + +### SQL Server + +CDC Sink checks that Change Data Capture is enabled on the database and on each captured table, +and that the SQL Server Agent service is running (it drives the CDC capture jobs). + +Learn more in [CDC Sink for SQL Server](../../../server/ongoing-tasks/cdc-sink/source-database-setup/sql-server/overview.mdx). + + + + +### MySQL / MariaDB + +CDC Sink checks that binary logging is enabled with `binlog_format = ROW`, +and that the connecting user has the `REPLICATION SLAVE` and `REPLICATION CLIENT` privileges required to read the binlog. + +Learn more in [CDC Sink for MySQL](../../../server/ongoing-tasks/cdc-sink/source-database-setup/mysql/overview.mdx). + + + + + + + +Before streaming live changes, CDC Sink performs a full scan of every configured **root** and **embedded** table, using keyset pagination ordered by primary key. +**Linked** tables are neither scanned during the initial load nor registered for change capture - +their foreign-key columns, stored on the owning root or embedded row, are resolved into document-ID references in the target collection. + +**Progress tracking:** + +During the initial load, CDC Sink stores each table's progress in the task's **state document**. +If the task is restarted, an interrupted initial load resumes from the last processed primary-key value instead of re-scanning the entire table. +The state document is described in [State persistence](#state-persistence). + +**Batch pipelining:** + +While one batch is being written to RavenDB, the next batch is read from the source database in parallel, +so reads and writes overlap instead of running one after another. + +**Ordering:** + +Tables are scanned in dependency order. +Root tables are loaded first, then embedded tables. +This minimizes the number of stub documents created (see [Child before parent](#child-before-parent) below). + +--- + + + +#### Change tracking during initial load + +Change tracking is set up before the initial load begins, but CDC streaming does not start until after the initial load completes. +The sequence is: + +1. Change-tracking infrastructure is created in the source database + (e.g., a replication slot and publication for PostgreSQL, or CDC capture for SQL Server). +2. The full initial table scan runs - + changes made during this time are captured by the source database but not yet consumed. +3. Once the initial load is complete, CDC streaming starts from the position captured in step 1. + +This guarantees no changes are missed: anything that happened during the initial load is retained by the source database and will be processed immediately after. + +For very large databases, the source database must retain sufficient change history for the duration of the initial load +(e.g., enough WAL disk space on PostgreSQL, or a long enough retention window on SQL Server CDC tables). +Plan accordingly before starting the initial load on large tables. + + + + + + + +After the initial load completes, CDC Sink begins consuming changes from the source database's change log, +starting from the position captured **before** the initial load began. + +CDC Sink processes changes in source transaction order. +Changes from a transaction are applied to RavenDB only after the transaction is committed in the source database, so partial transactions are never written. + +**Document merging:** +When an UPDATE arrives, CDC Sink merges the new column values onto the existing RavenDB document. +Properties that are not part of the column mapping are preserved. +This allows RavenDB-side annotations and computed fields to coexist with CDC-managed properties. +See [Property Retention](../../../server/ongoing-tasks/cdc-sink/property-retention.mdx) for details. + + + + + +CDC Sink preserves the full order of operations within a source database transaction. +If a single transaction performs multiple operations on the same row, all operations are applied in order. + +For example, consider the following source transaction: + + + +```sql +BEGIN; +INSERT INTO items (id, name) VALUES (1, 'Alpha'); +UPDATE items SET name = 'Beta' WHERE id = 1; +DELETE FROM items WHERE id = 1; +INSERT INTO items (id, name) VALUES (1, 'Gamma'); +UPDATE items SET name = 'Delta' WHERE id = 1; +COMMIT; +``` + + + +CDC Sink applies all five operations in order. The final RavenDB document has `name` set to `Delta`. + +Multiple documents modified in the same transaction are also applied atomically within a single RavenDB batch. + + + + + + + +### The state document + +CDC Sink stores processing progress in a single **state document per task**, separate from the task definition: + * The task definition is stored in the database record. + * The processing state is stored as a RavenDB document in the `@cdc-states` system collection. + +![The state document](./assets/the-state-document.png) + +1. **The `@cdc-states` collection** + In the **Documents** view, expand the **@system** group and open the **@cdc-states** collection. + It holds one state document per CDC Sink task. + +2. **The state document ID** + The document is named `@cdc-states/` (here, `@cdc-states/MyCdcSinkTask`). + +3. **The document contents** + The JSON tracks both streaming progress and initial-load progress: + * **`ConfigurationName`** + The name of the CDC Sink task this state belongs to. + * **`LastLsn`** + The last processed source-database change position, used to resume streaming after a restart. + * **`Tables`** + A per-table map, keyed by `schema.tableName`, that stores initial-load progress: + * **`InitialLoadCompleted`** - whether the full scan for that table has finished. + * **`LastKeyValues`** - the last primary-key values read, used to resume an interrupted initial load. + + In this example both tables show `InitialLoadCompleted: true` and an empty `LastKeyValues`, + meaning the initial load has finished and the task is now streaming changes driven by `LastLsn`. + + + + +### Reusing saved state + +* Deleting a CDC Sink task removes the task definition, but leaves the matching state document. + If you create a new task with the same name, CDC Sink can reuse that state document and resume from the saved position. + +* To start from scratch, create the replacement task with a different name. + Since no `@cdc-states/` document exists yet, CDC Sink has no saved position to resume from. + It performs a fresh initial load unless the task is configured with `SkipInitialLoad = true`. + +* See [Troubleshooting](../../../server/ongoing-tasks/cdc-sink/troubleshooting.mdx) for details on state loss and recovery options. + + + + +### Cluster replication + +Like any RavenDB document, this state document is subject to [internal replication](../../../server/clustering/replication/replication-overview.mdx#replication-types) behavior. +Different nodes in a cluster may temporarily have different versions of this document. + + + + + + + +When the cluster elects a new mentor node for the CDC Sink task, the new node reads the state document that was replicated to it. +If the previous mentor processed changes but its latest state update had not yet replicated, the new mentor may resume from an earlier position. + +This means: + +* **No data is lost**: + CDC Sink resumes from a known position and the source database retains all changes from that position onward. +* **Some changes may be re-read**: + Changes between the replicated state and the previous mentor's actual progress will be processed again. + +Re-reading is normal and expected. The document merge strategy means that re-applying the same INSERT or UPDATE is safe - +column values are simply overwritten with the same values. + + + +Patches that are not idempotent can produce incorrect results if the same change is re-read after a failover. +Design patches to handle re-processing safely. +See [Patching](../../../server/ongoing-tasks/cdc-sink/patching.mdx) for guidance. + + + + + + + +When you update a CDC Sink task configuration - +for example, by adding or removing columns from the `Columns` list, changing a `Patch` script, or modifying embedded table configuration - +the changes take effect **only for new CDC events going forward**. + +CDC Sink does not retroactively reprocess existing documents. +Documents that were already created retain their current structure. + +Common configuration changes behave as follows: + +* **Adding a new column mapping**: + Existing documents will not have the new property. + Only documents created or updated by a subsequent CDC event will include it. +* **Removing a column mapping**: + Existing documents retain the property. + It will no longer be updated by CDC events, but it is not removed from existing documents. +* **Changing a patch script**: + The new patch runs on future events only. + Existing documents reflect the results of the previous patch. +* **Adding an embedded or linked table**: + Existing parent documents will not reflect the new embedded data or linked references + until a CDC event arrives for the relevant source row. + +--- + + + +#### Applying changes to existing documents + +* Simply restarting the task does not reprocess existing documents. + A restart resumes from the task's saved position (see [State persistence](#state-persistence)), + so it only continues streaming new changes. + +* To apply configuration changes to **all** documents (not just new events), delete the existing CDC Sink task + and create a new task with a new name and `SkipInitialLoad` disabled. The new task has no matching state document, + so it performs a fresh initial load, reprocessing all rows with the updated configuration. + +* If you only need to backfill a specific property on existing documents, + you can also use a RavenDB [patch by query](../../../client-api/operations/patching/set-based.mdx) to update existing documents independently of the CDC Sink task. + + + + + + + +Tables are scanned one at a time during the initial load. +Root tables are scanned before their embedded tables, but source data can still change while the initial load is running. + +For example, _Orders_ are scanned first. Then, while _OrderLines_ are being scanned, +CDC Sink encounters an _OrderLine_ that belongs to an _Order_ inserted after the _Orders_ scan had already finished. + +In this case, CDC Sink creates a **stub document** for the root document containing only the embedded child data. +When CDC streaming begins after the initial load, it picks up the missing root row and merges its columns onto the stub document. + +The final document contains both the root fields and the embedded items. +No data is lost - the brief intermediate state is resolved automatically once streaming starts. + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/_category_.json b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/_category_.json new file mode 100644 index 0000000000..f24e44ba19 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/_category_.json @@ -0,0 +1,4 @@ +{ + "position": 3, + "label": "Manage CDC Sink Tasks" +} diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-1.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-1.png new file mode 100644 index 0000000000..9c9c7d441a Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-1.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-10.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-10.png new file mode 100644 index 0000000000..cb89b14f71 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-10.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-11.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-11.png new file mode 100644 index 0000000000..0074d11d20 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-11.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-12.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-12.png new file mode 100644 index 0000000000..358f5723ca Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-12.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-2.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-2.png new file mode 100644 index 0000000000..59c70626e0 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-2.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-3.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-3.png new file mode 100644 index 0000000000..80edcaba1d Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-3.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-4.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-4.png new file mode 100644 index 0000000000..6b59da9595 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-4.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-5.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-5.png new file mode 100644 index 0000000000..d23af318bc Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-5.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-6.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-6.png new file mode 100644 index 0000000000..925ef25b66 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-6.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-7.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-7.png new file mode 100644 index 0000000000..2d8fc8a850 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-7.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-8.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-8.png new file mode 100644 index 0000000000..3ca81b6514 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-8.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-9.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-9.png new file mode 100644 index 0000000000..2e8654cffb Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/create-task-9.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/delete-task.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/delete-task.png new file mode 100644 index 0000000000..cd050c9944 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/delete-task.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/enable-disable-1.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/enable-disable-1.png new file mode 100644 index 0000000000..72a7c6d968 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/enable-disable-1.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/enable-disable-2.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/enable-disable-2.png new file mode 100644 index 0000000000..2fb3a11e52 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/enable-disable-2.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/get-task-info.png b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/get-task-info.png new file mode 100644 index 0000000000..f887eb99cc Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/get-task-info.png differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-1.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-1.snagx new file mode 100644 index 0000000000..0866e57c8d Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-1.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-10.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-10.snagx new file mode 100644 index 0000000000..15eb7b2963 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-10.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-11.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-11.snagx new file mode 100644 index 0000000000..4079dddf77 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-11.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-12.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-12.snagx new file mode 100644 index 0000000000..594e0dea78 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-12.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-2.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-2.snagx new file mode 100644 index 0000000000..7f900018d4 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-2.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-3.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-3.snagx new file mode 100644 index 0000000000..0060289b58 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-3.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-4.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-4.snagx new file mode 100644 index 0000000000..02331ff305 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-4.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-5.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-5.snagx new file mode 100644 index 0000000000..1f5c86a080 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-5.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-6.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-6.snagx new file mode 100644 index 0000000000..310061c6df Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-6.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-7.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-7.snagx new file mode 100644 index 0000000000..f478319a04 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-7.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-8.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-8.snagx new file mode 100644 index 0000000000..77e728c9d3 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-8.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-9.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-9.snagx new file mode 100644 index 0000000000..86e9726e0d Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/create-task-9.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/delete-task.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/delete-task.snagx new file mode 100644 index 0000000000..7a6929a7ad Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/delete-task.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/enable-disable-1.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/enable-disable-1.snagx new file mode 100644 index 0000000000..d0964a21c0 Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/enable-disable-1.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/enable-disable-2.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/enable-disable-2.snagx new file mode 100644 index 0000000000..03b253956a Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/enable-disable-2.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/get-task-info.snagx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/get-task-info.snagx new file mode 100644 index 0000000000..e5b3bc480c Binary files /dev/null and b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/assets/snagit/get-task-info.snagx differ diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/backup-restore-task.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/backup-restore-task.mdx new file mode 100644 index 0000000000..41618e5058 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/backup-restore-task.mdx @@ -0,0 +1,31 @@ +--- +title: "CDC Sink: Backup/Restore Task" +sidebar_label: Backup/Restore Task +description: "How CDC Sink tasks are included in database backups and how they behave when a database is restored, via the RavenDB Client API and the Management Studio." +sidebar_position: 7 +supported_languages: ["csharp"] +see_also: + - title: "Export / Import a Task" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/export-import-task" + source: "docs" + path: "Server > CDC Sink" + - title: "Enable / Disable a Task" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/enable-disable-task" + source: "docs" + path: "Server > CDC Sink" + - title: "Failover and Consistency" + link: "server/ongoing-tasks/cdc-sink/failover-and-consistency" + source: "docs" + path: "Server > CDC Sink" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import BackupRestoreTaskCsharp from './content/_backup-restore-task-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference.mdx new file mode 100644 index 0000000000..2a741f89c9 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference.mdx @@ -0,0 +1,27 @@ +--- +title: "CDC Sink: Configuration Reference (Client API)" +sidebar_label: Configuration Reference +description: "The Client API configuration classes used to define a CDC Sink task in code: CdcSinkConfiguration, table configs, CdcColumnMapping, and related types." +sidebar_position: 8 +supported_languages: ["csharp"] +see_also: + - title: "Create a Task" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task" + source: "docs" + path: "Server > CDC Sink" + - title: "Schema Design" + link: "server/ongoing-tasks/cdc-sink/document-modeling/schema-design" + source: "docs" + path: "Server > CDC Sink" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import ConfigurationReferenceCsharp from './content/_configuration-reference-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_backup-restore-task-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_backup-restore-task-csharp.mdx new file mode 100644 index 0000000000..0f1bfe59d7 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_backup-restore-task-csharp.mdx @@ -0,0 +1,178 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* CDC Sink task configuration is stored in the database record, + so it is included in database **backups** (logical and snapshot) and recreated when the database is **restored**. + +* Unlike an [imported CDC Sink task](../../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/export-import-task.mdx#import-cdc-sink-task), + which is created disabled, a restored CDC Sink task keeps the enabled/disabled state it had in the backup, + unless you set `DisableOngoingTasks = true`, which disables all ongoing tasks on the restored database. + +* In this article: + * [Back up CDC Sink task](#back-up-cdc-sink-task) + * [Backup via Client API](#backup-via-client-api) + * [Backup via Studio](#backup-via-studio) + * [Restore CDC Sink task](#restore-cdc-sink-task) + * [Restore via Client API](#restore-via-client-api) + * [Restore via Studio](#restore-via-studio) + * [Restore considerations](#restore-considerations) + * [Behavior summary](#behavior-summary) + + + + + +CDC Sink tasks are backed up automatically as part of the database record. +No CDC Sink-specific backup setting is required; the task configuration is included in both +[logical and snapshot backups](../../../../../backup/overview.mdx#backup-type). + +This applies whether you use a per-database backup task or a [server-wide backup task](../../../../../backup/create/periodic-tasks/server-wide-backup.mdx), +which applies the backup configuration to each included database. + +--- + +### Backup via Client API + +Use `UpdatePeriodicBackupOperation` to create or update a periodic backup task. +Learn more in [Creating and managing periodic backups using the client API](../../../../../backup/create/periodic-tasks/database-backup.mdx#creating-and-managing-periodic-backups-using-the-client-api). + +```csharp +var backupConfig = new PeriodicBackupConfiguration +{ + Name = "DailyBackup", + BackupType = BackupType.Backup, // or BackupType.Snapshot + FullBackupFrequency = "0 2 * * *", // every day at 02:00 + LocalSettings = new LocalSettings { FolderPath = @"/backups/db" } +}; + +await store.Maintenance.SendAsync(new UpdatePeriodicBackupOperation(backupConfig)); +``` + +--- + +### Backup via Studio + +* Create or manage a periodic backup task in Studio - see + [Creating and managing periodic backups via Studio](../../../../../backup/create/periodic-tasks/database-backup.mdx#creating-and-managing-periodic-backups-via-studio). + +* CDC Sink tasks are included automatically as part of the database record. + + + + + +Restoring the database recreates the CDC Sink task from the database record in the backup. +The restored task keeps the enabled/disabled state it had in the backup. + +Set `DisableOngoingTasks = true` if you do **not** want the CDC Sink task (or any other ongoing task) +to start after the database is restored. + +--- + +### Restore via Client API + +Restore the database with `RestoreBackupOperation`. +Learn more in [Restoring from backup using the client API](../../../../../backup/restore.mdx#restoring-from-backup-using-the-client-api): + +```csharp +var restoreConfig = new RestoreBackupConfiguration +{ + DatabaseName = "RestoredDb", + BackupLocation = @"/backups/db/2026-07-07-02-00", + + // Recommended when restoring to a new environment so ongoing tasks + // stay disabled until you've reviewed their settings: + DisableOngoingTasks = true +}; + +var restoreOp = await store.Maintenance.Server.SendAsync( + new RestoreBackupOperation(restoreConfig)); +await restoreOp.WaitForCompletionAsync(); +``` + +--- + +### Restore via Studio + +* From **Databases** → **New database** → **from Backup**, choose the backup location. + Learn more in [Restoring from backup using Studio](../../../../../backup/restore.mdx#restoring-from-backup-using-studio). + +* Enable **Disable ongoing tasks after restore** if you do **not** want the CDC Sink task + to begin consuming changes until you are ready. + +--- + +### Restore considerations + + + +#### Restored enabled tasks can start immediately + +* If `DisableOngoingTasks` is left as `false`, + a CDC Sink task that was enabled when the backup was created becomes **active immediately** + and can connect to the source database. + +* When restoring to a new environment, verify the connection string and source availability first. + For **PostgreSQL**, also confirm that the referenced publication exists and that the replication slot + can be consumed, or configure dedicated names before enabling the restored task. + +* The task's saved CDC progress is restored from the [`@cdc-states/` document](../../../../../server/ongoing-tasks/cdc-sink/how-it-works.mdx#the-state-document) + as it existed in the backup. If no matching state document exists, the task starts fresh + and performs an initial load unless `SkipInitialLoad` is set. + + + + + +#### PostgreSQL: shared replication slot + +* Restore copies the task's `SlotName` and `PublicationName` unchanged. + +* A PostgreSQL replication slot can be consumed by **only one connection at a time**. + If the origin database's task is still enabled, the restored task cannot attach to the same slot + and enters fallback mode, reporting an error such as `Process is in error recovery (fallback mode)...`. + Learn more in [One Active Consumer per Slot](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup.mdx#one-active-consumer-per-slot). + +* Disable the task on one of the two databases (or restore with `DisableOngoingTasks = true`), + or give the restored task its own slot and publication before enabling it. + + + + + +#### PostgreSQL: saved LSN no longer retained by the slot + +* The restored task uses the `LastLsn` saved in its `@cdc-states/` document. + This works only if the PostgreSQL replication slot can still serve WAL from that position. + +* If the slot was dropped and recreated after the backup, + or if PostgreSQL no longer retains the WAL needed for the saved `LastLsn`, + the slot's `restart_lsn` can be ahead of the restored `LastLsn`. + Changes in that gap cannot be recovered from the slot. + +* When this happens, CDC Sink raises a warning notification stating that the slot's restart LSN is ahead of the saved position. + To start over from the current source data, remove the saved CDC Sink state (`@cdc-states/`) + or create the task with a new name, then let the initial load run again. + If the gap might include deletes, verify or rebuild the affected target documents as needed. + +* To avoid this gap, restore the source database and the RavenDB database to consistent points in time, + and do not drop the slot while a backup that references it may still be restored. + + + + + + + +| Aspect | Backup / Restore behavior | +|----------------------------------------------------|---------------------------| +| Task configuration | Included in the backup; recreated on restore | +| Enabled/disabled state | Preserved from the backup, **unless** `DisableOngoingTasks = true` | +| Saved CDC progress
(`@cdc-states/`) | Restored as it existed in the backup; if no matching state document exists,
the task starts fresh unless `SkipInitialLoad` is set | +| PostgreSQL slot / publication | Not backed up; the restored task references the same names,
so the publication must exist and the slot must be consumable | + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_configuration-reference-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_configuration-reference-csharp.mdx new file mode 100644 index 0000000000..2769dec393 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_configuration-reference-csharp.mdx @@ -0,0 +1,179 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* This article documents the Client API configuration classes used to define a CDC Sink task in code. + Pass a `CdcSinkConfiguration` to `AddCdcSinkOperation` (create) or `UpdateCdcSinkOperation` (update). + +* In this article: + * [CdcSinkConfiguration](#cdcsinkconfiguration) + * [CdcSinkPostgresSettings](#cdcsinkpostgressettings) + * [CdcSinkTableConfig](#cdcsinktableconfig) + * [CdcSinkEmbeddedTableConfig](#cdcsinkembeddedtableconfig) + * [CdcSinkLinkedTableConfig](#cdcsinklinkedtableconfig) + * [CdcColumnMapping](#cdccolumnmapping) + * [CdcColumnType](#cdccolumntype) + * [CdcSinkOnDeleteConfig](#cdcsinkondeleteconfig) + * [CdcSinkRelationType](#cdcsinkrelationtype) + + + + + +The top-level configuration object for a CDC Sink task. + +| Property | Type | Required | Description | +| ---------------------- | -------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Name` | `string` | ✓ | Unique task name | +| `ConnectionStringName` | `string` | ✓ | Name of the SQL connection string (uses the existing `SqlConnectionString` type - the same as SQL ETL). The connection string's `FactoryName` determines the source database engine. | +| `Tables` | `List` | ✓ | Root table configurations (at least one required) | +| `Postgres` | `CdcSinkPostgresSettings` | | PostgreSQL-specific settings (slot and publication names) | +| `SkipInitialLoad` | `bool` | | When `true`, skip the initial full-table scan and start streaming CDC changes immediately. Only applies on first startup - has no effect once the initial load has completed. Default: `false` | +| `Disabled` | `bool` | | Create the task paused (without deleting it). Default: `false` | +| `MentorNode` | `string` | | Preferred cluster node for execution | +| `PinToMentorNode` | `bool` | | Pin the task to the mentor node. Default: `false` | +| `TaskId` | `long` | | Server-assigned task ID. To update a task, pass the existing task ID to `UpdateCdcSinkOperation(taskId, configuration)`; do not use this property as the update selector. | + +**Supported `FactoryName` values (set on the connection string):** + +| FactoryName | Database Engine | +| ------------------------------------------------------------------ | --------------- | +| `Npgsql` | PostgreSQL | +| `Microsoft.Data.SqlClient` | SQL Server | +| `MySql.Data.MySqlClient` or `MySqlConnector.MySqlConnectorFactory` | MySQL / MariaDB | + + + + + +PostgreSQL-specific settings. Assigned to `CdcSinkConfiguration.Postgres`. +Leave `null` for non-PostgreSQL connections. + +| Property | Type | Description | +| ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `SlotName` | `string` | Name of the PostgreSQL logical replication slot. If omitted on creation, auto-generated as `rvn_cdc_s_{taskId}`. Immutable once set. Max 63 characters, alphanumeric and underscores only. | +| `PublicationName` | `string` | Name of the PostgreSQL publication. If omitted on creation, auto-generated as `rvn_cdc_p_{taskId}` (note the `_p_` prefix vs. the slot's `_s_`). Same immutability rules as `SlotName`. | + +Setting these explicitly is useful when a DBA pre-creates the slot and publication with human-readable names, +when migrating from a previous task and reusing an existing slot, or when running multiple environments with predictable names. + + + + + +Configures a root table - one SQL table mapped to one RavenDB collection. + +| Property | Type | Required | Description | +| ------------------- | ---------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| `CollectionName` | `string` | ✓ | RavenDB collection name (e.g., `"Orders"`) | +| `SourceTableName` | `string` | ✓ | SQL table name (e.g., `"orders"`) | +| `SourceTableSchema` | `string` | | SQL schema name. Default is provider-specific: PostgreSQL `"public"`, SQL Server `"dbo"`, MySQL/MariaDB the database name from the connection string. | +| `PrimaryKeyColumns` | `List` | ✓ | SQL columns used for document ID generation | +| `Columns` | `List` | ✓ | Column mappings - each entry defines a SQL column, its document property name, and how to store it | +| `Patch` | `string` | | JavaScript patch for INSERT and UPDATE | +| `OnDelete` | `CdcSinkOnDeleteConfig` | | Delete behavior. Default: delete document | +| `EmbeddedTables` | `List` | | Nested table configurations | +| `LinkedTables` | `List` | | Foreign key reference configurations | +| `Disabled` | `bool` | | Skip this table. Default: `false` | + + + + + +Configures a table whose rows are embedded as nested objects within a parent document. + +| Property | Type | Required | Description | +| ------------------- | ---------------------------------- | -------- | --------------------------------------------------------- | +| `SourceTableName` | `string` | ✓ | SQL table name | +| `SourceTableSchema` | `string` | | SQL schema name (provider-specific default) | +| `PropertyName` | `string` | ✓ | Property name in the parent document (e.g., `"Lines"`) | +| `Type` | `CdcSinkRelationType` | ✓ | `Array`, `Map`, or `Value` | +| `JoinColumns` | `List` | ✓ | FK columns referencing parent's `PrimaryKeyColumns` | +| `PrimaryKeyColumns` | `List` | ✓ | PK columns for matching items on UPDATE/DELETE | +| `Columns` | `List` | ✓ | Column mappings | +| `Patch` | `string` | | JavaScript patch on **parent** document for INSERT/UPDATE | +| `OnDelete` | `CdcSinkOnDeleteConfig` | | Delete behavior for embedded items | +| `CaseSensitiveKeys` | `bool` | | Case-sensitive PK matching. Default: `false` | +| `EmbeddedTables` | `List` | | Nested embedded tables | +| `LinkedTables` | `List` | | Linked tables within embedded items | + + + + + +Configures a foreign key reference that becomes a document ID in the parent document. + +| Property | Type | Required | Description | +| ---------------------- | -------------- | -------- | ----------------------------------------------------------------- | +| `SourceTableName` | `string` | ✓ | SQL table name of the referenced table | +| `SourceTableSchema` | `string` | | SQL schema name (provider-specific default) | +| `PropertyName` | `string` | ✓ | Property name in the parent document (e.g., `"Customer"`) | +| `LinkedCollectionName` | `string` | ✓ | Target RavenDB collection for ID generation (e.g., `"Customers"`) | +| `JoinColumns` | `List` | ✓ | FK columns used to build the referenced document ID | + + + + + +A single column mapping entry within a `Columns` list. + +| Property | Type | Required | Description | +| -------- | --------------- | -------- | -------------------------------------------------------------------- | +| `Column` | `string` | ✓ | SQL column name (case-insensitive) | +| `Name` | `string` | ✓ | Document property name (or attachment name when `Type = Attachment`) | +| `Type` | `CdcColumnType` | | How to store the value. Default: `Default` | + + + + + +Controls how a SQL column value is stored in RavenDB. + +| Value | Behavior | +| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Default` | Standard type conversion: int→long, numeric/decimal→decimal, real→float, double precision→double, date→DateOnly, timestamp→DateTime, uuid→string, varchar/text→string. SQL arrays→JSON arrays. JSON/JSONB stored as plain string. | +| `Json` | Parses the string value as a native JSON value (object, array, string, number, boolean, etc.) in the document. Use for PostgreSQL `json`/`jsonb` or SQL Server `nvarchar(max)` with JSON content. | +| `Attachment` | Stores the raw column value as a RavenDB attachment. `byte[]`→binary (`application/octet-stream`), `string`→UTF-8 text (`text/plain; charset=utf-8`), `float[]`/`double[]`→raw binary vector data (`application/octet-stream`), not JSON arrays. The `Name` field becomes the attachment name. | + + + + + +Controls how DELETE events are handled for a table or embedded table. + +| Property | Type | Default | Description | +| --------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------- | +| `Patch` | `string` | null | JavaScript patch that runs when a DELETE event arrives | +| `IgnoreDeletes` | `bool` | `false` | When `true`, CDC Sink skips the automatic document delete or embedded-item removal. If `Patch` is set, the patch still runs. | + +**Available patch variables for OnDelete:** + +* `this` - the document (root or parent for embedded) +* `$row` - all SQL columns from the DELETE event +* `$old` - the previous stored document or embedded item before the DELETE event + +**Behavior matrix:** + +| IgnoreDeletes | Patch | Result | +| ------------- | ----- | -------------------------------- | +| `false` | null | Normal delete (default) | +| `false` | set | Patch runs, then delete proceeds | +| `true` | null | DELETE discarded silently | +| `true` | set | Patch runs, delete skipped | + + + + + +Specifies the structure of embedded data in the document. + +| Value | Description | +| ------- | -------------------------------------------------------------------------- | +| `Array` | One-to-many: stored as a JSON array. Items matched by PK for UPDATE/DELETE | +| `Map` | One-to-many: stored as a JSON object keyed by PK value(s) | +| `Value` | Many-to-one: stored as a single embedded object | + + diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_create-task-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_create-task-csharp.mdx new file mode 100644 index 0000000000..3473269612 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_create-task-csharp.mdx @@ -0,0 +1,385 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; +import ContentFrame from "@site/src/components/ContentFrame"; + + + +* A CDC Sink task consumes Change Data Capture streams from a relational source database + (PostgreSQL, SQL Server, or MySQL/MariaDB) and applies inserts, updates, and deletes to documents in RavenDB. + +* This article shows how to **create** a CDC Sink task using the Client API, Studio, or REST API. + +* In this article: + * [Create a CDC Sink task via Client API](#create-a-cdc-sink-task-via-client-api) + * [Create a CDC Sink task via Studio](#create-a-cdc-sink-task-via-studio) + * [Create task](#create-task) + * [Configure basic settings](#configure-basic-settings) + * [Schema explorer](#schema-explorer) + * [Configured tables](#configured-tables) + * [Configure linked tables](#configure-linked-tables) + * [Configure embedded tables](#configure-embedded-tables) + * [Test mapping](#test-mapping) + * [Advanced settings](#advanced-settings) + * [The created task](#the-created-task) + * [Create a CDC Sink task via REST API](#create-a-cdc-sink-task-via-rest-api) + + + + + +Use `AddCdcSinkOperation` to create a new CDC Sink task from a `CdcSinkConfiguration`. + +The configuration references a registered SQL connection string by name. +CDC Sink uses the same `SqlConnectionString` type as [SQL ETL](../../../../../server/ongoing-tasks/etl/sql.mdx); +the connection string's `FactoryName` determines the source engine. + +Optionally set `SkipInitialLoad = true` only when the target RavenDB database already contains the initial data and the task should start from new CDC changes. + +See [Configuration Reference](../../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference.mdx) +for the full list of configuration classes and properties. + + + +```csharp +var config = new CdcSinkConfiguration +{ + // Unique name for the CDC Sink task + Name = "OrdersSync", + + // Name of an existing SqlConnectionString, + // its FactoryName selects the source database engine + ConnectionStringName = "MyPostgresConnection", + + // Optional: set to true only if RavenDB already contains the initial data. + // SkipInitialLoad = true, + + Tables = new List + { + new CdcSinkTableConfig + { + // Map the source table to the Orders collection in RavenDB + CollectionName = "Orders", + SourceTableName = "orders", + + // Primary key columns are used to generate document IDs + // and must be included in Columns + PrimaryKeyColumns = new List { "id" }, + + // Map source columns to RavenDB document properties. + Columns = + [ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "customer_name", Name = "CustomerName" }, + new CdcColumnMapping() { Column = "total", Name = "Total" }, + ] + } + } +}; + +// Execute the operation by passing it to Maintenance.Send +var result = store.Maintenance.Send(new AddCdcSinkOperation(config)); + +long taskId = result.TaskId; +``` + + +```csharp +var config = new CdcSinkConfiguration +{ + // Unique name for the CDC Sink task + Name = "OrdersSync", + + // Name of an existing SqlConnectionString, + // its FactoryName selects the source database engine + ConnectionStringName = "MyPostgresConnection", + + // Optional: set to true only if RavenDB already contains the initial data. + // SkipInitialLoad = true, + + Tables = new List + { + new CdcSinkTableConfig + { + // Map the source table to the Orders collection in RavenDB + CollectionName = "Orders", + SourceTableName = "orders", + + // Primary key columns are used to generate document IDs + // and must be included in Columns + PrimaryKeyColumns = new List { "id" }, + + // Map source columns to RavenDB document properties. + Columns = + [ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "customer_name", Name = "CustomerName" }, + new CdcColumnMapping() { Column = "total", Name = "Total" }, + ] + } + } +}; + +// Execute the operation by passing it to Maintenance.SendAsync +var result = await store.Maintenance.SendAsync(new AddCdcSinkOperation(config)); + +long taskId = result.TaskId; +``` + + + +`AddCdcSinkOperationResult`: + +| Property | Type | Description | +|--------------------|--------|-------------| +| `TaskId` | `long` | Assigned task ID | +| `RaftCommandIndex` | `long` | Raft index of the command | + + + + + +### Create task + +![Create task 1](../assets/create-task-1.png) + +1. Navigate to **Databases** → your database → **Tasks** → **Ongoing Tasks**. +2. Click **Add a Database Task** + +--- + +![Create task 2](../assets/create-task-2.png) + +1. The **Add a database task** view opens. +2. Filter by the **Sink** category to show only source-to-RavenDB sink tasks. +3. Select the **CDC Sink** task type. + +--- + +### Configure basic settings + +![Create task 3](../assets/create-task-3.png) + +1. **Configure basic settings** + In this section you specify the task name, select a connection string, and configure task options. +2. **Task Name** + Enter a unique name for the CDC Sink task. +3. **Task State** + Set the initial state, **Enabled** or **Disabled**. +4. **Connection String** + Select an existing SQL connection string to the source database. +5. Alternatively, click **Create a new SQL connection string** to define a new one. +6. For PostgreSQL connections, you can optionally set a custom **Publication Name** and **Slot Name**. + Leave these fields empty to let the server auto-generate both values. +7. **Set Responsible Node** + Optionally pin the task to a specific node in the database group. +8. **Skip initial load** + Enable only if RavenDB already contains the initial data and the task should start from new CDC changes. +9. **Raw config** + Toggle to view or paste the task's JSON configuration directly. + Useful for copying a configuration between environments. + +--- + +### Schema explorer + +![Create task 4](../assets/create-task-4.png) + +1. **Schema Explorer** + In this section you can discover existing tables from the configured connection. +2. **Discover tables** + Click to query the source database for tables. +3. **Available tables** + Discovered tables appear here showing each table's name, primary keys, column count, and any warnings. + +![Create task 5](../assets/create-task-5.png) + +1. **Select tables** + Check the box next to each source table you want to sink into RavenDB as documents. +2. **Configure selected tables** + Click _Configure selected tables_ to add the selected tables to the configuration panel below, + where you map their columns to RavenDB collections and document properties. + +--- + +### Configured tables + +![Create task 6](../assets/create-task-6.png) + +1. **Configured Tables** - In this section, configure how source tables are mapped to target RavenDB collections. +2. **Tables** - the list of configured tables. Select a table to edit its mapping. + * A table can be a **root** table (mapped directly to its own collection), + or a table nested under a root table - either **linked** or **embedded** (see the sections below). + * In this example, `orders` is a root table with `customers` nested under it as a linked table (marked with 🔗); + `customers` also appears on its own as a separate root table. +3. **Source schema** - the schema the source table belongs to (e.g. `public`). +4. **Source table** - the name of the source table. +5. **Target collection** - the RavenDB collection that documents are written to. +6. **Primary key columns** - Columns that uniquely identify each source row. + Their values are used to derive the RavenDB document ID. + Click **Add primary key column** to add more. +7. **Source column** - the column name in the source table. Click **Add field mapping** to add more. +8. **Target property** - the RavenDB document field where the source column value will be stored. +9. **Type** - select how the source column value is stored in the document: + * `Default` - stored as a regular property + * `JSON` - parsed as JSON before storing + * `Attachment` - stored as a RavenDB attachment +10. **Advanced settings** - expand to configure patch scripts and delete handling. +11. Once you've finished configuring all tables and their advanced settings (see below), + click **Save task configuration** to save the task. +12. Click **Test** to preview the mapping output before saving. + +--- + +### Configure linked tables + +A **linked table** turns a foreign-key relationship into a document-ID reference: +instead of copying the related row's data into the document, +the parent document stores a reference (a document ID) to a document in another collection. + +In this example, the `customers` table is linked under the `orders` root table. +The resulting Order document gets a `Customer` property holding a document-ID reference into the `Customers` collection. + +![Create task 7](../assets/create-task-7.png) + +1. **The source table that is linked** +2. **Source schema** + The related source schema. +3. **Source table** + The related source table. +4. **Target property** + The RavenDB document field that will hold the reference to the linked (related) document. +5. **Linked collection** + The RavenDB collection the reference points to. + Its name is combined with the join column value(s) to form the referenced document ID + (e.g. `Customers` + `customer_id` → `Customers/1`). +6. **Join columns** + The foreign key column(s) used to join this linked table to the parent table. + Their values, combined with the linked collection name, form the related document ID. + Click **Add join column** for a composite key. + +--- + +### Configure embedded tables + +An **embedded table** nests the related rows _inside_ the parent document instead of referencing them. + +![Create task 8](../assets/create-task-8.png) + +1. On a root (or embedded) table in the **Tables** list, click the three-dots **Table actions** icon. +2. Select **Add new embedded table**. + +--- + +![Create task 9](../assets/create-task-9.png) + +1. The embedded table appears nested under its parent in the **Tables** list (marked with `{+}`). +2. **Source schema** + The schema of the related source table (e.g. `public`). +3. **Source table** + The related source table (e.g. `customers`). +4. **Target property** + The RavenDB document field where the embedded related data will be stored. +5. **Relation type** + How related rows are embedded in the parent document: + * `Array` - multiple rows stored as an array of objects + * `Map` - multiple rows stored as a keyed object (dictionary) + * `Value` - a single row stored directly as an object +6. **Primary key columns** + Columns that uniquely identify rows in this related table. +7. **Join columns** + Columns used to match rows in this related table with rows from the root table. +8. **Field mapping** + Map the related table's columns to document properties, + each with its own **Type** (Default / JSON / Attachment). + +--- + +### Test mapping + +![Create task 10](../assets/create-task-10.png) + +1. Select the table whose mapping you want to test. +2. Click **Test** to open the test panel. +3. The **Test mapping** panel shows the selected table-to-collection mapping (e.g. `customers` → `Customers`). +4. **Settings** + * **Row selector** + Choose which source rows to pull for the test: + * `First row` + Fetch rows from the top of the table. + Set **Max rows** to control how many rows to fetch and preview. + Each fetched row is turned into a RavenDB document, so you can preview **multiple** documents at once. + * `Primary key` + Fetch a **single** specific row by entering its primary key value(s). + For a composite primary key, enter one value per key column - together they identify one row. + * **Simulate on delete** + Run the test as if the rows were DELETE events instead of inserts/updates, + to preview your delete handling (Skip deletion / Delete patch). +5. Click **Run test** to execute the test against the source data. +6. The resulting RavenDB document(s) are shown as **JSON**. + +--- + +### Advanced settings + +![Create task 11](../assets/create-task-11.png) + +1. Expand **Advanced settings** for the selected table. +2. **Patch** + An optional script that transforms `this` (the document) AFTER column mapping, on INSERT and UPDATE events. +3. **Patch syntax help** + Click the **(?)** icon for patch syntax help and examples. +4. **Skip deletion** + Enable to ignore source DELETE events (useful for soft-delete scenarios). +5. **Delete patch** + An optional script that runs on DELETE events BEFORE the delete is applied or ignored. +6. **Delete-patch syntax help** + Click the **(?)** icon for delete-patch syntax help and examples. + +--- + +### The created task + +![Create task 12](../assets/create-task-12.png) + +1. The new task appears under **Ongoing Tasks**. +2. It is listed under the **SINK (SOURCE ⇒ RavenDB)** group, showing the task name, its responsible node, and its state. + Use the controls on the right to enable/disable, edit, or delete the task. + + + + + + + +### Discover the source database schema + +Use this endpoint to list tables, columns, primary keys, and foreign keys from the source database. +Use the returned schema metadata to build the `Tables` section of the `CdcSinkConfiguration` before sending the create-task request. + +The request body should include either: +* `ConnectionStringName`, to use a saved SQL connection string, or +* `Connection`, to pass an inline `SqlConnectionString`. + +For PostgreSQL, you can include `Schemas` to limit schema discovery (defaults to `["public"]`). + +| Method | Endpoint | Auth | +|--------|----------|------| +| `POST` | `/databases/{databaseName}/admin/cdc-sink/schema` | `DatabaseAdmin` | + + + + +### Create the task + +Send a `PUT` request whose body is the JSON `CdcSinkConfiguration`. +Omit the `id` query-string parameter to create a new task; the response returns the assigned `TaskId`. + +| Method | Endpoint | Auth | +|--------|----------|------| +| `PUT` | `/databases/{databaseName}/admin/cdc-sink` | `DatabaseAdmin` | + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_delete-task-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_delete-task-csharp.mdx new file mode 100644 index 0000000000..54c308c00a --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_delete-task-csharp.mdx @@ -0,0 +1,92 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* Deleting a CDC Sink task: + * removes the task definition from RavenDB + * stops RavenDB from consuming new changes from the source database + +* Deleting a CDC Sink task does **not** delete: + * target documents already written to RavenDB + * the CDC Sink progress document (`@cdc-states/`) + * source-database change-capture or replication objects + +* This article shows how to delete a CDC Sink task using the Client API, Studio, or REST API, + and what source-side cleanup may still be needed. + +* In this article: + * [Delete a CDC Sink task via Client API](#delete-a-cdc-sink-task-via-client-api) + * [Delete a CDC Sink task via Studio](#delete-a-cdc-sink-task-via-studio) + * [Delete a CDC Sink task via REST API](#delete-a-cdc-sink-task-via-rest-api) + * [Clean up source-database artifacts](#clean-up-source-database-artifacts) + + + + + +Use `DeleteOngoingTaskOperation` with the numeric CDC Sink task ID and `OngoingTaskType.CdcSink`. + + + +```csharp +store.Maintenance.Send( + new DeleteOngoingTaskOperation(taskId, OngoingTaskType.CdcSink)); +``` + + +```csharp +await store.Maintenance.SendAsync( + new DeleteOngoingTaskOperation(taskId, OngoingTaskType.CdcSink)); +``` + + + + + + + +![Delete task](../assets/delete-task.png) + +1. Navigate to **Databases** → your database → **Tasks** → **Ongoing Tasks**. +2. On the CDC Sink task, click the **delete** (trash) button and confirm. + + + + + +To delete a CDC Sink task through REST, call the shared ongoing-task delete endpoint. +Pass the numeric task ID as `id` and the task type as `type=CdcSink`. + +| Method | Endpoint | Auth | +|----------|----------|------| +| `DELETE` | `/databases/{databaseName}/admin/tasks?id={taskId}&type=CdcSink` | `DatabaseAdmin` | + + + + + +Deleting a CDC Sink task removes it on the **RavenDB** side only. +RavenDB does **not** connect back to the source database to remove or disable CDC, replication, or privilege-related objects there. +That cleanup must be performed by an administrator with permissions on the source database. + +What remains depends on the source engine: + +* **PostgreSQL (logical replication)** + A **replication slot** and a **publication** are left behind. + Removing the slot is the priority: while it exists, PostgreSQL retains WAL segments indefinitely and can eventually fill the disk. + Drop the slot with `pg_drop_replication_slot()` and the publication with `DROP PUBLICATION`. + See [Cleanup and Maintenance](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/cleanup-and-maintenance.mdx). + +* **SQL Server (native CDC)** + CDC stays enabled at the table (and database) level, including the capture instances, their change tables, and the CDC capture/cleanup **SQL Server Agent jobs**. + Disable CDC on the affected tables with `sys.sp_cdc_disable_table` only if those capture instances are no longer used by any other consumer; + disable CDC on the database with `sys.sp_cdc_disable_db` only if no other tables use CDC. + +* **MySQL / MariaDB (binlog)** + Binary logging is a server-wide setting, not a per-task object, so there is usually nothing task-specific to remove. + Any dedicated **replication user or granted privileges** created for the task remain and can be revoked if no longer needed. + + diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_enable-disable-task-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_enable-disable-task-csharp.mdx new file mode 100644 index 0000000000..ee24dece73 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_enable-disable-task-csharp.mdx @@ -0,0 +1,181 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* You can enable or disable CDC Sink at two levels: + * **Task level**: + Pause or resume the entire CDC Sink task without deleting it. + A disabled task keeps its full configuration but stops consuming changes from the source. + * **Table level**: + Enable or disable an individual configured table. + A disabled table is skipped by CDC Sink while the task can continue syncing its other enabled tables. + +* A CDC Sink task can be reported as one of the following: + * `Enabled` - the task is enabled, with all configured tables enabled. + * `PartiallyEnabled` - the task is enabled, but some configured tables are disabled. + * `Disabled` - the task itself is disabled, or all configured tables are disabled. + +* This article shows how to enable or disable a CDC Sink task or a specific table using the Client API, Studio, + or REST API where applicable. + +* In this article: + * [Enable/Disable a CDC Sink task via Client API](#enable-disable-a-cdc-sink-task-via-client-api) + * [Enable/Disable a CDC Sink task via Studio](#enable-disable-a-cdc-sink-task-via-studio) + * [Enable/Disable a CDC Sink task via REST API](#enable-disable-a-cdc-sink-task-via-rest-api) + * [Enable/Disable a configured table via Client API](#enable-disable-a-configured-table-via-client-api) + * [Enable/Disable a configured table via Studio](#enable-disable-a-configured-table-via-studio) + * [Source-side effects of pausing a CDC Sink task](#source-side-effects-of-pausing-a-cdc-sink-task) + + + + + +Pause or resume the entire CDC Sink task using `ToggleOngoingTaskStateOperation`. +Pass the CDC Sink task ID, `OngoingTaskType.CdcSink`, and `disable: true` to pause or `disable: false` to resume. + +This operation affects the task as a whole. +To enable or disable a single table, see [Enable/Disable a configured table via Client API](#enable-disable-a-configured-table-via-client-api). + + + +```csharp +// Pause (disable) the task +store.Maintenance.Send( + new ToggleOngoingTaskStateOperation(taskId, OngoingTaskType.CdcSink, disable: true)); + +// Resume (enable) the task +store.Maintenance.Send( + new ToggleOngoingTaskStateOperation(taskId, OngoingTaskType.CdcSink, disable: false)); +``` + + +```csharp +// Pause (disable) the task +await store.Maintenance.SendAsync( + new ToggleOngoingTaskStateOperation(taskId, OngoingTaskType.CdcSink, disable: true)); + +// Resume (enable) the task +await store.Maintenance.SendAsync( + new ToggleOngoingTaskStateOperation(taskId, OngoingTaskType.CdcSink, disable: false)); +``` + + + + + + + +To enable or disable the entire CDC Sink task from the Studio: + +![Enable/Disable task](../assets/enable-disable-1.png) + +1. Navigate to **Databases** → your database → **Tasks** → **Ongoing Tasks**. +2. On the CDC Sink task, click its state dropdown and select **Enable** or **Disable**. + Alternatively, open the task for editing and use the **Task State** selector, then **Save**. + +To enable or disable individual tables, see [Enable/Disable a configured table via Studio](#enable-disable-a-configured-table-via-studio). + + + + + +Use the shared ongoing-task state endpoint to pause or resume the entire CDC Sink task. +Pass the CDC Sink task ID as `key`, `type=CdcSink`, and `disable=true` to pause or `disable=false` to resume. +No request body is required. + +| Method | Endpoint | Auth | +|--------|----------|------| +| `POST` | `/databases/{databaseName}/admin/tasks/state?key={taskId}&type=CdcSink&disable={true\|false}` | `DatabaseAdmin` | + +This endpoint affects the task as a whole. +To enable or disable a single table, see: + * [Enable/Disable a configured table via Client API](#enable-disable-a-configured-table-via-client-api) + * [Enable/Disable a configured table via Studio](#enable-disable-a-configured-table-via-studio) + + + + + +In addition to enabling or disabling the task as a whole, you can enable or disable an **individual table** within a CDC Sink task. +Each table is defined by a `CdcSinkTableConfig` that carries its own `Disabled` flag. + +When a table is disabled, CDC Sink skips it: +the table is not initial-loaded, and changes from that table are not applied to RavenDB while it is disabled. +The task's other enabled tables keep syncing. + +Disabling a single table is a configuration change, so there is no dedicated toggle operation for it. +Set `Disabled` on the relevant table and send the complete updated configuration with `UpdateCdcSinkOperation` +(the same operation used to [update a task](../../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/update-task.mdx)). + + + +```csharp +// config holds the existing task configuration. + +// Disable a single table by setting its Disabled flag +var table = config.Tables.First(t => t.SourceTableName == "orders"); +table.Disabled = true; + +// To re-enable the table later, set: +// table.Disabled = false; + +// Send the complete updated configuration +store.Maintenance.Send(new UpdateCdcSinkOperation(taskId, config)); +``` + + +```csharp +// config holds the existing task configuration. + +// Disable a single table by setting its Disabled flag +var table = config.Tables.First(t => t.SourceTableName == "orders"); +table.Disabled = true; + +// To re-enable the table later, set: +// table.Disabled = false; + +// Send the complete updated configuration +await store.Maintenance.SendAsync(new UpdateCdcSinkOperation(taskId, config)); +``` + + + + + + + +To enable or disable a configured root table from the Studio, open the CDC Sink task for editing +(**Ongoing Tasks** → click the task name or the edit icon), then go to the **Configured Tables** section: + +![Enable/Disable table](../assets/enable-disable-2.png) + +1. In the **Tables** list, click the three-dots **Table actions** icon for the table you want to change. +2. Select **Disable** (the same action shows **Enable** when the table is already disabled). + +Click **Save task configuration** to apply the change. + +A disabled table is shown dimmed in the **Tables** list. +If the task itself remains enabled and at least one other configured table remains enabled, +the task is listed as `PartiallyEnabled` in the **Ongoing Tasks** view. + + + + + +These effects apply when the **whole task is paused** and CDC Sink stops advancing its source-database position. +They do not apply in the same way when an individual table is disabled while the task keeps running. + +**PostgreSQL:** +Pausing a CDC Sink task stops the replication slot from being consumed. +PostgreSQL retains WAL segments for unconsumed slots, so pausing for an extended period causes WAL to accumulate on disk. +Monitor disk usage if a task is paused for more than a short time. +See [Monitoring PostgreSQL](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/monitoring-postgres.mdx). + +**SQL Server:** +Pausing a CDC Sink task for longer than SQL Server's CDC retention period can cause SQL Server to clean up CDC change-table rows that have not yet been consumed. +If cleanup occurs while the task is paused, CDC Sink will report an error on resume - the task may need to be recreated rather than simply resumed. + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_export-import-task-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_export-import-task-csharp.mdx new file mode 100644 index 0000000000..67847accec --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_export-import-task-csharp.mdx @@ -0,0 +1,202 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* A CDC Sink task definition is exported/imported through the database record, while its progress is stored separately. + +* **Task configuration** + The CDC Sink task definition (tables, column mappings, patches, provider settings) and the SQL connection string are database-record items. + They are included when the `DatabaseRecord` is exported/imported (`CdcSinks` and `SqlConnectionStrings` are part of the smuggler defaults). + +* **CDC progress** + The `LastLsn` checkpoint and per-table initial-load state are stored in [`@cdc-states/` documents](../../../../../server/ongoing-tasks/cdc-sink/how-it-works.mdx#the-state-document). + These are ordinary documents, so they are included only when documents are included and the `@cdc-states` collection is not filtered out. + +* **CDC-created documents** + Documents created by the CDC Sink task are also ordinary RavenDB documents. + They are exported/imported with the normal document stream and are not controlled by the `CdcSinks` flag. + +* In this article: + * [Export CDC Sink task](#export-cdc-sink-task) + * [Export via Client API](#export-via-client-api) + * [Export via Studio](#export-via-studio) + * [Import CDC Sink task](#import-cdc-sink-task) + * [Import via Client API](#import-via-client-api) + * [Import via Studio](#import-via-studio) + * [Import considerations](#import-considerations) + * [What is transferred](#what-is-transferred) + + + + + +By default, exporting a database includes: + +* the CDC Sink task definition (`CdcSinks`) and SQL connection strings (`SqlConnectionStrings`), + which are part of the database record. +* the `@cdc-states/` progress documents, + which are part of the normal document stream. + +You can exclude the task definition, SQL connection strings, the progress documents, or any combination of them. + + + +**For PostgreSQL**: +The export includes the configured `SlotName` and `PublicationName` because they are part of the task definition. +The actual replication slot and publication are not exported; they remain in PostgreSQL. +Reusing the same slot name when the imported task is enabled has additional implications, +see [Import CDC Sink task](#import-cdc-sink-task). + + + +--- + +### Export via Client API + +```csharp +// The default options include documents and the database record (with CDC Sink tasks) +var exportOptions = new DatabaseSmugglerExportOptions(); +// exportOptions.OperateOnTypes includes +// DatabaseItemType.DatabaseRecord. and DatabaseItemType.Documents by default +// exportOptions.OperateOnDatabaseRecordTypes includes +// DatabaseRecordItemType.CdcSinks by default + +var exportOp = await store.Smuggler.ExportAsync(exportOptions, "export.ravendbdump"); +await exportOp.WaitForCompletionAsync(); +``` +
+ +This writes the export to the file named in the second argument - here `export.ravendbdump`. +The [import example](#import-cdc-sink-task) below reads from that same file. + +To leave the CDC Sink task **configuration** out of the export, +clear the `CdcSinks` flag from `OperateOnDatabaseRecordTypes`: + +```csharp +exportOptions.OperateOnDatabaseRecordTypes &= ~DatabaseRecordItemType.CdcSinks; +``` +
+ +Clearing `CdcSinks` removes only the CDC Sink task definition from the export. +It does not remove SQL connection strings or `@cdc-states/` progress documents. +To exclude the progress documents as well, filter the `@cdc-states` collection out of the export. + +--- + +### Export via Studio + +* In Studio, open your database and go to **Tasks** → **Export Database**. + See [Export database](../../../../../studio/database/tasks/export-database.mdx). + +* **Include Configuration and Ongoing Tasks** is enabled by default, + so the CDC Sink task definition is exported with no extra step. + +* To exclude the CDC Sink task definition, turn on **Customize Configuration and Ongoing Tasks** under **Advanced** + and toggle off **CDC Sinks** under **Ongoing tasks**. This does not exclude `@cdc-states/` progress documents. + Explicitly exclude the `@cdc-states` collection if you want to leave them out as well. + + + + + +By default, import includes the CDC Sink task definition and SQL connection strings from the database record, +and imports any `@cdc-states/` progress documents present in the dump. + +When RavenDB imports a CDC Sink task definition, it creates the task **disabled**. +You must verify the imported settings and re-enable the task manually when the source database is ready. +RavenDB assigns the imported task a new task ID. + +--- + +### Import via Client API + +```csharp +// The default options include documents and the database record (with CDC Sink task definitions) +var importOptions = new DatabaseSmugglerImportOptions(); + +var importOp = await destinationStore.Smuggler.ImportAsync(importOptions, "export.ravendbdump"); +await importOp.WaitForCompletionAsync(); +``` +
+ +To skip importing the CDC Sink task **definition**, clear the `CdcSinks` flag from `OperateOnDatabaseRecordTypes`: + +```csharp +importOptions.OperateOnDatabaseRecordTypes &= ~DatabaseRecordItemType.CdcSinks; +``` + +--- + +### Import via Studio + +* In Studio, go to **Tasks** → **Import Data** → **From file (.ravendbdump)** and choose the exported file. + See [Import data from file](../../../../../studio/database/tasks/import-data/import-data-file.mdx). + +* **Include Configuration and Ongoing Tasks** is enabled by default, so the CDC Sink task definition is imported with no extra step. + To exclude the CDC Sink task definition, turn on **Customize Configuration and Ongoing Tasks** and toggle off **CDC Sinks** under **Ongoing tasks**. + +* After import, if the CDC Sink task definition was included, open **Ongoing Tasks**, find the imported task, + verify its connection string, and enable it when the source database is ready. + +--- + +### Import considerations + + + +#### Existing task with the same name + +If a CDC Sink task with the same **name** already exists in the target database, the imported task definition replaces it: +RavenDB deletes the existing task definition and adds the imported definition as a new, disabled task. + + + + + +#### Resume vs. fresh start + +Whether the imported task resumes or starts over depends on whether `@cdc-states/` exists in the target database: + +* **If the progress document exists in the target database**, the task resumes from that checkpoint when enabled - + continuing from the last `LastLsn` and skipping already-completed initial loads. +* **If it is absent**, the task starts fresh with a full initial load + (unless `SkipInitialLoad` is set on the configuration). + +To force a fresh start, delete `@cdc-states/` in the target database before enabling the task, +or exclude the `@cdc-states` collection from the import. + + + + + +#### PostgreSQL: shared replication slot + +The imported task carries the same `SlotName` and `PublicationName` as the original task. +A PostgreSQL replication slot allows only **one active consumer**, so before you enable the imported task, +either disable the original task or give the imported task its own slot and publication +(set a distinct `SlotName`/`PublicationName` in `CdcSinkPostgresSettings` before enabling). + +Otherwise, the imported task cannot attach to the slot and drops into fallback mode. + +See [One Active Consumer per Slot](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup.mdx#one-active-consumer-per-slot). + + + + + + + +| Item | Transferred on export/import? | +|-----------------------------------------------------------------------------------------|-------------------------------| +| CDC Sink task definition (tables, column mappings, patches, provider settings) | Yes (unless excluded) - it is part of the database record | +| SQL connection string | Yes (unless excluded) - it is part of the database record | +| Task enabled/disabled state | No - imported tasks are always **disabled** | +| Task ID | No - a new task ID is assigned on import | +| CDC progress state document (`@cdc-states/`: `LastLsn` + initial-load state) | Yes (unless the collection is excluded) - it is an ordinary document and moves with the document stream | +| Runtime position
(last consumed change / LSN) | Reused only if the matching `@cdc-states` document exists in the target database; otherwise the task starts fresh
(initial load, unless `SkipInitialLoad`) | +| PostgreSQL replication slot / publication | No - these remain in PostgreSQL and must be available before the imported task can use them | + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_get-task-info-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_get-task-info-csharp.mdx new file mode 100644 index 0000000000..f7ead196bc --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_get-task-info-csharp.mdx @@ -0,0 +1,136 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; +import ContentFrame from "@site/src/components/ContentFrame"; + + + +* Retrieve information about a CDC Sink task, including its configuration, task state, + connection status, responsible node, and last reported error. + +* This article shows how to get CDC Sink task info using the Client API, Studio, or REST API. + +* In this article: + * [Get CDC Sink task info via Client API](#get-cdc-sink-task-info-via-client-api) + * [Get CDC Sink task info via Studio](#get-cdc-sink-task-info-via-studio) + * [Get CDC Sink task info via REST API](#get-cdc-sink-task-info-via-rest-api) + + + + + +Use `GetOngoingTaskInfoOperation` with `OngoingTaskType.CdcSink` +to retrieve information about a specific CDC Sink task. + + + +### Get task info by task ID + +Pass the numeric `taskId` together with `OngoingTaskType.CdcSink`. + + + +```csharp +var taskInfo = (OngoingTaskCdcSink)store.Maintenance.Send( + new GetOngoingTaskInfoOperation(taskId, OngoingTaskType.CdcSink)); +``` + + +```csharp +var taskInfo = (OngoingTaskCdcSink)await store.Maintenance.SendAsync( + new GetOngoingTaskInfoOperation(taskId, OngoingTaskType.CdcSink)); +``` + + + + + + +### Get task info by task name + +Pass the task name together with `OngoingTaskType.CdcSink`. + + + +```csharp +var taskInfo = (OngoingTaskCdcSink)store.Maintenance.Send( + new GetOngoingTaskInfoOperation("MyCdcSinkTaskName", OngoingTaskType.CdcSink)); +``` + + +```csharp +var taskInfo = (OngoingTaskCdcSink)await store.Maintenance.SendAsync( + new GetOngoingTaskInfoOperation("MyCdcSinkTaskName", OngoingTaskType.CdcSink)); +``` + + + + + +--- + +The returned `OngoingTaskCdcSink` object exposes the task's identity, configuration, connection details, +and runtime status, including: + +**Identity** + +| Field | Description | +|-------|-------------| +| `TaskId` / `TaskName` | Task identity | + +**State & status** + +| Field | Description | +|-------|-------------| +| `TaskState` | `Enabled`, `Disabled`, or `PartiallyEnabled` | +| `TaskConnectionStatus` | `Active`, `NotActive`, `Reconnect`, or `NotOnThisNode` | +| `ResponsibleNode` | The cluster node currently running the task. | +| `Error` | The last error reported by the task, if any. | +| `HealthIssue` | `null` when healthy; a diagnostic message when the task detects a problem
(fallback mode, stale connection, etc.) | + +**Connection & configuration** + +| Field | Description | +|-------|-------------| +| `ConnectionStringName` | Name of the connection string the task uses. | +| `FactoryName` | The source provider (PostgreSQL, SQL Server, or MySQL/MariaDB). | +| `Configuration` | The full `CdcSinkConfiguration` (tables, column mappings, and other saved settings). | + +**Runtime & lag** + +| Field | Description | +|-------|-------------| +| `LastBatchTime` | UTC time of the last successfully completed batch.
`null` if no batch has completed yet. | +| `LastCheckpoint` | The last successfully persisted checkpoint (LSN/GTID). | +| `SecondsSinceLastBatch` | Seconds since the last successful batch. Provides a simple lag indicator.
`null` if no batch has completed yet. | +| `LastActivityTime` | UTC time of the last activity from the source
(poll iteration, replication message, or binlog event).
`null` before the first activity. | +| `SecondsSinceLastActivity` | Seconds since the last source activity.
A large value may indicate a dead connection. | + + +For performance and progress metrics (per-table statistics, initial-load progress), +see [Monitoring](../../../../../server/ongoing-tasks/cdc-sink/monitoring.mdx). + + + + + + +![Get task info](../assets/get-task-info.png) + +1. Navigate to **Databases** → your database → **Tasks** → **Ongoing Tasks**. +2. On the CDC Sink task row, click the **details toggle** to expand the task and open the detail view. + + + + + +CDC Sink uses the shared single-task info endpoint. +Identify the task either by ID (`key`) or by name (`taskName`), together with `type=CdcSink`. + +| Method | Endpoint | Auth | +|--------|----------|------| +| `GET` | `/databases/{databaseName}/task?key={taskId}&type=CdcSink` | `ValidUser` | +| `GET` | `/databases/{databaseName}/task?taskName={taskName}&type=CdcSink` | `ValidUser` | + + diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_update-task-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_update-task-csharp.mdx new file mode 100644 index 0000000000..b8628feabf --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/content/_update-task-csharp.mdx @@ -0,0 +1,110 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* Update an existing CDC Sink task to change its column mappings, add or remove tables, adjust patch scripts, + or modify other configuration settings. + +* This article shows how to **update** a CDC Sink task using the Client API, Studio, or the REST API. + +* In this article: + * [Update a CDC Sink task via Client API](#update-a-cdc-sink-task-via-client-api) + * [Update a CDC Sink task via Studio](#update-a-cdc-sink-task-via-studio) + * [Update a CDC Sink task via REST API](#update-a-cdc-sink-task-via-rest-api) + + + + + +Use `UpdateCdcSinkOperation` to modify an existing task. + +Start from the existing task configuration, apply your changes, and send the complete updated `CdcSinkConfiguration`. + +The update operation replaces the saved task configuration, so include any unchanged settings you want to keep. +The task to update is selected by the `taskId` passed to `UpdateCdcSinkOperation`. + + + +```csharp +// config contains the existing task configuration with the changes applied. + +config.Tables.Add(new CdcSinkTableConfig +{ + CollectionName = "Customers", + SourceTableName = "customers", + PrimaryKeyColumns = new List { "id" }, + Columns = + [ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "name", Name = "Name" }, + new CdcColumnMapping() { Column = "email", Name = "Email" }, + ] +}); + +// Execute the operation by passing it to Maintenance.Send +store.Maintenance.Send(new UpdateCdcSinkOperation(taskId, config)); +``` + + +```csharp +// config contains the existing task configuration with the changes applied. + +config.Tables.Add(new CdcSinkTableConfig +{ + CollectionName = "Customers", + SourceTableName = "customers", + PrimaryKeyColumns = new List { "id" }, + Columns = + [ + new CdcColumnMapping() { Column = "id", Name = "Id" }, + new CdcColumnMapping() { Column = "name", Name = "Name" }, + new CdcColumnMapping() { Column = "email", Name = "Email" }, + ] +}); + +// Execute the operation by passing it to Maintenance.SendAsync +await store.Maintenance.SendAsync(new UpdateCdcSinkOperation(taskId, config)); +``` + + + +--- + + + +For PostgreSQL tasks, adding or removing tables can require updating the PostgreSQL publication. +See [PostgreSQL - Cleanup and Maintenance](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/cleanup-and-maintenance.mdx) for details. + + + + + + + +To edit a CDC Sink task in the Studio: + +![Create task 12](../assets/create-task-12.png) + +1. Navigate to **Databases** → your database → **Tasks** → **Ongoing Tasks**. +2. Under the **SINK (SOURCE ⇒ RavenDB)** group, + click the task's name (or the edit icon) to open the same configuration form used for creation, now in edit mode. +3. Modify the connection string, tables, column mappings, or patches as needed, then click **Save**. + + + + + +To update a CDC Sink task through REST, send a `PUT` request to the same endpoint used to create a task, +with the task ID in the `id` query string parameter. + +The request body must contain the complete updated JSON `CdcSinkConfiguration`. +If `id` is omitted, the same endpoint creates a new CDC Sink task. + +| Method | Endpoint | Auth | +|--------|----------|------| +| `PUT` | `/databases/{databaseName}/admin/cdc-sink?id={taskId}` | `DatabaseAdmin` | + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task.mdx new file mode 100644 index 0000000000..af8cab7d9c --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task.mdx @@ -0,0 +1,35 @@ +--- +title: "CDC Sink: Create Task" +sidebar_label: Create Task +description: "How to create a CDC Sink task using the RavenDB Client API (AddCdcSinkOperation) and through the Management Studio." +sidebar_position: 1 +supported_languages: ["csharp"] +see_also: + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" + - title: "Update a Task" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/update-task" + source: "docs" + path: "Server > CDC Sink" + - title: "Overview" + link: "server/ongoing-tasks/cdc-sink/overview" + source: "docs" + path: "Server > CDC Sink" + - title: "What are Operations" + link: "client-api/operations/what-are-operations" + source: "docs" + path: "Client API > Operations" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import CreateTaskCsharp from './content/_create-task-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/delete-task.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/delete-task.mdx new file mode 100644 index 0000000000..3d5fda63be --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/delete-task.mdx @@ -0,0 +1,31 @@ +--- +title: "CDC Sink: Delete Task" +sidebar_label: Delete Task +description: "How to delete a CDC Sink task using the RavenDB Client API (DeleteOngoingTaskOperation) and through the Management Studio, including source-database cleanup." +sidebar_position: 4 +supported_languages: ["csharp"] +see_also: + - title: "Create a Task" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task" + source: "docs" + path: "Server > CDC Sink" + - title: "Cleanup and Maintenance" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/cleanup-and-maintenance" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Ongoing Task Operations" + link: "client-api/operations/maintenance/ongoing-tasks/ongoing-task-operations" + source: "docs" + path: "Client API > Operations" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import DeleteTaskCsharp from './content/_delete-task-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/enable-disable-task.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/enable-disable-task.mdx new file mode 100644 index 0000000000..c7f51ca05f --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/enable-disable-task.mdx @@ -0,0 +1,31 @@ +--- +title: "CDC Sink: Enable/Disable Task" +sidebar_label: Enable/Disable Task +description: "How to pause and resume a CDC Sink task using the RavenDB Client API (ToggleOngoingTaskStateOperation) and through the Management Studio." +sidebar_position: 5 +supported_languages: ["csharp"] +see_also: + - title: "Get Task Info" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/get-task-info" + source: "docs" + path: "Server > CDC Sink" + - title: "Monitoring" + link: "server/ongoing-tasks/cdc-sink/monitoring" + source: "docs" + path: "Server > CDC Sink" + - title: "Ongoing Task Operations" + link: "client-api/operations/maintenance/ongoing-tasks/ongoing-task-operations" + source: "docs" + path: "Client API > Operations" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import EnableDisableTaskCsharp from './content/_enable-disable-task-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/export-import-task.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/export-import-task.mdx new file mode 100644 index 0000000000..c0e54e8ad1 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/export-import-task.mdx @@ -0,0 +1,31 @@ +--- +title: "CDC Sink: Export/Import Task" +sidebar_label: Export/Import Task +description: "How CDC Sink tasks are handled when exporting and importing a database using the RavenDB Client API (Smuggler) and the Management Studio." +sidebar_position: 6 +supported_languages: ["csharp"] +see_also: + - title: "Backup / Restore a Task" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/backup-restore-task" + source: "docs" + path: "Server > CDC Sink" + - title: "Enable / Disable a Task" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/enable-disable-task" + source: "docs" + path: "Server > CDC Sink" + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import ExportImportTaskCsharp from './content/_export-import-task-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/get-task-info.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/get-task-info.mdx new file mode 100644 index 0000000000..372e42311a --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/get-task-info.mdx @@ -0,0 +1,31 @@ +--- +title: "CDC Sink: Get Task Info" +sidebar_label: Get Task Info +description: "How to retrieve the current state of a CDC Sink task using the RavenDB Client API (GetOngoingTaskInfoOperation) and through the Management Studio." +sidebar_position: 3 +supported_languages: ["csharp"] +see_also: + - title: "Enable / Disable a Task" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/enable-disable-task" + source: "docs" + path: "Server > CDC Sink" + - title: "Monitoring" + link: "server/ongoing-tasks/cdc-sink/monitoring" + source: "docs" + path: "Server > CDC Sink" + - title: "Ongoing Task Operations" + link: "client-api/operations/maintenance/ongoing-tasks/ongoing-task-operations" + source: "docs" + path: "Client API > Operations" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import GetTaskInfoCsharp from './content/_get-task-info-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/update-task.mdx b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/update-task.mdx new file mode 100644 index 0000000000..fccf9fb447 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/update-task.mdx @@ -0,0 +1,31 @@ +--- +title: "CDC Sink: Update Task" +sidebar_label: Update Task +description: "How to update an existing CDC Sink task using the RavenDB Client API (UpdateCdcSinkOperation) and through the Management Studio." +sidebar_position: 2 +supported_languages: ["csharp"] +see_also: + - title: "Create a Task" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task" + source: "docs" + path: "Server > CDC Sink" + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" + - title: "Source Schema Changes" + link: "server/ongoing-tasks/cdc-sink/source-schema-changes" + source: "docs" + path: "Server > CDC Sink" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import UpdateTaskCsharp from './content/_update-task-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/monitoring.mdx b/docs/server/ongoing-tasks/cdc-sink/monitoring.mdx new file mode 100644 index 0000000000..33420c4421 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/monitoring.mdx @@ -0,0 +1,35 @@ +--- +title: "CDC Sink: Monitoring" +sidebar_label: Monitoring +description: "Explains how to monitor a CDC Sink task - its running state, progress, fallback behavior, and statistics available through Management Studio and the API." +sidebar_position: 11 +supported_languages: ["csharp"] +see_also: + - title: "How It Works" + link: "server/ongoing-tasks/cdc-sink/how-it-works" + source: "docs" + path: "Server > CDC Sink" + - title: "Failover and Consistency" + link: "server/ongoing-tasks/cdc-sink/failover-and-consistency" + source: "docs" + path: "Server > CDC Sink" + - title: "Server Configuration" + link: "server/ongoing-tasks/cdc-sink/server-configuration" + source: "docs" + path: "Server > CDC Sink" + - title: "Troubleshooting" + link: "server/ongoing-tasks/cdc-sink/troubleshooting" + source: "docs" + path: "Server > CDC Sink" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import MonitoringCsharp from './content/_monitoring-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/overview.mdx b/docs/server/ongoing-tasks/cdc-sink/overview.mdx new file mode 100644 index 0000000000..4022bd3345 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/overview.mdx @@ -0,0 +1,170 @@ +--- +title: "CDC Sink: Overview" +sidebar_label: Overview +description: "An introduction to CDC Sink, the RavenDB ongoing task that reads Change Data Capture streams from a relational database and writes documents into RavenDB." +sidebar_position: 0 +see_also: + - title: "Create a Task" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task" + source: "docs" + path: "Server > CDC Sink" + - title: "How It Works" + link: "server/ongoing-tasks/cdc-sink/how-it-works" + source: "docs" + path: "Server > CDC Sink" + - title: "Schema Design" + link: "server/ongoing-tasks/cdc-sink/document-modeling/schema-design" + source: "docs" + path: "Server > CDC Sink" + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" + - title: "PostgreSQL Prerequisites Checklist" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/prerequisites-checklist" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "PostgreSQL Initial Setup" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "ETL Basics" + link: "server/ongoing-tasks/etl/basics" + source: "docs" + path: "Server > Ongoing Tasks" + - title: "Queue Sink Overview" + link: "server/ongoing-tasks/queue-sink/overview" + source: "docs" + path: "Server > Ongoing Tasks" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* **CDC Sink** is a RavenDB ongoing task that reads **Change Data Capture (CDC) streams** from a relational database + and writes the resulting documents into RavenDB. + +* CDC Sink is the reverse of SQL ETL: + instead of pushing data _from_ RavenDB _to_ a relational database, + CDC Sink _pulls_ data _from_ the relational database _into_ RavenDB. + +* The relational database remains the source of truth, + while the CDC Sink task continuously maps normalized relational tables into RavenDB documents. + +* Supported source databases: + * **PostgreSQL** - via logical replication + * **SQL Server** - via native CDC (polling change tables) + * **MySQL / MariaDB** - via binlog replication + +* In this article: + * [Why use CDC Sink](#why-use-cdc-sink) + * [How it works](#how-it-works) + * [Task lifecycle](#task-lifecycle) + * [Licensing](#licensing) + + + + + +CDC Sink solves the problem of moving data from a relational database into RavenDB without changing the source application. + +* **Migrate from SQL to RavenDB** + Automatically and continuously transform normalized relational tables into RavenDB documents, + such as an _Order_ document that embeds _LineItems_ and references a _Customer_ document, + without changing your application. + +* **Build a read-optimized view** + Your transactional system uses a relational database, but your API layer needs denormalized documents. + CDC Sink creates and maintains those documents without touching your existing application. + +* **Gradual migration** + Keep your existing application running while RavenDB documents are built in the background. + Applications can start reading from RavenDB while writes still go to the relational database. + +* **Event-driven side effects** + Using JavaScript patches, every INSERT, UPDATE, or DELETE in the source database can trigger custom logic in RavenDB - + computing derived fields, maintaining running totals, or writing custom transformations. + + + + + +A CDC Sink task continuously reads changes from the source relational database and applies them to RavenDB documents. + +### Initial load + +By default, when a CDC Sink task starts for the first time, it performs a full scan of every configured table using keyset pagination. +This populates RavenDB with the current state of the data before streaming begins. + +The task can also be configured to skip the full scan and start from new CDC changes only, without scanning existing rows - +useful when you only need to capture changes going forward. + +Initial load progress is persisted per-table. +If the task is restarted, it resumes from where it left off rather than re-scanning. + +### Streaming changes + +After the initial load, CDC Sink switches to streaming changes continuously. +Changes are grouped into transactions, preserving the exact order of operations from the source database. +Partial transactions are never written to RavenDB - all changes within a source database transaction are applied together. + +### Document model + +The relational model is mapped to RavenDB documents through configuration: + +* **Root tables** map to RavenDB collections (one document per row) +* **Embedded tables** become nested arrays or objects within parent documents +* **Linked tables** become document ID references to _related documents_ + +See [Schema design](../../../server/ongoing-tasks/cdc-sink/document-modeling/schema-design.mdx) for details. + + + + + +1. **Create** + Define the CDC Sink task in Studio or via the Client API. + Specify the connection string, table mappings, and transformation options. + See [Create a CDC Sink task](../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/create-task.mdx). + You can later [Update](../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/update-task.mdx) or + [Enable/Disable](../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/enable-disable-task.mdx) the task. + +2. **Verify** + CDC Sink verifies that the source database is properly configured, + including permissions, replication prerequisites, and table configuration. + +3. **Initial load** + Full table scan populates RavenDB with current data. + Progress is tracked per table and persists across restarts. + +4. **Stream** + Continuous change streaming begins. + All INSERTs, UPDATEs, and DELETEs are applied to RavenDB documents as they occur. + +5. **Monitor** + [Monitor the task state](../../../server/ongoing-tasks/cdc-sink/monitoring.mdx), + or [Get task info](../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/get-task-info.mdx) via Studio or via the Client API. + +6. **Retire** + Delete the task in RavenDB when no longer needed. + Retiring a task removes it on the **RavenDB** side only. + + Change-tracking artifacts created in the source database (such as a PostgreSQL replication slot and publication) + are **not** removed automatically and must be cleaned up by the source-database administrator. + Learn more in [Delete a CDC Sink task](../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/delete-task.mdx). + + + + + + +CDC Sink is available on an **Enterprise** license. + + +Learn more about licensing in [Licensing overview](../../../licensing/overview.mdx). + + diff --git a/docs/server/ongoing-tasks/cdc-sink/patching.mdx b/docs/server/ongoing-tasks/cdc-sink/patching.mdx new file mode 100644 index 0000000000..e514f5427f --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/patching.mdx @@ -0,0 +1,39 @@ +--- +title: "CDC Sink: Patching" +sidebar_label: Patching +description: "Explains how JavaScript patch scripts work in CDC Sink, covering available variables, patch scope, common scenarios, idempotency, and limitations." +sidebar_position: 6 +supported_languages: ["csharp"] +see_also: + - title: "Handling Deletes" + link: "server/ongoing-tasks/cdc-sink/handling-deletes" + source: "docs" + path: "Server > CDC Sink" + - title: "Column Mapping" + link: "server/ongoing-tasks/cdc-sink/document-modeling/column-mapping" + source: "docs" + path: "Server > CDC Sink" + - title: "Property Retention" + link: "server/ongoing-tasks/cdc-sink/property-retention" + source: "docs" + path: "Server > CDC Sink" + - title: "Failover and Consistency" + link: "server/ongoing-tasks/cdc-sink/failover-and-consistency" + source: "docs" + path: "Server > CDC Sink" + - title: "Patching Configuration" + link: "server/configuration/patching-configuration" + source: "docs" + path: "Server > Configuration" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import PatchingCsharp from './content/_patching-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/property-retention.mdx b/docs/server/ongoing-tasks/cdc-sink/property-retention.mdx new file mode 100644 index 0000000000..26f19bf54d --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/property-retention.mdx @@ -0,0 +1,164 @@ +--- +title: "CDC Sink: Property Retention" +sidebar_label: Property Retention +description: "Explains how CDC Sink merges UPDATE events onto existing RavenDB documents, preserving properties that are not part of the column mapping." +sidebar_position: 5 +see_also: + - title: "Column Mapping" + link: "server/ongoing-tasks/cdc-sink/document-modeling/column-mapping" + source: "docs" + path: "Server > CDC Sink" + - title: "Patching" + link: "server/ongoing-tasks/cdc-sink/patching" + source: "docs" + path: "Server > CDC Sink" + - title: "How It Works" + link: "server/ongoing-tasks/cdc-sink/how-it-works" + source: "docs" + path: "Server > CDC Sink" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; +import ContentFrame from '@site/src/components/ContentFrame'; + + + +* When CDC Sink applies an UPDATE to an existing RavenDB document, + it merges the mapped column values into the current document instead of replacing the whole document. + +* **Unmapped properties**: + Unmapped document properties are left unchanged. + For example, if you add `ReviewedAt`, `InternalNotes`, or a computed flag in RavenDB, + and that property is not mapped from a source column, later CDC updates will keep it. + +* **Mapped properties**: + Mapped properties are different: each CDC update writes the current source value again. + Custom `@metadata` keys are also not retained by the root document write unless a patch re-applies them. + +* In this article: + * [How merging works](#how-merging-works) + * [What is and is not preserved](#what-is-and-is-not-preserved) + * [Editing documents directly in RavenDB](#editing-documents-directly-in-ravendb) + * [Implications for patches](#implications-for-patches) + + + + + +When a CDC UPDATE arrives for a document that already exists: + +1. The existing document is loaded. +2. Mapped column values from the CDC event overwrite the corresponding document properties. +3. Unmapped document properties are left unchanged. +4. CDC Sink writes the merged document back, with fresh root `@metadata` containing the collection name. + +**Example:** + +Existing RavenDB document before the CDC UPDATE: + + + +```json +{ + "Id": 1, + "Name": "Alice", + "Email": "alice@example.com", + "InternalNotes": "VIP customer", + "@metadata": { "@collection": "Customers" } +} +``` + + + +SQL UPDATE: `UPDATE customers SET email = 'alice.new@example.com' WHERE id = 1` + +Document after CDC UPDATE: + + + +```json +{ + "Id": 1, + "Name": "Alice", + "Email": "alice.new@example.com", + "InternalNotes": "VIP customer", + "@metadata": { "@collection": "Customers" } +} +``` + + + +`Email` is updated from SQL because it is mapped, +while `InternalNotes` is not in the `Columns` list and is therefore preserved. + + + + + +**Preserved across CDC updates:** + +* Document properties that are not listed in the `Columns` list. +* Properties added or edited directly in RavenDB, such as annotations, computed values, or flags, + as long as they are not mapped from source columns. + +**Overwritten on CDC update:** + +* Properties mapped via the `Columns` list - + the CDC event writes the current source value for each mapped property. +* Custom `@metadata` keys - + when CDC Sink creates or updates the root document, it writes fresh document metadata containing only the collection name; + RavenDB then re-adds system metadata such as `@id`, `@change-vector`, and `@last-modified`. + +If you manually edit a mapped property in RavenDB, +the next CDC UPDATE for that row will overwrite your edit with the source value. + + + + + +You can safely add unmapped document properties to CDC-managed documents: + + + + +```json +{ + "Id": 1, + "Name": "Alice", + "Email": "alice@ex.com", + "InternalNotes": "...", + "ReviewedAt": "...", + "Tags": ["vip"] +} +``` + + + + +Properties mapped from source columns are overwritten on the next UPDATE from the source database. +Do not rely on manual edits to mapped properties surviving future CDC updates. + +**CDC Sink does not detect or protect manual edits to mapped properties**. If you need to preserve your own value, +add a separate RavenDB-only property and leave the source-mapped property as-is. + + + + + +Patches run after column mapping. +On UPDATE, the patch sees the merged document and can add unmapped properties or override values that came from mapped columns. + +* If a patch sets an unmapped property, such as `this.ComputedField = ...`, + that property is stored on the document and is preserved by later CDC updates unless a later patch changes or removes it. + +* If a patch sets a property that is also mapped in the `Columns` list, + the patch value wins for that CDC event because mapping is applied first and the patch runs afterward. + +For values maintained by patches, such as `RunningTotal`, the patch script is responsible for keeping the value correct across the events it handles. + +Learn more in [Patching](../../../server/ongoing-tasks/cdc-sink/patching.mdx). + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/server-configuration.mdx b/docs/server/ongoing-tasks/cdc-sink/server-configuration.mdx new file mode 100644 index 0000000000..0c1949b25b --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/server-configuration.mdx @@ -0,0 +1,109 @@ +--- +title: "CDC Sink: Server Configuration" +sidebar_label: Server Configuration +description: "Documents the RavenDB server configuration keys that control CDC Sink task behavior, including batch size, fallback timeout, and poll interval." +sidebar_position: 10 +see_also: + - title: "Overview" + link: "server/ongoing-tasks/cdc-sink/overview" + source: "docs" + path: "Server > CDC Sink" + - title: "Monitoring" + link: "server/ongoing-tasks/cdc-sink/monitoring" + source: "docs" + path: "Server > CDC Sink" + - title: "Failover and Consistency" + link: "server/ongoing-tasks/cdc-sink/failover-and-consistency" + source: "docs" + path: "Server > CDC Sink" + - title: "Configuration Overview" + link: "server/configuration/configuration-options" + source: "docs" + path: "Server > Configuration" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import ContentFrame from '@site/src/components/ContentFrame'; + + + +* This article documents the RavenDB server configuration keys that control CDC Sink task behavior. + +* These keys can be set in `settings.json` or passed as environment variables. + All CDC Sink keys have a `Server-wide or per database` scope, so they can be set per database as well as server-wide. + Learn how to apply these keys in the [Configuration overview](../../../server/configuration/configuration-options.mdx) article. + +* In this article: + * [CdcSink.MaxBatchSize](#cdcsinkmaxbatchsize) + * [CdcSink.MaxFallbackTimeInSec](#cdcsinkmaxfallbacktimeinsec) + * [CdcSink.SqlServer.PollIntervalInSec](#cdcsinksqlserverpollintervalinsec) + * [CdcSink.Postgres.ReplicationTimeoutInSec](#cdcsinkpostgresreplicationtimeoutinsec) + + + + + +### CdcSink.MaxBatchSize + +Target number of change rows processed in a single batch before writing to RavenDB. +A batch may exceed this size when a source database transaction contains more rows, +since transactions are never split across batches. + +Larger values increase throughput but also increase memory usage per batch. + +- **Type**: `int` +- **Default**: `1024` +- **Scope**: Server-wide or per database + + + + + +### CdcSink.MaxFallbackTimeInSec + +Maximum number of seconds used as the base cap for CDC Sink fallback mode after a failure. +The first retry after a failure waits 5 seconds. +Subsequent retries roughly double based on the time since the previous error, up to this configured base cap. + +A small random jitter is added after the base delay to avoid synchronized retries across multiple tasks, +so the actual wait can be slightly higher than this value. + +- **Type**: `int` +- **Default**: `900` (15 minutes) +- **Scope**: Server-wide or per database + + + + + +### CdcSink.SqlServer.PollIntervalInSec + +How frequently (in seconds) the SQL Server CDC Sink polls for new change rows. +Lower values reduce latency but increase load on the source database. + +PostgreSQL and MySQL use streaming replication and ignore this setting. + +- **Type**: `int` +- **Default**: `1` +- **Scope**: Server-wide or per database + + + + + +### CdcSink.Postgres.ReplicationTimeoutInSec + +Timeout (in seconds) for the PostgreSQL replication connection. +Controls both the server-side `wal_sender_timeout` (keepalives arrive at roughly half this interval) and the client-side `WalReceiverTimeout`. + +Lower values detect dead connections faster but increase keepalive traffic. + +SQL Server and MySQL ignore this setting. + +- **Type**: `int` +- **Default**: `10` +- **Scope**: Server-wide or per database + + diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/_category_.json b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/_category_.json new file mode 100644 index 0000000000..542c452f50 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/_category_.json @@ -0,0 +1,4 @@ +{ + "position": 2, + "label": "Source Database Setup" +} diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/mysql/_category_.json b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/mysql/_category_.json new file mode 100644 index 0000000000..c0d9bbe1c7 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/mysql/_category_.json @@ -0,0 +1,4 @@ +{ + "position": 2, + "label": "MySQL" +} diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/mysql/content/_overview-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/mysql/content/_overview-csharp.mdx new file mode 100644 index 0000000000..61eb49a138 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/mysql/content/_overview-csharp.mdx @@ -0,0 +1,125 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* CDC Sink supports MySQL and MariaDB as source databases using + [binary log (binlog) replication](https://dev.mysql.com/doc/refman/8.0/en/replication.html). + +* MySQL and MariaDB record row-level changes in the binlog. + The CDC Sink task reads the binlog stream continuously and processes INSERT, UPDATE, and DELETE events according to the task configuration. + +* In this article: + * [Connection string](#connection-string) + * [Prerequisites](#prerequisites) + * [Streaming behavior](#streaming-behavior) + * [Task configuration](#task-configuration) + + + + + +Use a [SqlConnectionString](../../../../../../server/ongoing-tasks/etl/sql.mdx#add-an-sql-connection-string) with `FactoryName` set to `"MySqlConnector.MySqlConnectorFactory"`. + +The legacy `"MySql.Data.MySqlClient"` factory name is still accepted, +but the server automatically replaces it with `"MySqlConnector.MySqlConnectorFactory"` and raises a deprecation notification. + + + +```csharp +store.Maintenance.Send(new PutConnectionStringOperation( + new SqlConnectionString + { + Name = "MySqlConnection", + FactoryName = "MySqlConnector.MySqlConnectorFactory", + ConnectionString = + "Server=myserver;Port=3306;" + + "Database=mydb;User=ravendb_cdc;Password=..." + })); +``` + + +```csharp +await store.Maintenance.SendAsync(new PutConnectionStringOperation( + new SqlConnectionString + { + Name = "MySqlConnection", + FactoryName = "MySqlConnector.MySqlConnectorFactory", + ConnectionString = + "Server=myserver;Port=3306;" + + "Database=mydb;User=ravendb_cdc;Password=..." + })); +``` + + + + + + + +MySQL/MariaDB must have binary logging enabled with row-based format: + + + +```ini +# my.cnf / my.ini +[mysqld] +log-bin = mysql-bin +binlog_format = ROW +binlog_row_image = FULL +server-id = 1 + +# MySQL only: CDC Sink requires GTID. +# This is not needed for MariaDB. +enforce_gtid_consistency = ON +gtid_mode = ON +``` + + + + + +On **MySQL**, CDC Sink requires `gtid_mode = ON`. +If GTID is not already enabled, configure it before starting the CDC Sink task. +For an existing MySQL server, follow MySQL's GTID enablement procedure. + +**MariaDB** always has GTID support enabled, so no `gtid_mode` setting is needed there. + + + +The MySQL/MariaDB user in the CDC Sink connection string needs the following privileges: + + + +```sql +GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'ravendb_cdc'@'%'; +GRANT SELECT ON mydb.* TO 'ravendb_cdc'@'%'; +``` + + + +* `REPLICATION SLAVE` - required to read the binlog stream +* `REPLICATION CLIENT` - required to query binlog status +* `SELECT` - required for the initial full-table load + + + + + +After the initial load, MySQL/MariaDB CDC Sink reads row-based binlog events continuously. +It does not poll the source database for changes, so the `CdcSink.SqlServer.PollIntervalInSec` server configuration setting +applies only to SQL Server CDC Sink tasks and has no effect on MySQL/MariaDB. + + + + + +Most CDC Sink task configuration is shared across supported source database engines. +See [Configuration Reference](../../../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference.mdx) +for full details, including table mappings, delete behavior, and patches. + +When `SourceTableSchema` is omitted, MySQL/MariaDB uses the database name from the connection string as the default schema. + + diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/mysql/overview.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/mysql/overview.mdx new file mode 100644 index 0000000000..bcba6e5e89 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/mysql/overview.mdx @@ -0,0 +1,27 @@ +--- +title: "CDC Sink for MySQL: Overview" +sidebar_label: Overview +description: "CDC Sink support for MySQL and MariaDB - using binlog replication to stream changes into RavenDB in real time." +sidebar_position: 0 +supported_languages: ["csharp"] +see_also: + - title: "CDC Sink Overview" + link: "server/ongoing-tasks/cdc-sink/overview" + source: "docs" + path: "Server > CDC Sink" + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import OverviewCsharp from './content/_overview-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/_category_.json b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/_category_.json new file mode 100644 index 0000000000..3825482859 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/_category_.json @@ -0,0 +1,4 @@ +{ + "position": 0, + "label": "PostgreSQL" +} diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/cleanup-and-maintenance.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/cleanup-and-maintenance.mdx new file mode 100644 index 0000000000..4d473e8ee7 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/cleanup-and-maintenance.mdx @@ -0,0 +1,404 @@ +--- +title: "CDC Sink for PostgreSQL: Cleanup and Maintenance" +sidebar_label: Cleanup and Maintenance +description: "How to identify and remove orphaned PostgreSQL replication slots and publications after a CDC Sink task is deleted, and how to avoid slot accumulation." +sidebar_position: 8 +see_also: + - title: "Initial Setup" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Monitoring PostgreSQL" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/monitoring-postgres" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "WAL Configuration" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/wal-configuration" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* When a CDC Sink task is deleted from RavenDB, PostgreSQL-side resources used by that task are **not** automatically cleaned up. + This includes the replication slot, the publication, and any `REPLICA IDENTITY FULL` changes that CDC Sink applied to source tables. + +* A database administrator must decide which resources to keep and which to clean up manually. + +* In this article: + * [Why manual cleanup is required](#why-manual-cleanup-is-required) + * [Disabled and stopped tasks keep their slot](#disabled-and-stopped-tasks-keep-their-slot) + * [Finding orphaned slots and publications](#finding-orphaned-slots-and-publications) + * [Dropping a replication slot](#dropping-a-replication-slot) + * [Dropping a publication](#dropping-a-publication) + * [Too many replication slots](#too-many-replication-slots) + * [Preventing WAL from filling the disk](#preventing-wal-from-filling-the-disk) + * [Reverting REPLICA IDENTITY changes](#reverting-replica-identity-changes) + * [Slot and publication names are immutable](#slot-and-publication-names-are-immutable) + * [Keeping the publication in sync when tables change](#keeping-the-publication-in-sync-when-tables-change) + + + + + +A replication slot tells PostgreSQL to retain WAL until the consumer confirms it has processed the changes by advancing the slot's `confirmed_flush_lsn`. +If CDC Sink stops consuming from the slot because the task was deleted, `confirmed_flush_lsn` no longer advances, so PostgreSQL can continue retaining WAL for that slot. + +This can lead to: + +* Disk space exhaustion on the PostgreSQL server +* Degraded performance as old WAL segments pile up + +When the slot is no longer needed, a database administrator should drop it. + + + +#### Why RavenDB does not drop slots automatically: + +There are valid reasons to keep a slot after a task is deleted. + +For example, you may want to create a new CDC Sink task that resumes from the same position by reusing the existing slot, +or you may want to review what changes are pending before cleaning up. +The PostgreSQL user configured in the CDC Sink connection string may also not have the permissions required to drop replication slots, +even if it had the permissions to create them. + +For these reasons, PostgreSQL-side cleanup is the responsibility of the database administrator. + + + + + + + +Deleting a task is not the only way WAL accumulates. + +A task that is **disabled** or **stopped** (for example, because the responsible node is down or the connection to PostgreSQL is failing) +is not consuming from PostgreSQL, but it **keeps its replication slot**. + +While the task is not consuming, `confirmed_flush_lsn` stops advancing and PostgreSQL retains WAL for that slot exactly as it would for a deleted task - +except that the task still exists, so the slot will not show up as "orphaned". + +This makes long-disabled tasks a common and easily overlooked cause of disk growth. +When you disable a task that you do not intend to re-enable soon, decide deliberately whether to keep the slot so the task can resume from the same position later, +or drop it and accept that the retained WAL position is gone. + + + +To see how much WAL each slot is holding - active or not - use the retained-WAL query in [Monitoring PostgreSQL](./monitoring-postgres.mdx). + + + + + + + +List CDC Sink replication slots that use the auto-generated prefix: + + + +```sql +SELECT slot_name, active, confirmed_flush_lsn +FROM pg_replication_slots +WHERE slot_name LIKE 'rvn_cdc_s_%'; +``` + + + +An `active = false` slot is not being consumed. +Compare the list to your CDC Sink tasks in RavenDB - any slot whose corresponding task no longer exists is orphaned. + +Note that an inactive slot does not always mean the task is gone: +a disabled or stopped task also leaves its slot inactive (see [Disabled and stopped tasks keep their slot](#disabled-and-stopped-tasks-keep-their-slot)). + +To see how much WAL each slot is retaining, use the query in [Monitoring PostgreSQL](./monitoring-postgres.mdx). + +List CDC Sink publications that use the auto-generated publication prefix: + + + +```sql +SELECT p.pubname, + p.puballtables, + n.nspname AS schema_name, + c.relname AS table_name +FROM pg_publication p +LEFT JOIN pg_publication_rel pr ON pr.prpubid = p.oid +LEFT JOIN pg_class c ON c.oid = pr.prrelid +LEFT JOIN pg_namespace n ON n.oid = c.relnamespace +WHERE p.pubname LIKE 'rvn_cdc_p_%' +ORDER BY p.pubname, n.nspname, c.relname; +``` + + + + + +The `rvn_cdc_s_` and `rvn_cdc_p_` prefixes match only **auto-generated** names. + +If a slot or publication was created with a custom name, either supplied when the task was created or pre-created by an administrator, +drop the `LIKE` filter and cross-check the full list against your CDC Sink task configurations instead. + + + + + + + +Use the actual slot name from the CDC Sink task configuration or from `pg_replication_slots`: + + + +```sql +SELECT pg_drop_replication_slot(''); +``` + + + + + +You cannot drop an active replication slot (one with `active = true`). +Disable or delete the CDC Sink task, or otherwise stop the process that is consuming the slot, before dropping it. + + + +Dropping a slot discards the retained WAL position for that slot. +If a CDC Sink task is later started with saved state that points to WAL no longer retained by the slot, +CDC Sink can report an LSN gap; changes in that gap cannot be recovered from the slot. + +Even after the task is stopped or deleted, the slot can remain `active` for a short time. +PostgreSQL does not release the slot until the streaming connection (the WAL sender) closes, +which can take up to `wal_sender_timeout` (60 seconds by default). + +If `pg_drop_replication_slot` fails with *"replication slot is active for PID …"*, either wait for the timeout, +or terminate the lingering WAL sender: + + + +```sql +SELECT pg_terminate_backend(active_pid) +FROM pg_replication_slots +WHERE slot_name = '' AND active_pid IS NOT NULL; +``` + + + + + +This is also where you drop a slot that a task reports as using the **wrong output plugin** (*"exists but uses plugin … instead of 'pgoutput'"*). +Dropping it here lets CDC Sink recreate it with the correct plugin on the next run. +See [Task enters error state](../../../../../server/ongoing-tasks/cdc-sink/troubleshooting.mdx#task-enters-error-state) for the diagnosis. + + + + + + + +Use the actual publication name from the CDC Sink task configuration or from `pg_publication`: + + + +```sql +DROP PUBLICATION IF EXISTS ""; +``` + + + +Publications are not consumed like slots - they do not accumulate data or hold WAL segments. +However, they should still be dropped when they are no longer needed, to keep the PostgreSQL database clean. + + + +Drop a publication only after confirming that no CDC Sink task or other replication consumer still depends on it. +This is especially important for custom or pre-created publication names. + + + + + + + +PostgreSQL limits the total number of replication slots to `max_replication_slots`. +If the limit is reached, CDC Sink tasks that need to create a new replication slot can fail to start. + +Check how many slots exist and what the configured limit is: + + + +```sql +SELECT count(*) AS used_slots FROM pg_replication_slots; +SHOW max_replication_slots; +``` + + + +To resolve this, first identify slots that are not currently being consumed: + + + +```sql +SELECT slot_name, active +FROM pg_replication_slots +WHERE active = false; +``` + + + +Compare the inactive slots with your CDC Sink task configurations. Drop only slots that are no longer needed: + + + +```sql +SELECT pg_drop_replication_slot(''); +``` + + + +If the remaining slots are all still needed, increase `max_replication_slots` in `postgresql.conf` and restart PostgreSQL. + + + + + +Regular cleanup removes slots you no longer need, but as a safety net you can also cap how much WAL a single slot is allowed to retain. +On PostgreSQL 13 and later, set `max_slot_wal_keep_size` in `postgresql.conf`, or with `ALTER SYSTEM`: + + + +```sql +-- Retain at most 10 GB of WAL for any single slot +ALTER SYSTEM SET max_slot_wal_keep_size = '10GB'; +SELECT pg_reload_conf(); +``` + + + +If a slot falls further behind than this limit, PostgreSQL can stop retaining WAL for it and the slot can become **lost**. +This protects the server from running out of disk, but the slot can no longer stream the missing changes. + +If CDC Sink has saved state that points to WAL no longer retained by the slot, CDC Sink can report an LSN gap; +changes in that gap cannot be recovered from the slot. + +Check each slot's WAL status: + + + + +```sql +SELECT slot_name, active, wal_status, safe_wal_size +FROM pg_replication_slots; +``` + + + + +A `wal_status` of `lost` means the slot has already been invalidated. +Use `safe_wal_size` to see how much more WAL can be written before the slot is at risk. +Treat this setting as a safeguard, not a substitute for dropping slots you no longer need. + + + + + +Besides the slot and publication, CDC Sink may modify **embedded source tables**. + +When an embedded table's current replica identity does not include the columns needed to route `DELETE` events, +CDC Sink can run `ALTER TABLE REPLICA IDENTITY FULL` on that table. +This change is made on the PostgreSQL table itself, so it **persists after the CDC Sink task is deleted**. + +`REPLICA IDENTITY FULL` causes PostgreSQL to write the full old row into WAL for every `UPDATE` and `DELETE` on that table, +which increases WAL volume even for other consumers. + +After cleanup, revert the table only if no remaining replication or CDC process depends on the current setting. + + + + +```sql +-- Inspect current replica identity (d = default, f = full, i = index, n = nothing) +SELECT n.nspname AS schema_name, + c.relname AS table_name, + c.relreplident +FROM pg_class c +JOIN pg_namespace n ON n.oid = c.relnamespace +WHERE n.nspname = '' + AND c.relname = ''; + +-- Revert to the default only if DEFAULT is the desired setting after cleanup +ALTER TABLE ""."" REPLICA IDENTITY DEFAULT; +``` + + + + + +Only change replica identity for tables that CDC Sink changed and that no other replication or CDC tool relies on. +If a table still needs `REPLICA IDENTITY FULL` or a replica identity index for another consumer, leave it. + + + + + + +The replication slot name and publication name are fixed when the task is created and stored with the task. +You can supply your own names when creating the task; if you omit them, RavenDB auto-generates them (`rvn_cdc_s_` and `rvn_cdc_p_`). + +After that, task updates reuse the same slot and publication. +Adding or removing tables, renaming the task, or changing other task settings does not change those names. + +When updating a task, you may omit `SlotName` and `PublicationName`; RavenDB keeps the existing values. +However, attempting to set a different non-null `SlotName` or `PublicationName` results in an error. + +If you need to use a different slot or publication, delete the task and create a new one. +If the old slot and publication are no longer needed, clean them up manually - +see [Dropping a replication slot](#dropping-a-replication-slot) and [Dropping a publication](#dropping-a-publication) above. + + + + + +While the slot and publication names never change, the **publication** itself controls which PostgreSQL tables CDC Sink receives changes for. +The publication must include all root and embedded source tables configured in the task. + +* **Adding tables** + When the task starts after new tables were added to the CDC Sink configuration, CDC Sink checks whether those tables are in the publication. + If the PostgreSQL user configured in the CDC Sink connection string has permission, CDC Sink attempts to add the missing tables automatically: + + ```sql + ALTER PUBLICATION "" ADD TABLE "".""; + ``` + + If this fails, a database administrator must add the missing tables manually. + +* **Removing tables** + CDC Sink does not remove tables from the publication automatically. + If a table was removed from the CDC Sink task, a database administrator should remove it from the publication when it is no longer needed: + + ```sql + ALTER PUBLICATION "" DROP TABLE "".""; + ``` + + + +If the publication contains tables that are **not** configured in the CDC Sink task, +CDC Sink discards changes from those tables and raises a warning notification in RavenDB. + +For a normal table-list publication, +remove extra tables with `ALTER PUBLICATION "" DROP TABLE ...`. + +For a publication created `FOR ALL TABLES`, +you cannot narrow it with `ALTER PUBLICATION ... DROP TABLE` or `ALTER PUBLICATION ... SET TABLE`; + +PostgreSQL rejects table-list changes on `FOR ALL TABLES` publications. +Drop and recreate it as a table-list publication that lists only the tables configured in the CDC Sink task. + + + +See [Initial Setup](./initial-setup.mdx) for publication setup and permission guidance. + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/content/_initial-setup-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/content/_initial-setup-csharp.mdx new file mode 100644 index 0000000000..b1bbc09356 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/content/_initial-setup-csharp.mdx @@ -0,0 +1,241 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* When CDC Sink starts, it checks for the **replication slot** and **publication** required for logical replication, + and creates them if they are missing and the PostgreSQL user has the required permissions. + +* If the PostgreSQL user configured in the CDC Sink connection string cannot create these objects, + a database administrator must create them manually before the task can start. + +* In this article: + * [Automatic setup](#automatic-setup) + * [Manual setup](#manual-setup) + * [Specifying custom slot and publication names](#specifying-custom-slot-and-publication-names) + * [Slot and publication naming (auto-generated)](#slot-and-publication-naming-auto-generated) + * [Verifying setup](#verifying-setup) + * [One active consumer per slot](#one-active-consumer-per-slot) + + + + + +If the PostgreSQL user configured in the CDC Sink connection string has the permissions required for automatic setup, +CDC Sink can create and update the PostgreSQL objects it needs: + +1. Use the replication slot and publication names that were assigned when the task was created + (derived from the task id and stored with the task) +2. Check whether the replication slot and publication already exist +3. Create the publication if it does not exist +4. Create the replication slot if it does not exist +5. If configured tables are missing from the publication, + attempt `ALTER PUBLICATION ... ADD TABLE` automatically +6. Begin the initial load + +For the required PostgreSQL permissions, see [Permissions and Roles](../../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/permissions-and-roles.mdx). + +No manual database administration is needed in this case. + + + + + +If the PostgreSQL user configured in the CDC Sink connection string cannot create the replication slot +or create/update the publication, a database administrator must create the required PostgreSQL objects manually +before the task can run successfully. + +--- + +**Step 1: Determine the slot and publication names** + +The simplest approach is to specify the names explicitly in `CdcSinkPostgresSettings`, so both you and the database administrator know which names to use. +See [Specifying custom slot and publication names](#specifying-custom-slot-and-publication-names). + +If `SlotName` and `PublicationName` are not set explicitly, CDC Sink generates names using the `rvn_cdc_s_` and `rvn_cdc_p_` prefixes followed by the task id. +See [Slot and publication naming (auto-generated)](#slot-and-publication-naming-auto-generated). + +You can also find the names CDC Sink expects by creating the task and reading the startup error message, +which includes the expected slot or publication name. + +--- + +**Step 2: Create the publication** + +Create a publication that includes all the tables CDC Sink will replicate: + + + + +```sql +CREATE PUBLICATION rvn_cdc_p_ +FOR TABLE orders, order_lines, customers; +``` + + + + +The publication must include all tables from the task's root and embedded table configurations. + +--- + +**Step 3: Create the replication slot** + + + +```sql +SELECT pg_create_logical_replication_slot( + 'rvn_cdc_s_', + 'pgoutput' +); +``` + + + +--- + +**Step 4: Start the CDC Sink task** + +Once the slot and publication exist, CDC Sink will detect them on startup and proceed with the initial load. + + + +The slot and publication names are determined at task creation time and do not change +when you later modify the task. If you created these objects manually, the same names +will continue to be used regardless of subsequent configuration changes. + + + + + + + +Set `CdcSinkConfiguration.Postgres` when you want to choose the PostgreSQL replication slot and publication names explicitly: + + + +```csharp +var config = new CdcSinkConfiguration +{ + Name = "OrdersSync", + ConnectionStringName = "MyPostgresConnection", + Postgres = new CdcSinkPostgresSettings + { + SlotName = "orders_sync_slot", + PublicationName = "orders_sync_pub" + }, + Tables = [ ... ] +}; +``` + + + +Use names that follow PostgreSQL identifier rules: +start with a letter or underscore, contain only alphanumeric characters and underscores, and stay within PostgreSQL's 63-character limit. + +Custom names are useful when: + +* A database administrator pre-creates the slot and publication with human-readable names before starting the task +* You are migrating a task and intentionally reusing an existing replication slot +* You need predictable names across environments (dev/staging/prod) + +**Immutability**: +`SlotName` and `PublicationName` are fixed when the task is created. +If you need different names, delete the task and create a new one. + + + + + +If `CdcSinkConfiguration.Postgres` is omitted, or if either `SlotName` or `PublicationName` is left null, +CDC Sink fills in each missing name at task creation time using the task id: + +* **Slot name**: `rvn_cdc_s_{taskId}` +* **Publication name**: `rvn_cdc_p_{taskId}` + +Here `{taskId}` is the task id assigned when the CDC Sink task is created. + +**Example:** + +```plain +rvn_cdc_s_9481 +rvn_cdc_p_9481 +``` +
+ +The slot and publication use the same task id, making it easy to match them to a specific task. + + + +These names are stored with the task and are fixed after task creation. +Updating the task later does not regenerate or change them. + + + + + + + +To verify the slot and publication were created, query the actual names configured for the task +(whether custom or auto-generated): + + + +```sql +-- Verify the replication slot +SELECT slot_name, plugin, slot_type, active +FROM pg_replication_slots +WHERE slot_name = ''; + +-- Verify the publication and its tables +SELECT p.pubname, n.nspname AS schema_name, c.relname AS table_name +FROM pg_publication p +JOIN pg_publication_rel pr ON pr.prpubid = p.oid +JOIN pg_class c ON c.oid = pr.prrelid +JOIN pg_namespace n ON n.oid = c.relnamespace +WHERE p.pubname = '' +ORDER BY p.pubname, n.nspname, c.relname; +``` + + + + + + + +A PostgreSQL logical replication slot can be consumed by **only one connection at a time**. + +This matters when the same CDC Sink task configuration exists in more than one RavenDB database, +for example after **export/import** or **backup/restore**, which copy the task configuration verbatim, +including the immutable `SlotName` and `PublicationName`. + +Because both task copies point at the **same slot**, only one enabled task can consume it at a time. +The other task cannot stream from that slot, enters fallback (error-recovery) mode, and retries periodically. +While this conflict exists, changes from the PostgreSQL source may be delayed or interrupted in one of the RavenDB databases, +and the task reports an error such as: + +```plain +Process is in error recovery (fallback mode). Last error: . Next retry in: 464s. +``` +
+ +**Guidance:** + +* Keep the task **enabled in only one RavenDB database at a time** when the copied configuration uses the same slot. + * **Export/Import**: + Imported tasks are created **disabled** by default. Before enabling the imported task, + either disable the task in the origin database or create the imported task with its own slot and publication names. + * **Backup/Restore**: + A restored task keeps its original **enabled** state unless you set the `DisableOngoingTasks` restore option. + Restore with that option set, or disable the task in one of the databases, to avoid a slot conflict. + +* To run both task copies simultaneously, each copy must use its own replication slot. + If the copied task was already created with the original names, delete it and create a new task with distinct + `SlotName` and `PublicationName`, or have a database administrator pre-create the required PostgreSQL objects. + +See [Export / Import a Task](../../../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/export-import-task.mdx) and +[Backup / Restore a Task](../../../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/backup-restore-task.mdx). + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/content/_overview-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/content/_overview-csharp.mdx new file mode 100644 index 0000000000..6e9d7e811e --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/content/_overview-csharp.mdx @@ -0,0 +1,120 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* CDC Sink supports PostgreSQL as a source database by using [logical replication](https://www.postgresql.org/docs/current/logical-replication.html). + +* PostgreSQL streams row-level changes continuously through a logical replication slot. + The CDC Sink task consumes this stream and applies INSERTs, UPDATEs, and DELETEs to RavenDB documents as they are received. + +* In this article: + * [Connection string](#connection-string) + * [How it works](#how-it-works) + * [Prerequisites](#prerequisites) + * [PostgreSQL-specific settings](#postgresql-specific-settings) + * [Task configuration](#task-configuration) + + + + + +Create a [SqlConnectionString](../../../../../../server/ongoing-tasks/etl/sql.mdx#add-an-sql-connection-string) with `FactoryName` set to `"Npgsql"`. + + + +```csharp +store.Maintenance.Send(new PutConnectionStringOperation( + new SqlConnectionString + { + Name = "MyPostgresConnection", + FactoryName = "Npgsql", + ConnectionString = "Host=myserver;Port=5432;Database=mydb;Username=cdc_user;Password=..." + })); +``` + + +```csharp +await store.Maintenance.SendAsync(new PutConnectionStringOperation( + new SqlConnectionString + { + Name = "MyPostgresConnection", + FactoryName = "Npgsql", + ConnectionString = "Host=myserver;Port=5432;Database=mydb;Username=cdc_user;Password=..." + })); +``` + + + + + + + +CDC Sink uses PostgreSQL logical replication to receive a continuous stream of row-level changes without polling. + +Key PostgreSQL concepts: + +* **Replication slot** + A server-side object that tracks how far CDC Sink has read in the logical replication stream. + It also causes PostgreSQL to retain the required WAL segments until CDC Sink confirms receipt. +* **Publication** + Defines which tables are included in the replication stream. + CDC Sink creates or reuses a publication for the configured tables when the PostgreSQL user in the connection string has sufficient permissions. +* **WAL (Write-Ahead Log)** + The durable log PostgreSQL uses as the source for logical replication. + If a CDC Sink task is inactive or paused while its replication slot exists, WAL segments can accumulate. + + + + + +Before creating a CDC Sink task for PostgreSQL, verify the source database settings, permissions, and table requirements. +See [Prerequisites Checklist](../../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/prerequisites-checklist.mdx) +for the full list, including: + +* PostgreSQL version 10 or later +* WAL level set to `logical` +* Sufficient `max_replication_slots` and `max_wal_senders` +* A PostgreSQL user with `REPLICATION` privilege and `SELECT` on each table CDC Sink will read +* `REPLICA IDENTITY` configuration for embedded tables whose join columns are not part of the primary key + + + + + +Use `CdcSinkConfiguration.Postgres` when you want to specify the PostgreSQL replication slot and publication names explicitly: + + + +```csharp +Postgres = new CdcSinkPostgresSettings +{ + SlotName = "my_slot", + PublicationName = "my_publication" +} +``` + + + +If either name is omitted, the CDC Sink task generates it when the task is created: +`rvn_cdc_s_{taskId}` for the slot and `rvn_cdc_p_{taskId}` for the publication. +The names are stored with the task and do not change on later updates. + +To let the CDC Sink task create the PostgreSQL slot and publication itself, +the PostgreSQL user configured in the CDC Sink connection string must have sufficient permissions. + +See [Initial Setup](../../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup.mdx) for details. + + + + + +Most CDC Sink task configuration is shared across supported source database engines. +See [Configuration Reference](../../../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference.mdx) for the full API, +including table mappings, delete behavior, patches, and PostgreSQL-specific settings. + +For PostgreSQL, `SourceTableSchema` defaults to `"public"` when omitted. + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup.mdx new file mode 100644 index 0000000000..cd4eb7b82e --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup.mdx @@ -0,0 +1,35 @@ +--- +title: "CDC Sink for PostgreSQL: Initial Setup" +sidebar_label: Initial Setup +description: "How CDC Sink creates and verifies the replication slot and publication on startup, with options for automatic or manual setup and custom naming." +sidebar_position: 5 +supported_languages: ["csharp"] +see_also: + - title: "Prerequisites Checklist" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/prerequisites-checklist" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Permissions and Roles" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/permissions-and-roles" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "REPLICA IDENTITY" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Cleanup and Maintenance" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/cleanup-and-maintenance" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import InitialSetupCsharp from './content/_initial-setup-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/monitoring-postgres.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/monitoring-postgres.mdx new file mode 100644 index 0000000000..df31a5804b --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/monitoring-postgres.mdx @@ -0,0 +1,124 @@ +--- +title: "CDC Sink for PostgreSQL: Monitoring PostgreSQL" +sidebar_label: "Monitoring PostgreSQL" +description: "PostgreSQL-side monitoring for CDC Sink, including replication slot health, replication lag, and WAL disk usage queries." +sidebar_position: 9 +see_also: + - title: "Cleanup and Maintenance" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/cleanup-and-maintenance" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Monitoring" + link: "server/ongoing-tasks/cdc-sink/monitoring" + source: "docs" + path: "Server > CDC Sink" + - title: "Failover and Consistency" + link: "server/ongoing-tasks/cdc-sink/failover-and-consistency" + source: "docs" + path: "Server > CDC Sink" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* This page covers PostgreSQL-side monitoring for CDC Sink tasks: + replication slot health, replication lag, and WAL disk usage. + +* For RavenDB-side monitoring, see [Monitoring](../../../../../server/ongoing-tasks/cdc-sink/monitoring.mdx). + +* In this article: + * [Replication slot health](#replication-slot-health) + * [Replication lag](#replication-lag) + * [WAL disk usage](#wal-disk-usage) + + + + + +List CDC Sink replication slots that use the auto-generated slot-name prefix and check whether they are active: + + + +```sql +SELECT slot_name, active, confirmed_flush_lsn +FROM pg_replication_slots +WHERE slot_name LIKE 'rvn_cdc_s_%'; +``` + + + +The `rvn_cdc_s_` prefix matches only auto-generated slot names. +If the task uses a custom `SlotName`, replace the filter with that name or remove the filter and cross-check the results against your CDC Sink task configurations. + +| Column | Meaning | +|-----------------------|----------------------------------------------------------------------------------------------------------| +| `active` | `true` if CDC Sink is connected and consuming;
`false` if no process is currently connected to the slot. | +| `confirmed_flush_lsn` | The LSN up to which changes have been confirmed as consumed. | + +A slot with `active = false` means CDC Sink is not currently connected. +This is expected during failover or when the task is disabled or stopped. +If the slot remains inactive for an extended period, investigate the task state in RavenDB Studio. + + + + + +Replication lag measures how far the slot is behind the current WAL position: + + + +```sql +SELECT slot_name, + pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) AS lag_bytes +FROM pg_replication_slots +WHERE slot_name LIKE 'rvn_cdc_s_%'; +``` + + + +The `rvn_cdc_s_` prefix matches only auto-generated slot names. +If the task uses a custom `SlotName`, adjust or remove the filter. + +A consistently growing `lag_bytes` means CDC Sink is not keeping up with the rate +of changes in the source database. Consider: + +* Checking CDC Sink performance statistics in RavenDB Studio for slow processing stages or scripts +* Increasing `CdcSink.MaxBatchSize` if the task is limited by batch size +* Reducing load on the source database + + + + + +PostgreSQL retains WAL that is still required by replication slots. +An inactive or slow CDC Sink slot can cause WAL files to accumulate on disk. + +Check approximate WAL retained by CDC Sink slots that use the auto-generated slot-name prefix: + + + +```sql +SELECT slot_name, active, + pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS retained_wal_bytes +FROM pg_replication_slots +WHERE slot_name LIKE 'rvn_cdc_s_%' +ORDER BY retained_wal_bytes DESC NULLS LAST; +``` + + + +The `rvn_cdc_s_` prefix matches only auto-generated slot names. +If the task uses a custom `SlotName`, adjust or remove the filter. + +A high `retained_wal_bytes` value means PostgreSQL is retaining WAL for that slot. +* If the slot is active, CDC Sink is connected but may be processing changes slowly. +* If it is inactive, no process is currently consuming from it. + +Before dropping a slot, verify that no CDC Sink task still needs it. +For cleanup guidance, see [Cleanup and Maintenance](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/cleanup-and-maintenance.mdx). + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/overview.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/overview.mdx new file mode 100644 index 0000000000..7cbcc63874 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/overview.mdx @@ -0,0 +1,18 @@ +--- +title: "CDC Sink for PostgreSQL: Overview" +sidebar_label: "Overview" +description: "CDC Sink support for PostgreSQL - using logical replication to stream changes into RavenDB in real time." +sidebar_position: 1 +supported_languages: ["csharp"] +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import OverviewCsharp from './content/_overview-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/permissions-and-roles.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/permissions-and-roles.mdx new file mode 100644 index 0000000000..c241c5f97b --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/permissions-and-roles.mdx @@ -0,0 +1,157 @@ +--- +title: "CDC Sink for PostgreSQL: Permissions and Roles" +sidebar_label: Permissions and Roles +description: "PostgreSQL permissions required by the CDC Sink user, including minimum privileges, automatic setup permissions, and dedicated user creation examples." +sidebar_position: 4 +see_also: + - title: "Prerequisites Checklist" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/prerequisites-checklist" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Initial Setup" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Cleanup and Maintenance" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/cleanup-and-maintenance" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* This page documents the PostgreSQL permissions required by the role configured in the CDC Sink connection string, and how to grant them. + +* In this article: + * [Minimum permissions](#minimum-permissions) + * [Permissions for automatic setup](#permissions-for-automatic-setup) + * [Creating a dedicated CDC user](#creating-a-dedicated-cdc-user) + + + + + +The **CDC Sink user** is the PostgreSQL role in the connection string that the CDC Sink task uses to connect to the source database. +Throughout this page, `cdc_user` is used as the example username for this role. + +At a minimum, the CDC Sink user needs: + +* **`REPLICATION`** role attribute - allows the user to use logical replication and create a logical replication slot. +* **`SELECT`** on each table mapped by the CDC Sink task - required for the initial load phase. + +Example: + + + +```sql +-- Grant replication privilege +ALTER USER cdc_user REPLICATION; + +-- Grant SELECT on each table +GRANT SELECT ON TABLE orders TO cdc_user; +GRANT SELECT ON TABLE order_lines TO cdc_user; +GRANT SELECT ON TABLE customers TO cdc_user; +``` + + + +With only these permissions, CDC Sink can create the replication slot, +but a database administrator must create or update the publication before the task starts +unless the user also has the publication permissions described below. + +See [Initial Setup](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup.mdx). + + + + + +To let CDC Sink create the replication slot and create or update the publication automatically, +the PostgreSQL user needs the minimum permissions above plus publication-management permissions. + +**Create replication slots:** + +The `REPLICATION` role attribute listed under [Minimum permissions](#minimum-permissions) is enough for CDC Sink to create the logical replication slot: + + + +```sql +ALTER USER cdc_user REPLICATION; +``` + + + +**Create and update publications:** + +Creating a publication requires `CREATE` on the database. +Adding tables to the publication requires ownership of those tables (`SUPERUSER` satisfies both requirements): + + + +```sql +GRANT CREATE ON DATABASE mydb TO cdc_user; + +-- The user must also own the tables added to the publication, +-- or a database administrator must create/update the publication manually. +``` + + + + + +Granting `SUPERUSER` gives the user unrestricted access to the database server. +For production environments, prefer granting only the specific privileges required, or have a database administrator manage the publication manually. + + + + + +For added security in production, consider having your database administrator create or update the publication manually, +rather than granting CDC Sink ownership of application tables. +The CDC Sink user still needs `REPLICATION` and `SELECT` on the relevant tables. +See [Initial Setup](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup.mdx). + + + + + + + +It is recommended to use a dedicated PostgreSQL user for CDC Sink rather than an application or admin user. + +Example setup: + + + +```sql +-- Create the user with replication privilege +CREATE USER cdc_user WITH PASSWORD 'secure_password' REPLICATION; + +-- Grant SELECT on every table mapped by the CDC Sink task +GRANT SELECT ON TABLE orders TO cdc_user; +GRANT SELECT ON TABLE order_lines TO cdc_user; + +-- If CDC Sink should create the publication automatically: +GRANT CREATE ON DATABASE mydb TO cdc_user; + +-- The user must also own every table added to the publication, +-- or a database administrator must create/update the publication manually. +ALTER TABLE orders OWNER TO cdc_user; +ALTER TABLE order_lines OWNER TO cdc_user; +``` + + + + + +Transferring table ownership with `ALTER TABLE ... OWNER TO` can affect the owning application, +including who can alter or drop the table and who controls table-level privileges. +For production environments, prefer having a database administrator create or update the publication manually unless table ownership transfer is acceptable. + + + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/prerequisites-checklist.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/prerequisites-checklist.mdx new file mode 100644 index 0000000000..7600619662 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/prerequisites-checklist.mdx @@ -0,0 +1,153 @@ +--- +title: "CDC Sink for PostgreSQL: Prerequisites Checklist" +sidebar_label: "Prerequisites Checklist" +description: "Verify all requirements before creating a CDC Sink task for PostgreSQL, including WAL configuration, user permissions, table requirements, and network access." +sidebar_position: 2 +see_also: + - title: "WAL Configuration" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/wal-configuration" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Permissions and Roles" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/permissions-and-roles" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Initial Setup" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "REPLICA IDENTITY" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* Before creating a CDC Sink task for PostgreSQL, review the prerequisites on this page to confirm that + the source database is ready for logical replication, the PostgreSQL user has the required permissions, + the mapped tables meet CDC Sink requirements, and RavenDB can connect to the database. + +* In this article: + * [Source database requirements](#source-database-requirements) + * [User permissions](#user-permissions) + * [Table requirements](#table-requirements) + * [Network access](#network-access) + + + + + +* **PostgreSQL version**: + 10 or later. + +* **WAL level**: + `wal_level` must be set to `logical`. + +* **Replication slots**: + `max_replication_slots` must allow at least one slot for each CDC Sink task. + +* **WAL senders**: + `max_wal_senders` must allow at least one active replication connection for each running CDC Sink task. + +Verify the current settings with: + + + +```sql +SHOW wal_level; +SHOW max_replication_slots; +SHOW max_wal_senders; +``` + + + +For instructions on changing `wal_level`, `max_replication_slots`, or `max_wal_senders`, +see [WAL Configuration](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/wal-configuration.mdx). + + + + + +The PostgreSQL user configured in the CDC Sink connection string must be able to use logical replication +and read every table mapped by the CDC Sink task. + +**Minimum required permissions:** + + + +```sql +-- Required for logical replication +ALTER USER cdc_user REPLICATION; + +-- Required for the initial load and for reading mapped tables +GRANT SELECT ON TABLE orders TO cdc_user; +GRANT SELECT ON TABLE order_lines TO cdc_user; +``` + + + +**Permissions for automatic setup:** + +To let CDC Sink create the replication slot and create or update the publication automatically, +the PostgreSQL user also needs: + +* **Create replication slots**: + the `REPLICATION` role attribute. + +* **Create publications**: + `CREATE` on the database, plus ownership of the tables being published. + `SUPERUSER` satisfies both publication requirements. + +See [Permissions and Roles](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/permissions-and-roles.mdx) for the exact grants. + +If the PostgreSQL user configured in the CDC Sink connection string does not have these permissions, +a database administrator must set up the replication slot and publication manually before the task starts. +See [Initial Setup](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup.mdx). + + + + + +* **Configured key columns**: + Each root table and embedded table must define `PrimaryKeyColumns`. + These columns must exist in the source table and must also be included in the table's `Columns` mappings. + +* **Embedded join columns**: + Each embedded table must define `JoinColumns` that match the direct parent table's `PrimaryKeyColumns` + in count and order. These columns must exist in the embedded source table. + +* **REPLICA IDENTITY for embedded deletes**: + For embedded tables where the join columns are not part of the primary key, + the table must have `REPLICA IDENTITY FULL` or a replica identity index that covers both the join columns and the embedded table's `PrimaryKeyColumns`. + + Without this, DELETE events will not include enough old row values for CDC Sink to route the delete and identify the embedded item to remove. + If deletes for that embedded table are ignored with `OnDelete.IgnoreDeletes = true`, + no `REPLICA IDENTITY` change is required for delete routing. + Learn more in [REPLICA IDENTITY](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx). + +* **Mapped source columns**: + Every source column referenced by `Columns`, `PrimaryKeyColumns`, `JoinColumns`, or linked-table `JoinColumns` + must exist in the SQL table. + + + + + +* The RavenDB node that runs the CDC Sink task must be able to connect to the PostgreSQL host and port in the connection string (default port: `5432`). + +* In a RavenDB cluster, every node that can become responsible for the task should have the same access, + so the task can resume after failover. + +* PostgreSQL and the network path must allow connections from those RavenDB node addresses. + Check PostgreSQL host-based authentication, firewall rules, routing, proxies, load balancers, and security groups as applicable. + +* The replication connection is long-lived. + Ensure that firewalls, proxies, and load balancers do not terminate idle or long-lived connections. + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity-manual-setup.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity-manual-setup.mdx new file mode 100644 index 0000000000..f6e7dca15f --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity-manual-setup.mdx @@ -0,0 +1,125 @@ +--- +title: "CDC Sink for PostgreSQL: Manual REPLICA IDENTITY Setup" +sidebar_label: Manual REPLICA IDENTITY Setup +description: "Step-by-step instructions for a database administrator to configure REPLICA IDENTITY manually when CDC Sink lacks permission to alter tables." +sidebar_position: 7 +see_also: + - title: "REPLICA IDENTITY" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Initial Setup" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* If the PostgreSQL user configured in the CDC Sink connection string cannot alter an affected table, + a database administrator must configure `REPLICA IDENTITY` manually before the task starts. + +* This article shows the SQL commands for manual setup. For background on what `REPLICA IDENTITY` is, + when it is required, and when CDC Sink can configure it automatically, + see [REPLICA IDENTITY](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx). + +* In this article: + * [When manual setup is required](#when-manual-setup-is-required) + * [Setting REPLICA IDENTITY FULL](#setting-replica-identity-full) + * [Using an index instead of FULL](#using-an-index-instead-of-full) + * [Verifying the configuration](#verifying-the-configuration) + + + + + +If the PostgreSQL user configured in the CDC Sink connection string owns theaffected embedded source table, or has `SUPERUSER`, +CDC Sink can automatically set `REPLICA IDENTITY FULL` when that table's current setting is insufficient. + +If that PostgreSQL user cannot alter the table, the task fails to start with a permissions error. +A database administrator must configure `REPLICA IDENTITY` manually before the task can run. + +See [Automatic vs manual configuration](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx#automatic-vs-manual-configuration) +for details on when this applies. + + + + + +The simplest approach is to set `REPLICA IDENTITY FULL` on each affected embedded source table: + + + +```sql +ALTER TABLE order_lines REPLICA IDENTITY FULL; +ALTER TABLE line_attributes REPLICA IDENTITY FULL; +``` + + + +With `REPLICA IDENTITY FULL`, PostgreSQL can include all old column values in `DELETE` and `UPDATE` events. +This is the most compatible option, but it increases WAL volume for tables with many or large columns. + + + + + +If WAL size is a concern, you can use a specific unique index instead of `FULL`. + +The designated replica identity index must cover both the join columns and `PrimaryKeyColumns` of the embedded table, +because CDC Sink needs those old values to find the parent document and the matching embedded item. + +--- + +**Step 1: Create a unique index covering the required columns** + + + +```sql +-- For order_lines: join column is order_id, PK is line_id +CREATE UNIQUE INDEX order_lines_replica_idx + ON order_lines (order_id, line_id); +``` + + + +**Step 2: Set REPLICA IDENTITY to use this index** + + + +```sql +ALTER TABLE order_lines + REPLICA IDENTITY USING INDEX order_lines_replica_idx; +``` + + + +This instructs PostgreSQL to include old row values for the indexed columns in `DELETE` and `UPDATE` events, +rather than all columns. + + + +* The index must be `UNIQUE` and `NOT DEFERRABLE`. + It cannot use expressions or partial predicates. + +* Every indexed column must be marked `NOT NULL`; + if a required join column is nullable, make it `NOT NULL` first or use `FULL`. + + + + + + + +* After applying the changes, confirm that each affected table shows the expected `REPLICA IDENTITY` mode using the query in + [Checking current setting](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx#checking-current-setting). + +* A table configured with `FULL` is ready for CDC Sink embedded-table delete processing. + A table configured with `INDEX` is ready when the designated replica identity index covers the required join columns and `PrimaryKeyColumns`. + + diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx new file mode 100644 index 0000000000..13adab7f91 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx @@ -0,0 +1,181 @@ +--- +title: "CDC Sink for PostgreSQL: REPLICA IDENTITY" +sidebar_label: REPLICA IDENTITY +description: "How PostgreSQL REPLICA IDENTITY affects CDC Sink's ability to process DELETE events for embedded tables, and when it must be configured." +sidebar_position: 6 +see_also: + - title: "Manual REPLICA IDENTITY Setup" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity-manual-setup" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Initial Setup" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* **REPLICA IDENTITY** determines which old column values PostgreSQL can include in logical replication messages for `DELETE` and `UPDATE` operations. + For CDC Sink, it mainly matters for `DELETE` events from embedded tables. + +* To process an embedded-table `DELETE`, CDC Sink needs the old values of the embedded row's join columns and primary key columns. + The join columns locate the parent document; the primary key columns locate the embedded item to remove. + +* This article explains when PostgreSQL's default setting is sufficient, when the table needs a different `REPLICA IDENTITY`, + and whether CDC Sink can configure it automatically. + +* In this article: + * [Why REPLICA IDENTITY matters](#why-replica-identity-matters) + * [REPLICA IDENTITY options](#replica-identity-options) + * [When is REPLICA IDENTITY required?](#when-is-replica-identity-required) + * [Checking current setting](#checking-current-setting) + * [Automatic vs manual configuration](#automatic-vs-manual-configuration) + + + + + +When CDC Sink receives a DELETE event for an embedded table, it needs to: + +1. Find the parent document using the old row's join column values +2. Find the matching item within the embedded array or map using the embedded table's primary key values +3. Remove it + +If the DELETE event does not include the old values for both sets of columns, CDC Sink cannot perform steps 1 or 2. + +By default, PostgreSQL includes only the primary key columns in DELETE events. +If the embedded table's join columns are part of the primary key, this is sufficient. +If they are not, which is common when a table has a surrogate primary key, +the DELETE event will be missing the columns needed to route it. + +UPDATE events carry the new row, so ordinary embedded updates can be routed under `DEFAULT`. +Old row values matter when an UPDATE changes the embedded row's join columns, +because CDC Sink must remove the item from the old parent document and add it to the new one. + + + + + +PostgreSQL supports four REPLICA IDENTITY modes: + +| Mode | Old row values available for DELETE and UPDATE events | Notes | +| --------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| `DEFAULT` | Primary key columns only | The default for every table.
On a table **without** a primary key, no old row values are sent | +| `FULL` | All columns | Works for any table structure; increases WAL size | +| `INDEX` | Columns covered by the designated replica identity index | More targeted than `FULL`, but the selected index must cover all columns CDC Sink needs | +| `NOTHING` | No old row values | Insufficient for CDC Sink embedded table deletes | + +For CDC Sink, `FULL` is the simplest and most compatible choice. +`INDEX` works if the designated replica identity index covers both the embedded +table's join columns and its `PrimaryKeyColumns`. + + + + + +| Scenario | REPLICA IDENTITY needed? | +| ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Root table deletes where the document ID is built from the table's PostgreSQL primary key | No - `DEFAULT` sends the primary key columns | +| Configured table has no PostgreSQL primary key and DELETEs must be processed | **Yes** - under `DEFAULT`, PostgreSQL sends no old row values; use `FULL` or a suitable replica identity index | +| Embedded table deletes where all required join columns and `PrimaryKeyColumns` are covered by the PostgreSQL primary key | No - `DEFAULT` is sufficient | +| Embedded table deletes where any required join column is not covered by the PostgreSQL primary key | **Yes** - use `FULL` or a replica identity index that covers the join columns and `PrimaryKeyColumns` | +| Embedded table where `OnDelete.IgnoreDeletes = true` is set without an `OnDelete.Patch` | No - the DELETE event is discarded before routing | +| Embedded table with an `OnDelete.Patch` | Same as a processed delete - the DELETE event must still be routed so the patch can run | +| Deep-nested embedded tables | Apply the same rule to each embedded source table: the DELETE event must include the columns needed to find the root document, navigate the embedded path, and identify the item to remove | + +--- + +**Example where REPLICA IDENTITY is needed:** + + + +```sql +-- order_lines has a surrogate PK (line_id), with order_id as foreign key +-- order_id is NOT part of the primary key +CREATE TABLE order_lines ( + line_id SERIAL PRIMARY KEY, -- PK + order_id INT NOT NULL, -- FK to orders (not part of PK) + product TEXT, + qty INT +); + +-- Without REPLICA IDENTITY FULL, a DELETE event only contains line_id. +-- CDC Sink cannot determine which parent document (order_id) to update. +ALTER TABLE order_lines REPLICA IDENTITY FULL; +``` + + + +**Example where REPLICA IDENTITY is NOT needed:** + + + +```sql +-- If order_id is part of a composite PK, DEFAULT is sufficient +CREATE TABLE order_lines ( + order_id INT NOT NULL, + line_num INT NOT NULL, + product TEXT, + qty INT, + PRIMARY KEY (order_id, line_num) +); +``` + + + + + + + + + + +```sql +SELECT + n.nspname AS schema_name, + c.relname AS table_name, + c.relreplident, + CASE c.relreplident + WHEN 'd' THEN 'DEFAULT' + WHEN 'f' THEN 'FULL' + WHEN 'i' THEN 'INDEX' + WHEN 'n' THEN 'NOTHING' + END AS replica_identity +FROM pg_class c +JOIN pg_namespace n ON n.oid = c.relnamespace +WHERE c.relkind IN ('r', 'p') + AND n.nspname NOT LIKE 'pg_%' + AND n.nspname <> 'information_schema' +ORDER BY n.nspname, c.relname; +``` + + + + +The `relreplident` column: + +| Value | Meaning | +| ----- | ----------------------------------------- | +| `d` | DEFAULT (primary key only) | +| `f` | FULL (all columns) | +| `i` | INDEX (designated replica identity index) | +| `n` | NOTHING | + + + + + +If the PostgreSQL user configured in the CDC Sink connection string owns the affected embedded source table, or has `SUPERUSER`, +CDC Sink can automatically set `REPLICA IDENTITY FULL` when that table's current replica identity is insufficient for embedded DELETE routing. + +If that PostgreSQL user cannot alter the table, the task fails to start with a permissions error. +A database administrator must configure `REPLICA IDENTITY` before starting the task. +See [Manual Setup](../../../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity-manual-setup.mdx). + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/wal-configuration.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/wal-configuration.mdx new file mode 100644 index 0000000000..228201fdb8 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/postgres/wal-configuration.mdx @@ -0,0 +1,134 @@ +--- +title: "CDC Sink for PostgreSQL: WAL Configuration" +sidebar_label: WAL Configuration +description: "How to verify and configure the PostgreSQL Write-Ahead Log level to enable logical replication required by CDC Sink." +sidebar_position: 3 +see_also: + - title: "Prerequisites Checklist" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/prerequisites-checklist" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Permissions and Roles" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/permissions-and-roles" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "Cleanup and Maintenance" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/cleanup-and-maintenance" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* CDC Sink uses PostgreSQL **logical replication**, which requires `wal_level` to be set to `logical`. + +* This page explains how to check the current `wal_level`, enable logical replication if needed, + and review related replication settings. + +* In this article: + * [Check current WAL level](#check-current-wal-level) + * [Enable logical replication](#enable-logical-replication) + * [Other required settings](#other-required-settings) + + + + + +Connect to your PostgreSQL instance and run: + + + +```sql +SHOW wal_level; +``` + + + +If the result is `logical`, no change to `wal_level` is needed. + +If the result is `replica` or `minimal`, +logical replication is not enabled and `wal_level` must be changed before CDC Sink can read changes from PostgreSQL. + + + + + +For self-managed PostgreSQL, edit `postgresql.conf` and set: + +```plain +wal_level = logical +``` +
+ + +Changing `wal_level` requires a full server restart, not just a configuration reload. +Plan for a brief maintenance window. + + +After restarting, verify the change took effect: + + + +```sql +SHOW wal_level; +-- Should return: logical +``` + + + + + +#### Managed PostgreSQL + +On managed PostgreSQL services, you usually cannot edit `postgresql.conf` directly. +Change the setting through the provider's configuration mechanism instead: + +* **Amazon RDS for PostgreSQL** + Set `rds.logical_replication = 1` in the DB parameter group, then reboot the DB instance. + +* **Amazon Aurora PostgreSQL** + Set `rds.logical_replication = 1` in the DB cluster parameter group, then reboot the writer instance. + +* **Google Cloud SQL for PostgreSQL** + Set the `cloudsql.logical_decoding` flag to `on`, then restart the primary instance. + +* **Azure Database for PostgreSQL Flexible Server** + Set the `wal_level` server parameter to `logical`, save the change, and restart the server. + + + + + + + +CDC Sink uses one PostgreSQL replication slot per task. +Make sure the following settings are high enough for your CDC Sink tasks and any other replication consumers: + +```plain +max_replication_slots = 10 # at least 1 per CDC Sink task +max_wal_senders = 10 # at least 1 per running CDC Sink task +``` +
+ +The defaults in a standard PostgreSQL installation are typically sufficient for a small number of tasks, +but you may need to increase them if you have many CDC Sink tasks or other replication consumers. + +Check current values: + + + +```sql +SHOW max_replication_slots; +SHOW max_wal_senders; +``` + + + +These settings require a server restart if changed. + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/sql-server/_category_.json b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/sql-server/_category_.json new file mode 100644 index 0000000000..4f139ac95c --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/sql-server/_category_.json @@ -0,0 +1,4 @@ +{ + "position": 1, + "label": "SQL Server" +} diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/sql-server/content/_overview-csharp.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/sql-server/content/_overview-csharp.mdx new file mode 100644 index 0000000000..90d5acb681 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/sql-server/content/_overview-csharp.mdx @@ -0,0 +1,126 @@ +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* CDC Sink supports **SQL Server** as a source database using native + [SQL Server Change Data Capture](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server). + +* SQL Server CDC captures row-level changes from the transaction log and stores them in CDC change tables. + The CDC Sink task polls SQL Server CDC for new changes and applies INSERTs, UPDATEs, and DELETEs + to RavenDB documents. + +* In this article: + * [Connection string](#connection-string) + * [Prerequisites](#prerequisites) + * [Polling behavior](#polling-behavior) + * [Task configuration](#task-configuration) + + + + + +Use a [SqlConnectionString](../../../../../../server/ongoing-tasks/etl/sql.mdx#add-an-sql-connection-string) with `FactoryName` set to `"Microsoft.Data.SqlClient"`: + + + +```csharp +store.Maintenance.Send(new PutConnectionStringOperation( + new SqlConnectionString + { + Name = "SqlServerConnection", + FactoryName = "Microsoft.Data.SqlClient", + ConnectionString = + "Server=myserver;Database=mydb;" + + "User Id=ravendb_cdc;Password=...;" + + "TrustServerCertificate=True;" + })); +``` + + +```csharp +await store.Maintenance.SendAsync(new PutConnectionStringOperation( + new SqlConnectionString + { + Name = "SqlServerConnection", + FactoryName = "Microsoft.Data.SqlClient", + ConnectionString = + "Server=myserver;Database=mydb;" + + "User Id=ravendb_cdc;Password=...;" + + "TrustServerCertificate=True;" + })); +``` + + + + +`Microsoft.Data.SqlClient` defaults to `Encrypt=True`. +Unless SQL Server presents a certificate trusted by the RavenDB server, add `TrustServerCertificate=True` +(as above) or `Encrypt=False` to the connection string. Otherwise, the CDC Sink task cannot connect. + + + + + + +Before CDC Sink can capture changes, +SQL Server CDC must be enabled on the source database and on every source table configured in the task. + +If the SQL Server user in the CDC Sink connection string has permission to enable CDC, +RavenDB enables it automatically when the task starts. Otherwise, ask a database administrator to run: + + + +```sql +-- Enable CDC on the database +EXEC sys.sp_cdc_enable_db; + +-- Enable CDC on a table +EXEC sys.sp_cdc_enable_table + @source_schema = N'dbo', + @source_name = N'orders', + @role_name = NULL; +``` + + + +The SQL Server Agent service must be running for CDC change capture to work. +The Agent runs the capture jobs that populate the change tables. + +On-premises, start the SQL Server Agent service; in Docker containers, start with `-e 'MSSQL_AGENT_ENABLED=true'`. +If the Agent is stopped, the task can still start and raise a warning, but no CDC changes are delivered until the Agent is running. + + + +#### Azure SQL Database + +Azure SQL Database has no classic SQL Server Agent. +Its CDC capture runs on a managed scheduler, so this requirement does not apply there. + + + + + + + +Unlike PostgreSQL and MySQL, SQL Server CDC Sink uses polling rather than a streaming replication connection. +The `CdcSink.SqlServer.PollIntervalInSec` server configuration key controls how frequently the task polls for new change rows. + +Lower values reduce latency, but increase load on the source database. +See [Server Configuration](../../../../../../server/ongoing-tasks/cdc-sink/server-configuration.mdx). + + + + + +SQL Server CDC Sink uses the same `CdcSinkConfiguration` table, mapping, patching, and delete-handling settings as the other source engines. +SQL Server does not use the PostgreSQL-specific `Postgres` settings. + +See [Configuration Reference](../../../../../../server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference.mdx) +for full details. + +When `SourceTableSchema` is omitted, SQL Server uses `"dbo"` as the default schema. + + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/source-database-setup/sql-server/overview.mdx b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/sql-server/overview.mdx new file mode 100644 index 0000000000..3edc42fec9 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-database-setup/sql-server/overview.mdx @@ -0,0 +1,31 @@ +--- +title: "CDC Sink for SQL Server: Overview" +sidebar_label: Overview +description: "CDC Sink support for SQL Server - using native SQL Server Change Data Capture to stream changes into RavenDB." +sidebar_position: 0 +supported_languages: ["csharp"] +see_also: + - title: "Overview" + link: "server/ongoing-tasks/cdc-sink/overview" + source: "docs" + path: "Server > CDC Sink" + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" + - title: "PostgreSQL Prerequisites Checklist" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/prerequisites-checklist" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; + +import OverviewCsharp from './content/_overview-csharp.mdx'; + + + + + + diff --git a/docs/server/ongoing-tasks/cdc-sink/source-schema-changes.mdx b/docs/server/ongoing-tasks/cdc-sink/source-schema-changes.mdx new file mode 100644 index 0000000000..892c2505cd --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/source-schema-changes.mdx @@ -0,0 +1,275 @@ +--- +title: "CDC Sink: Source Schema Changes" +sidebar_label: Source Schema Changes +description: "How to safely modify your source database schema while a CDC Sink task is running - behavior per database engine, recovery, and the SQL Server capture instance procedure." +sidebar_position: 8 +see_also: + - title: "Configuration Reference" + link: "server/ongoing-tasks/cdc-sink/manage-cdc-sink-tasks/configuration-reference" + source: "docs" + path: "Server > CDC Sink" + - title: "Column Mapping" + link: "server/ongoing-tasks/cdc-sink/document-modeling/column-mapping" + source: "docs" + path: "Server > CDC Sink" + - title: "How It Works: Updating the Task Configuration" + link: "server/ongoing-tasks/cdc-sink/how-it-works#updating-the-task-configuration" + source: "docs" + path: "Server > CDC Sink" + - title: "Troubleshooting" + link: "server/ongoing-tasks/cdc-sink/troubleshooting" + source: "docs" + path: "Server > CDC Sink" + - title: "SQL Server Overview" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/sql-server/overview" + source: "docs" + path: "Server > CDC Sink > SQL Server" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* This page explains how CDC Sink handles source-table schema changes while the task is running, + including added, dropped, renamed, and type-changed columns. + +* Behavior differs by database engine. + PostgreSQL is the most resilient, MySQL can detect many changes automatically but has positional and type-code limitations, + and SQL Server requires an explicit capture-instance procedure for every schema change. + +* In this article: + * [Quick reference](#quick-reference) + * [PostgreSQL](#postgresql) + * [MySQL](#mysql) + * [SQL Server](#sql-server) + * [Recovery from errors](#recovery-from-errors) + * [General recommendation](#general-recommendation) + + + + + +| Change | MySQL | PostgreSQL | SQL Server | +| -------------------- | ------------------------------------------------------------------------------ | ------------------------------ | ------------------------------------------------------------------------- | +| Add unmapped column | Transparent if after mapped columns; otherwise restarts and re-resolves schema | Transparent | Requires [procedure](#sql-server-schema-change-procedure) | +| Add mapped column | Apply change + update config | Apply change + update config | Requires [procedure](#sql-server-schema-change-procedure) + config update | +| Drop unmapped column | Transparent if after mapped columns; otherwise restarts and re-resolves schema | Transparent | Requires [procedure](#sql-server-schema-change-procedure) | +| Drop mapped column | Enters fallback; update config | Enters fallback; update config | Requires [procedure](#sql-server-schema-change-procedure) + config update | +| Change column type | Detects most changes; restarts and usually recovers | Transparent auto-rebuild | Requires [procedure](#sql-server-schema-change-procedure) | +| Rename mapped column | Enters fallback; update config | Enters fallback; update config | Requires [procedure](#sql-server-schema-change-procedure) + config update | + + + + + +PostgreSQL logical replication sends `RelationMessage` events inline in the replication stream +whenever the table schema changes. CDC Sink detects these automatically and rebuilds its source-column index mapping on the fly. +This makes PostgreSQL the **most resilient** to schema changes. + +* **Adding or dropping unmapped columns** - transparent. + +* **Changing a column type** - usually transparent. + A new `RelationMessage` updates the internal type and source-column mapping used by the sink. + +* **Dropping or renaming a mapped column**: + CDC Sink enters fallback because the configured column name is no longer found in the new schema. + Update the task configuration to remove the dropped mapping or point it to the new column name. + +### PostgreSQL examples + + + +```sql +-- Transparent: adding an unmapped column +ALTER TABLE orders ADD COLUMN internal_notes TEXT; +-- CDC Sink auto-rebuilds and continues + +-- Transparent: changing a column type +ALTER TABLE orders ALTER COLUMN status TYPE VARCHAR(100); +-- CDC Sink auto-rebuilds and continues + +-- Requires config update: renaming a mapped column +ALTER TABLE orders RENAME COLUMN name TO full_name; +-- CDC Sink enters fallback. Update the task configuration to map 'full_name' instead of 'name'. +``` + + + + + + + +MySQL CDC uses binlog replication. +Row events are associated with a `TableMapEvent` that contains column type codes, but not column names. + +CDC Sink compares the type-code prefix for the columns it reads against the schema resolved from `INFORMATION_SCHEMA`. +If the prefix no longer matches, the process detects a schema change, restarts, and re-resolves the current schema. + +* **Adding or dropping unmapped columns** - transparent *when the column is after all mapped columns*. + In that case, the mapped column positions do not change and the sink continues without interruption. + If the unmapped column is added or dropped *before* a mapped column, the positional shift may be detected as a schema change. + However, if the shifted columns have the same binlog type codes, the change may pass through silently. + +* **Changing a column type** - CDC Sink detects many type changes via binlog column type codes, restarts, re-resolves from `INFORMATION_SCHEMA`, and continues. + Some changes are **not** detected - for example `TEXT` to `BLOB`, `VARCHAR` to `VARBINARY`, + or a width change such as `VARCHAR(10)` to `VARCHAR(100)`. + These map to the same underlying type code. + +* **Dropping or renaming a mapped column** - CDC Sink enters fallback because the configured column name is no longer found. + Update the task configuration to match the new schema. + + + +**Apply schema changes one at a time**: + +MySQL CDC Sink detects schema changes by comparing binlog type codes at specific column positions. +If multiple changes are applied in a single `ALTER TABLE` statement (e.g., `ADD COLUMN ... AFTER ...`, or adding and removing columns together), +the positional shifts can compound and make the change harder to detect reliably. + +Apply one schema change, let CDC Sink catch up and confirm it is processing normally, then apply the next change. + + + +### MySQL examples + + + +```sql +-- Transparent: adding an unmapped column +ALTER TABLE orders ADD COLUMN internal_notes TEXT; +-- CDC Sink continues without interruption + +-- Auto-recovers when the binlog type code changes +ALTER TABLE orders MODIFY COLUMN priority BIGINT; +-- CDC Sink detects the change, restarts, and re-resolves the schema + +-- Requires config update: dropping a mapped column +ALTER TABLE orders DROP COLUMN legacy_field; +-- CDC Sink enters fallback. Update the config to remove 'legacy_field' from Columns. +``` + + + + + + + +SQL Server CDC uses **capture instances** - immutable snapshots of the table schema at the time `sp_cdc_enable_table` was executed. +The CDC change tables reflect the schema as it was when the capture instance was created, +**not** the current table schema. +This means that to move CDC Sink to the updated table schema, you must create a new capture instance. + +SQL Server supports at most **two** capture instances per table at a time. +CDC Sink reads from the oldest active instance. When the old instance is dropped, +CDC Sink detects that `fn_cdc_get_min_lsn` returns all zeros, confirms the instance is no longer listed in `cdc.change_tables`, +then restarts and picks up the new instance. + +### SQL Server schema change procedure + +For **any** schema change on a SQL Server CDC-tracked table, follow this procedure: + + + +```sql +-- Step 1: ALTER the table +ALTER TABLE dbo.orders ADD priority INT DEFAULT 0; + +-- Step 2: Create a NEW capture instance (the old one still exists) +EXEC sys.sp_cdc_enable_table + @source_schema = N'dbo', + @source_name = N'orders', + @capture_instance = N'dbo_orders_v2', + @role_name = NULL; + +-- Step 3: WAIT - let CDC Sink drain the OLD capture instance. +-- The sink reads from the oldest instance and processes all remaining changes. +-- Monitor the CDC Sink task status in Studio to confirm it has caught up. + +-- Step 4: Drop the OLD capture instance +EXEC sys.sp_cdc_disable_table + @source_schema = N'dbo', + @source_name = N'orders', + @capture_instance = N'dbo_orders'; + +-- CDC Sink detects the drop, restarts, and picks up dbo_orders_v2. +``` + + + +**What happens at each step:** + +1. **After step 2** - Two capture instances exist. + CDC Sink continues reading from the oldest one (`dbo_orders`). + New changes are captured by both instances, but the old instance still exposes only its original column set. +2. **After step 3** - CDC Sink has processed all changes from the old instance. It's safe to drop it. +3. **After step 4** - `fn_cdc_get_min_lsn('dbo_orders')` returns all zeros. + CDC Sink detects this on the next poll cycle, enters fallback, and restarts against the new instance. + +If the schema change also affects column mappings, update the CDC Sink task configuration after the new capture instance is active. + + + +Do not drop the old capture instance before CDC Sink has drained it - +you may lose changes that were captured in the old instance but not yet processed. + + + +### SQL Server Agent + +CDC requires SQL Server Agent to be running. +The capture job (which populates change tables) and cleanup job (which purges old changes) are Agent jobs. + + + +```sql +-- Check Agent status +SELECT * FROM sys.dm_server_services WHERE servicename LIKE '%Agent%'; +``` + + + +Without the Agent, changes are never captured and CDC Sink polls indefinitely without +finding data. + + + + + +All three providers use the same top-level recovery loop, but the outcome depends on the error type: + +1. **Transient error detected** - the process enters fallback mode. + The retry delay starts at 5 seconds and increases up to `CdcSink.MaxFallbackTimeInSec`. + A missing mapped column (a dropped or renamed column referenced by the task configuration) is transient: + the sink stays in fallback and recovers once the configuration matches the new schema. + +2. **On each retry** - the process starts a new attempt: + it re-opens connections and re-resolves the source schema or provider metadata. + +3. **If the transient error persists** - retries continue with increasing backoff. + A notification is raised for each error and is visible in Studio. + +4. **If the error is a permanent configuration/schema fault** - + (for example, a configured table does not resolve in the task's table mapping) - the process is marked faulted and stops retrying. + Correct the task configuration; saving the updated configuration recreates the process. + +5. **After the source issue or configuration is fixed** - + CDC Sink resumes automatically on the next retry or after the corrected configuration is saved. + No manual task restart is needed. + + + + + +For schema changes that affect mapped columns: + +1. Prepare the CDC Sink task configuration that matches the final source schema. +2. Apply the schema change on the source database. +3. Save the updated CDC Sink task configuration. +4. For SQL Server, follow the [capture instance procedure](#sql-server-schema-change-procedure) before saving mappings that depend on the new capture instance. + +CDC Sink will recover after the source schema and task configuration match. +Disabling the task beforehand is not required, but it can avoid a temporary fallback or faulted state while the schema and configuration are out of sync. + \ No newline at end of file diff --git a/docs/server/ongoing-tasks/cdc-sink/troubleshooting.mdx b/docs/server/ongoing-tasks/cdc-sink/troubleshooting.mdx new file mode 100644 index 0000000000..105f2d2e13 --- /dev/null +++ b/docs/server/ongoing-tasks/cdc-sink/troubleshooting.mdx @@ -0,0 +1,330 @@ +--- +title: "CDC Sink: Troubleshooting" +sidebar_label: Troubleshooting +description: "Covers common problems encountered when running CDC Sink tasks and how to resolve them, including startup failures, error states, and missing documents." +sidebar_position: 12 +see_also: + - title: "How It Works" + link: "server/ongoing-tasks/cdc-sink/how-it-works" + source: "docs" + path: "Server > CDC Sink" + - title: "Monitoring" + link: "server/ongoing-tasks/cdc-sink/monitoring" + source: "docs" + path: "Server > CDC Sink" + - title: "Patching" + link: "server/ongoing-tasks/cdc-sink/patching" + source: "docs" + path: "Server > CDC Sink" + - title: "Failover and Consistency" + link: "server/ongoing-tasks/cdc-sink/failover-and-consistency" + source: "docs" + path: "Server > CDC Sink" + - title: "PostgreSQL WAL Configuration" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/wal-configuration" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "PostgreSQL Permissions and Roles" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/permissions-and-roles" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" + - title: "PostgreSQL REPLICA IDENTITY" + link: "server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity" + source: "docs" + path: "Server > CDC Sink > PostgreSQL" +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Panel from '@site/src/components/Panel'; + + + +* Use this article to diagnose common CDC Sink task problems and find the relevant recovery steps. + +* In this article: + * [Task fails to start](#task-fails-to-start) + * [Task enters error state](#task-enters-error-state) + * [Documents not appearing](#documents-not-appearing) + * [Embedded items missing or incorrect](#embedded-items-missing-or-incorrect) + * [DELETE not applied](#delete-not-applied) + * [Patch errors](#patch-errors) + * [Source schema changes](#source-schema-changes) + * [Partial export/import and state loss](#partial-export-import-and-state-loss) + * [Shared replication slot across databases](#shared-replication-slot-across-databases) + + + + + +**Symptom**: +Task remains in `Error` state immediately after creation. + +**Common causes**: + +* **Invalid connection string** + Verify that the connection string name matches a SQL connection string defined for the database. + +* **Connection refused** + The source database is unreachable from the RavenDB server. Verify host, port, and firewall rules. + +* **Authentication failure** + The credentials in the connection string are incorrect or the source-database user does not have the required permissions. + For PostgreSQL, see [Permissions and Roles](../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/permissions-and-roles.mdx). + +* **PostgreSQL WAL level not set to `logical`** + CDC Sink requires logical replication enabled on the PostgreSQL source. + See [WAL Configuration](../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/wal-configuration.mdx). + +Check the error message in the Notification Center (bell icon in Studio) for the specific failure reason. + + + + + +**Symptom**: +Task was running, then stopped with an error. + +**Common causes**: + +* **PostgreSQL replication slot issue** + Verify that the configured slot exists and uses the `pgoutput` plugin. + + If a slot with the task's configured name already exists but uses a different plugin, + CDC Sink fails to start with an error such as + *"Replication slot '<name>' exists but uses plugin '<plugin>' instead of 'pgoutput'"*. + Drop the conflicting slot so CDC Sink can recreate it with `pgoutput` + (see [Dropping a replication slot](../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/cleanup-and-maintenance.mdx#dropping-a-replication-slot)). + To use a different slot name, delete the task and create a new one with a distinct `SlotName`. + + If the slot was dropped and recreated, CDC Sink may detect that the slot starts after the task's saved LSN; + changes in that gap cannot be recovered from the slot. + +* **Source table schema changed** + Mapped column changes, renamed columns, or SQL Server CDC schema changes can make the task enter fallback mode or require configuration/source-side updates. + See [Source Schema Changes](../../../server/ongoing-tasks/cdc-sink/source-schema-changes.mdx). + +* **PostgreSQL replication lag too large** + If CDC Sink falls behind, the PostgreSQL replication slot can retain WAL segments and consume disk space. + Check slot lag and relevant PostgreSQL WAL-retention settings, such as `max_slot_wal_keep_size`. + +* **Persistent connection failure** + The source has been unreachable through multiple retry cycles. The first fallback delay is 5 seconds; + subsequent retry delays roughly double up to the `CdcSink.MaxFallbackTimeInSec` base cap (default 15 minutes), with small jitter. + Restore connectivity and the task resumes automatically on the next retry. + + + + + +**Symptom**: +SQL rows exist but corresponding RavenDB documents are not created. + +**Check**: + +1. **Initial load still running** + CDC Sink scans configured tables before it starts streaming captured changes. + For large tables, rows that have not been scanned yet will not appear. Check the task progress in Studio. + +2. **`SkipInitialLoad = true`** + If the task was created with `SkipInitialLoad`, existing source rows are not scanned. + They appear only if they were already loaded into RavenDB or are later changed in the source database. + +3. **Task or table is disabled** + Verify the task state is `Enabled` or `PartiallyEnabled`, not `Disabled`, + and that the table's `Disabled` flag is not set to `true`. + +4. **Primary-key columns are wrong** + `PrimaryKeyColumns` are used to build the RavenDB document ID. + Verify they name source columns that exist, uniquely identify the row, and are included in `Columns`. + +5. **Expected columns are not mapped** + Only columns listed in `Columns` are written as document properties. + Add mappings for any source columns that should appear in the document. + + + + + +**Symptom**: +Parent document exists but embedded array/map is empty or items are missing. + +**Check**: + +1. **`JoinColumns` mismatch** + `JoinColumns` must contain the embedded table columns whose values identify the direct parent row. + CDC Sink pairs them with the parent's `PrimaryKeyColumns` by position, so the count and order must match. + +2. **Embedded `PrimaryKeyColumns` are wrong** + For `Array` and `Map` embedded tables, these columns identify the item inside the parent's array or map. + If they are incomplete or wrong, UPDATE events can create duplicates and DELETE events may not find the item to remove. + +3. **DELETE event missing routing columns** + For PostgreSQL embedded table deletes, CDC Sink needs the old row values for both the join columns and the embedded item's primary-key columns. + If the join columns are not part of the table's primary key, configure `REPLICA IDENTITY` so DELETE events include them. + See [REPLICA IDENTITY](../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx). + +4. **Embedded `Type` changed after data was written** + `Type` controls whether embedded data is stored as an `Array`, `Map`, or `Value`. + If you change it after documents already exist, migrate or re-process existing documents so the property has a consistent shape. + + + + + +**Symptom**: +A source row was deleted, but the RavenDB document or embedded item still exists. + +**Check**: + +1. **`OnDelete.IgnoreDeletes = true`** + When this flag is set, CDC Sink skips the automatic document delete or embedded-item removal. + If `OnDelete.Patch` is also configured, the patch still runs, but the item is kept unless the script removes it explicitly. + +2. **PostgreSQL DELETE event missing routing columns (embedded tables)** + For embedded table deletes, CDC Sink needs old row values that identify the parent document and the embedded item. + If those columns are not covered by the table's primary key or configured `REPLICA IDENTITY`, + the DELETE event may not contain enough data to route the delete. + See [REPLICA IDENTITY](../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/replica-identity.mdx). + +3. **Delete patch keeps the data intentionally** + Review `OnDelete.Patch` for archive or soft-delete logic, such as setting an `Archived` flag instead of deleting the document or embedded item. + + + + + +**Symptom**: +Task enters error state with a message referencing a JavaScript patch failure. + +**Check**: + +1. **Script syntax or runtime error** + Test the script with the CDC Sink task's **Test** tool. + The test runs CDC Sink mapping and the relevant `Patch` or `OnDelete.Patch` against source rows without saving documents. + +2. **Missing values in the patch context** + `$old` can be `null` for new documents/items, and DELETE events may not include every source column in `$row`. + Use optional chaining and nullish coalescing, for example: `$old?.Amount ?? 0`. + +3. **`load()` returns `null`** + A document loaded with `load(id)` may not exist yet. + Guard the result before reading from it: + + + +```javascript +const related = load("Collection/123"); +if (related) { + this.RelatedName = related.Name; +} +``` + + + +4. **Step limit exceeded** + Patch scripts have a step quota controlled by `Patching.MaxStepsForScript` (default `10_000`). + Long loops, repeated `load()` calls, or large in-script computations can exceed it. + See the [configuration reference](../../../server/configuration/patching-configuration.mdx) for this setting. + + + + + +* Adding, removing, renaming, or retyping columns in source tables can affect CDC Sink differently depending on the database engine. + +* PostgreSQL handles many changes transparently, MySQL can detect many changes but has positional and type-code limitations, + and SQL Server requires an explicit capture-instance procedure for schema changes. + +* See [Source Schema Changes](../../../server/ongoing-tasks/cdc-sink/source-schema-changes.mdx) for the full guide, + including per-engine behavior, examples, recovery steps, and the SQL Server procedure. + + + + + +**Symptom**: +After importing a database export, CDC Sink re-reads all data from the source database from the beginning, +performing a full initial load even though the task configuration is present and the data already exists in RavenDB. + +**Cause**: + +CDC Sink stores two separate pieces of information: + +* **Task configuration** + Stored in the cluster-wide **database record**. + It is included by default when the `CdcSinks` database-record item is exported/imported. +* **Processing state** + Stored as a document named `@cdc-states/` in the `@cdc-states` system collection. + This tracks the last processed source-database change position (`LastLsn`) and per-table initial-load progress. + +If you export/import the task configuration but skip documents, skip system documents, or exclude the `@cdc-states` collection, +the task configuration is restored but the processing state document is missing. + +When CDC Sink starts with a valid configuration and no matching state document, it starts fresh: +it captures a new source checkpoint, performs a full initial load of the configured enabled tables, +and then streams changes from that checkpoint. + +For large source databases, +the new initial load can take significant time and place heavy load on both the source database and RavenDB. + +**How to avoid this:** + +* Include documents in the export/import, including the `@cdc-states` collection. +* If using Smuggler or the REST API for selective export/import, + make sure system documents are included and the `@cdc-states` collection is not filtered out. +* Imported tasks are disabled by default. + Before enabling the task, verify that the matching `@cdc-states/` document was imported. +* If the state document is missing but the target RavenDB data is already up to date, + set `SkipInitialLoad = true` before the task's first start. + If the task has already started, recreate it with `SkipInitialLoad = true`. + + + +There is no _supported_ way to manually set the last LSN position. + +Although the state document in `@cdc-states` can technically be edited, doing so is risky: +an incorrect LSN can skip changes, replay changes from the wrong point, or produce inconsistent documents. + +If the state document is lost, the safest options are to re-read from the beginning or start with `SkipInitialLoad` +when you are sure the target data is already up to date. + + + + + + + +**Symptom (PostgreSQL)**: +After copying a CDC Sink task to a second database (via export/import or backup/restore), +one of the two tasks sits in fallback mode with an error like: + +```plain +Process is in error recovery (fallback mode). Last error: . Next retry in: 464s. +``` +
+ +**Cause**: + +Export/import and backup/restore copy the task configuration verbatim, +including the immutable `SlotName` and `PublicationName`. + +A PostgreSQL logical replication slot can be consumed by only **one active connection at a time**. +When two enabled CDC Sink tasks point to the same slot, +one task attaches to the slot and the other enters error-recovery (fallback) mode. + +While the second task cannot attach, it does not apply new source changes. +Processing resumes only after that task can connect to the slot on a later retry. + +**Resolution:** + +* Keep the task **enabled in only one database at a time** when the configuration is shared. + For export/import, imported tasks are disabled by default; + for restore, use `DisableOngoingTasks` or disable the task on one database before both can run. +* To run the task against **both** databases at once, give each copy its own dedicated slot and publication + (set a distinct `SlotName`/`PublicationName` in `CdcSinkPostgresSettings` before enabling the copy). + +See [One active consumer per slot](../../../server/ongoing-tasks/cdc-sink/source-database-setup/postgres/initial-setup.mdx#one-active-consumer-per-slot). + + \ No newline at end of file diff --git a/docs/studio/database/tasks/ongoing-tasks/general-info.mdx b/docs/studio/database/tasks/ongoing-tasks/general-info.mdx index 943c00d190..87e047a8ab 100644 --- a/docs/studio/database/tasks/ongoing-tasks/general-info.mdx +++ b/docs/studio/database/tasks/ongoing-tasks/general-info.mdx @@ -129,6 +129,9 @@ The following task types are available: * **[RabbitMQ Sink](../../../../studio/database/tasks/ongoing-tasks/rabbitmq-queue-sink.mdx)** Consume and process incoming messages from RabbitMQ queues. Add scripts to Load, Put, or Delete documents in RavenDB based on the incoming messages. +* **[CDC Sink](../../../../server/ongoing-tasks/cdc-sink/overview.mdx)** + Read Change Data Capture streams from a relational database (PostgreSQL, SQL Server, or MySQL/MariaDB). + Insert, update, and delete operations from the source database are continuously written as documents into RavenDB.