Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
results
regression_output
tmp_check
spock_create_subscriber
/utils/spock_create_subscriber/spock_create_subscriber
/utils/spock_create_subscriber/.deps/
.vimrc
*.o
*.so
Expand Down
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ EXTENSION = spock
PGFILEDESC = "spock - multi-master replication"

MODULES = spock_output
SUBDIRS = utils/spock_create_subscriber

# Lookup source directory
vpath % src src/compat/$(PGVER)
Expand Down
38 changes: 38 additions & 0 deletions docs/creating_subscriber_nodes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
## Creating a Subscriber Node with pg_basebackup

Spock supports creating a subscriber node by cloning the provider with [`pg_basebackup`](https://www.postgresql.org/docs/current/app-pgbasebackup.html) and starting it as a Spock subscriber. Use the `spock_create_subscriber` utility (located in the `bin` directory of your pgEdge platform installation) to register the node.

### Synopsis:

`spock_create_subscriber [OPTION]...`

**Options**

Specify the following options as needed.

| Option | Description
|----------|-------------
| `-D`, `--pgdata=DIRECTORY` | The `data` directory to be used for new node. This can be either an empty/non-existing directory, or a directory populated using the `pg_basebackup -X stream` command.
| `--databases` | An optional list of databases to replicate.
| `-n`, `--subscriber-name=NAME` | The name of the newly created subscriber.
| `--subscriber-dsn=CONNSTR` | A connection string to the newly created subscriber.
| `--provider-dsn=CONNSTR` | A connection string to the provider.
| `--replication-sets=SETS` | A comma-separated list of replication set names.
| `--apply-delay=DELAY` | The apply delay in seconds (by default 0).
| `--drop-slot-if-exists` | Drop replication slot of conflicting name.
| `-s`, `--stop` | Stop the server once the initialization is done.
| `-v` | Increase logging verbosity.
| `--extra-basebackup-args` | Additional arguments to pass to `pg_basebackup`. Safe options are: `-T`, `-c`, `--xlogdir`/`--waldir`
Comment on lines +9 to +25

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Document the --text-types option.

The --text-types CLI option (line 211 in spock_create_subscriber.c, which sets force_text_transfer) is neither listed in this options table nor in the usage() output (lines 578–603). Users have no way to discover it.

Add a row to the options table and a corresponding printf in usage().

📝 Proposed additions
 | `--extra-basebackup-args` | Additional arguments to pass to `pg_basebackup`. Safe options are: `-T`, `-c`, `--xlogdir`/`--waldir`
+| `--text-types` | Force text transfer for all columns.
 printf(_("  --extra-basebackup-args     additional arguments to pass to pg_basebackup.\n"));
 printf(_("                              Safe options: -T, -c, --xlogdir/--waldir\n"));
+printf(_("  --text-types                 force text transfer for all columns\n"));
🧰 Tools
🪛 LanguageTool

[grammar] ~20-~20: Use a hyphen to join words.
Context: ...r. | --replication-sets=SETS | A comma separated list of replication set names....

(QB_NEW_EN_HYPHEN)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/creating_subscriber_nodes.md` around lines 9 - 25, The `--text-types`
CLI option is missing from both the subscriber creation documentation and the
CLI help output, so users cannot discover it. Add a new row for `--text-types`
in the options table in the subscriber docs, and add a matching `printf` entry
in `usage()` in `spock_create_subscriber.c` alongside the existing option
descriptions, referencing the `force_text_transfer` behavior.

| `--text-types` | Transfer all column values as text rather than binary during initial sync. Use this when provider and subscriber differ in endianness or type representation.

**Configuration files overrides**

You can use the following options to override the location of the configuration files.

| Option | Description
|----------|-------------
|`--hba-conf` | path to the new `pg_hba.conf`
| `--postgresql-conf` | path to the new `postgresql.conf`
| `--recovery-conf` | path to the template recovery configuration

Unlike `spock.sub_create`'s other data sync options, this method of cloning ignores replication sets and copies all tables on all databases. However, it's often much faster, especially over high-bandwidth connections.
3 changes: 3 additions & 0 deletions docs/spock_functions/functions/spock_sub_alter_options.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ options
element is the string "all". Pass an empty array ([]) to disable
origin forwarding.

See [Origin Forwarding](../sub_mgmt.md#origin-forwarding) for the
restriction on combining this with other enabled subscriptions.

apply_delay

A PostgreSQL interval string (e.g. "2 seconds", "500ms", "0")
Expand Down
3 changes: 3 additions & 0 deletions docs/spock_functions/functions/spock_sub_create.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ forward_origins
replication to avoid forwarding changes in a loop). Use {all} to replicate all
changes regardless of origin. The default is `{}` (local-origin changes only).

See [Origin Forwarding](../sub_mgmt.md#origin-forwarding) for the
restriction on combining this with other enabled subscriptions.

apply_delay

An interval specifying how long to delay applying changes from the
Expand Down
4 changes: 4 additions & 0 deletions docs/spock_functions/functions/spock_sub_enable.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ spock.sub_enable(subscription_name name, immediate boolean)

The `spock.sub_enable()` function enables a subscription.

This fails if another subscription on this node already has origin
forwarding active; see
[Origin Forwarding](../sub_mgmt.md#origin-forwarding).

## Arguments

The function accepts the following arguments:
Expand Down
49 changes: 39 additions & 10 deletions docs/spock_functions/sub_mgmt.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,9 @@ Parameters include:
`false`.
- `synchronize_data` specifies if Spock should synchronize data from
provider to the subscriber; the default is `false`.
- `forward_origins` is an array of origin names to forward. Currently, the
only supported values are an empty array meaning don't forward any changes
that didn't originate on provider node (this is useful for two-way
replication between the nodes), or `{all}` which means replicate all changes
no matter what is their origin. The default is `{}` (an empty array, meaning only local changes are forwarded).
- `forward_origins` is an array of origin names to forward. See
[Origin Forwarding](#origin-forwarding) below for supported values and the
restriction on combining this with other enabled subscriptions.
- `apply_delay` specifies how long to delay replication; the default is `0`
seconds.
- set `force_text_transfer` to `true` to force the provider to replicate all
Expand Down Expand Up @@ -100,6 +98,36 @@ Drops a subscription named `accts`; if the subscription does not exist, an
error message will be suppressed by the `true` trailing parameter (`ifexists =
true`).

## Origin Forwarding

`forward_origins` controls whether a subscription's apply worker also
processes transactions that did not originate on the immediate provider, but
were relayed through it from a peer further upstream (a cascade or multi-hop
topology). `{}` (the default) forwards only the provider's own
locally-originated changes — the setting used for ordinary bidirectional
replication between two nodes, since forwarding a peer's changes back to
itself would loop. `{all}` forwards every change regardless of origin.

A forwarding worker advances the replication origin of the local subscription
matching the peer it's relaying (for example, a subscription to that peer
created disabled ahead of time, so it can later start from the right
position instead of a full resync). At most one local subscription may match
a given peer; if more than one does, replication fails with an ambiguous-match
error. If the matching subscription is also enabled, its own apply worker
owns the same origin, and the two would conflict. Spock avoids this by
requiring that a subscription with forwarding active be the only enabled
subscription on the node:

- Enabling forwarding (`{all}`) on a subscription — via `spock.sub_create`
with `enabled := true`, `spock.sub_enable`, or `spock.sub_alter_options`
on an already-enabled subscription — fails if another subscription is
already enabled on this node.
- Enabling a subscription (`spock.sub_create` with `enabled := true`, or
`spock.sub_enable`) fails if another subscription on this node already has
forwarding active.
- Changing `forward_origins` on a subscription that stays disabled is always
allowed; the restriction is only checked once the subscription is actually
enabled.

## Subscription Management Functions

Expand Down Expand Up @@ -127,11 +155,9 @@ Parameters:
structure from the provider to the subscriber; the default is `false`.
- `synchronize_data` tells Spock to synchronize data from provider to the
subscriber; the default is `false`.
- `forward_origins` is an array of origin names to forward. Currently, the
only supported values are an empty array meaning don't forward any changes
that didn't originate on the provider node (this is useful for two-way
replication between the nodes), or `{all}` which means replicate all
changes regardless of their origin. The default is `{}` (an empty array, meaning only local changes are forwarded).
- `forward_origins` is an array of origin names to forward. See
[Origin Forwarding](#origin-forwarding) above for supported values and the
restriction on combining this with other enabled subscriptions.
- `apply_delay` is the number of seconds to delay replication; the default
is `0` seconds.
- `force_text_transfer` forces the provider to replicate all columns using
Expand Down Expand Up @@ -192,6 +218,9 @@ Parameters:
the subscription is started immediately; if set to `false` (the default),
it will only be started at the end of the current transaction.

This fails if another subscription on this node already has forwarding
active; see [Origin Forwarding](#origin-forwarding) above.

### spock.sub_alter_interface

Use `spock.sub_alter_interface` to modify the subscription to use a different
Expand Down
7 changes: 7 additions & 0 deletions docs/spock_release_notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,13 @@ included in ORIGIN messages when the protocol version is 5 or higher.
This ensures that conflict evaluation on Node C has accurate origin
information even when changes pass through intermediate Node B.

A subscription cannot activate `forward_origins` while another subscription
is already enabled on the same node, and a subscription cannot be enabled
while another has forwarding active; clear `forward_origins`
(`spock.sub_alter_options`) or disable the other subscription first. See
[Origin Forwarding](spock_functions/sub_mgmt.md#origin-forwarding) for
details.

### Per-subscription conflict statistics

On PostgreSQL 18+, Spock registers a custom pgstat kind
Expand Down
Loading
Loading