PKG-895: Allow citus and timescaledb to compile by dutow · Pull Request #600 · percona/postgres

dutow · 2026-05-11T21:01:50Z

Let's talk about this patch again!

Same as before, only for the PSP18 branch for now, 17 needs the same changes of course.

The pg_tde repo will need a matching

allow_upstream_smgr_api = false;

change if we go with this patch as-is.

This commit reintroduces the old method signatures of some methods we changed to support pg_tde. The changed method signatures still exists with a "2" suffix, and all internal code calls the "2" methods.

The reintroduced original methods also check a global flag which can disable them - if this happens, they report a FATAL error instead. This variable is activated by pg_tde, which means that if something tries to use these while pg_tde is loaded, and with that, possibly ignores its file tracking mechanism, it reports an error.

In practice this means that timescaledb, or the columnar storage in citus won't work if pg_tde is loaded in the shared_preload_libraries, but they can be used with our distribution without pg_tde enabled.

Also verified that both citus and timescaledb do compile with these changes, and their basic test suite can be run againts our fork.

The original part of the version is kept as is, and Percona specific information is added after. For example, psql displays the following after this commit: SELECT version(); PostgreSQL 17.6 (PostgreSQL) - Percona Server for PostgreSQL 17.6.1 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit Where the addition is the extra version at the end: " - Percona Server for PostgreSQL 17.4.1" with the ".1" being the Percona version number. Similarly, commands show the same information: bin/pg_ctl --version pg_ctl (PostgreSQL) 17.6 - Percona Server for PostgreSQL 17.6.1

There are various reasons why one would want to create their own implementation of a storage manager, among which are block-level compression, encryption and offloading to cold storage. This patch is a first patch that allows extensions to register their own SMgr. Note, however, that this SMgr is not yet used - only the first SMgr to register is used, and this is currently the md.c smgr. Future commits will include facilities to select an SMgr for each tablespace.

With this change, mdcreate receives the old relfilelocator along with the new for operations that create a new file for an existing relation. This is required for tde_heap in pg_tde.

…(#156)

and allow extensions to override it For now, it extends on `pread` and `pwrite` from/into segment files. This is the minimum we need for full XLog encryption with pg_de.

This commit modifies the build scripts so that `llvmjit.so` links LLVM statically instead of dynamically. No other binary affected, the change shouldn't be user visible at all. The memory footprint of postgres is also unaffected: only the `llvmjit.so` dynamic library is affected. If postgres doesn't use JIT, the LLVM library isn't loaded, as before. If it uses JIT, it is loaded as part of `llvmjit.so`, sa before. The only change is the binary size of `llvmjit.so`.

Instead of checking for the oid of the AM we check for if the access method of the relation uses the routine struct as the heap does. This allows the tools to be support all access methods which are simple copies of heap. I am uncertain if this is something which the project would be interested in but it decreases the size of our diff against upstream.

Need to modify the existing license to include the portions of code unique to Percona

The pg_tde extension's custom storage manager needs to add its own AIO callback to decrypt pages before the buffer manager's callback is called but since the set of AIO callbacks is fixed we need to expose this API so that extensions can register their own types of callbacks. This new function for registering a callback is supposed to be called when intializing shared memory. We try to keep the number of callbacks low, setting it to 15, so that we have the 4 built-in callbacks and 11 slots that extensions can use.

This commit reintroduces the old method signatures of some methods we changed to support pg_tde. The changed method signatures still exists with a "2" suffix, and all internal code calls the "2" methods. The reintroduced original methods also check a global flag which can disable them - if this happens, they report a FATAL error instead. This variable is activated by pg_tde, which means that if something tries to use these while pg_tde is loaded, and with that, possibly ignores its file tracking mechanism, it reports an error. In practice this means that timescaledb, or the columnar storage in citus won't work if pg_tde is loaded in the shared_preload_libraries, but they can be used with our distribution without pg_tde enabled. Also verified that both citus and timescaledb do compile with these changes, and their basic test suite can be run againts our fork.

jeltz

Apaart from some naming and style feedback I have one big question:

It feels a bit scary that we error out (now with FATAL but I think it could just be ERROR, not that that would help that much) not when Timescale or Citus are loaded but instead later when something uses them. But I can't really think of a better solution. We could of course have pg_tde warn if we see them loaded but we do not want to prevent them for using their own patched Citus so we can jsut warn which people probably will miss.

Maybe this is the least bad, hmm.

jeltz · 2026-05-13T17:51:33Z

 	*minmulti = GetOldestMultiXactId();

-	srel = RelationCreateStorage(oldlocator, *newrlocator, persistence, true);
+	srel = RelationCreateStorage2(oldlocator, *newrlocator, persistence, true);


I wonder if we should not name it RelationCreateStoragePSP or RelationCreateStoragePercona instead to make it totally clear it is our function.

jeltz · 2026-05-13T17:52:00Z

+						 bool is_internal,
+						 Oid *constraintId)
+{
+	if(!allow_upstream_smgr_api) {


This does not follow the PostgreSQL code style.

jeltz · 2026-05-13T17:52:45Z

+						 Oid *constraintId)
+{
+	if(!allow_upstream_smgr_api) {
+		elog(FATAL, "An extension is trying to use the traditional index_create method, while another loaded extension (pg_tde) requires the new API.");


Should this be FATAL or ERROR? I am leaning towards ERROR. If the error happens in a critical section won't it be fatal anyway?

jeltz · 2026-05-13T17:53:31Z

+										  char relpersistence,
+										  bool register_delete)
+{
+	if(!allow_upstream_smgr_api) {


This also does not follow PG's C style.

jeltz · 2026-05-13T17:54:00Z


 SMgrId		storage_manager_id;

+bool allow_upstream_smgr_api = true;


Prefix with percona?

jeltz · 2026-05-13T17:54:18Z


+void smgrcreate(SMgrRelation reln, ForkNumber forknum, bool isRedo)
+{
+	if(!allow_upstream_smgr_api) {


This too does not follow code style. :)

dutow and others added 17 commits February 23, 2026 23:30

PG-1126: Expose Percona API version

c6ba821

Added percona server version scheme verification TAP test case

8eca938

Allow extensions to override the global storage manager

a4abce6

Refactor smgr API: mdcreate needs the old relfilelocator

03e8f15

With this change, mdcreate receives the old relfilelocator along with the new for operations that create a new file for an existing relation. This is required for tde_heap in pg_tde.

Pass proper old rlocator to smgr when creating relations and indexes …

0751dad

…(#156)

Make XLog storage extensible

4b851bb

and allow extensions to override it For now, it extends on `pread` and `pwrite` from/into segment files. This is the minimum we need for full XLog encryption with pg_de.

[PG-1115] Add TDE support in regression tests

c2b1dc0

Update expected files for regression tests to run pg_tde

2d99cd0

Updated readme and copyright file (#57)

1fe8c9f

Update COPYRIGHT (#317)

1248a66

Need to modify the existing license to include the portions of code unique to Percona

Add extension point for quick fix to support sync AIO in the SMGR

aa64b84

dutow force-pushed the PSP_REL_18_STABLE branch from 50dd77c to a02a779 Compare May 12, 2026 10:20

jeltz requested changes May 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PKG-895: Allow citus and timescaledb to compile#600

PKG-895: Allow citus and timescaledb to compile#600
dutow wants to merge 17 commits into
percona:PSP_REL_18_STABLEfrom
dutow:pkg895b

dutow commented May 11, 2026

Uh oh!

jeltz left a comment

Uh oh!

jeltz May 13, 2026

Uh oh!

jeltz May 13, 2026

Uh oh!

jeltz May 13, 2026

Uh oh!

jeltz May 13, 2026

Uh oh!

jeltz May 13, 2026

Uh oh!

jeltz May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants


		SMgrId storage_manager_id;

		bool allow_upstream_smgr_api = true;

Conversation

dutow commented May 11, 2026

Uh oh!

jeltz left a comment

Choose a reason for hiding this comment

Uh oh!

jeltz May 13, 2026

Choose a reason for hiding this comment

Uh oh!

jeltz May 13, 2026

Choose a reason for hiding this comment

Uh oh!

jeltz May 13, 2026

Choose a reason for hiding this comment

Uh oh!

jeltz May 13, 2026

Choose a reason for hiding this comment

Uh oh!

jeltz May 13, 2026

Choose a reason for hiding this comment

Uh oh!

jeltz May 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants