Skip to content

Helpers to move between raw tables and ps_untyped#189

Open
simolus3 wants to merge 3 commits into
mainfrom
raw-table-migration-helpers
Open

Helpers to move between raw tables and ps_untyped#189
simolus3 wants to merge 3 commits into
mainfrom
raw-table-migration-helpers

Conversation

@simolus3

Copy link
Copy Markdown
Contributor

With the _rest column pattern and powersync_trigger_resync, we have pretty decent tools integrating PowerSync into migrations altering existing tables.

However, steps to create and drop raw tables could still be simplified a bit:

  1. To add raw tables, we currently suggest a manual copy from ps_untyped into the raw table. We forgot to mention that moved rows should also be removed from ps_untyped afterwards (Room integration: Pre-sync data not applied to new tables after upgrade powersync-kotlin#355).
  2. For symmetry, one should also move rows back into ps_untyped before dropping raw tables. We have no documentation on this at all.

Both steps can be automated fairly easily: For 1, we can use the exact same logic we use in a sync_local step to parse the oplog JSON structure from ps_untyped through the (inferred or explicit) put statement, which copies it into the raw table.
For 2, we can use the existing logic we have to create crud triggers to JSON-ify table contents and write them back into ps_untyped.

Can't we do this implicitly?

Initially, I thought we could this in powersync_init() / powersync_update_schema(), just like we auto-migrate views:

  1. We could detect a new raw table in the schema and automatically copy data over.
  2. ... and that's where it stops being a good idea, because we can't detect deleted raw tables missing from the schema.

Even 1 breaks in practice because depending on the SDK, users might first call powersync_init with the raw table already in the schema before using execute() APIs in SDKs to actually create the raw table.

So, this PR adds an explicit function for this, powersync_raw_table_migrate. It's supposed to be called like this:

  • For new raw tables, after the table has been created and added to the app's schema. Invoke it with powersync_raw_table_migrate('create', 'name_of_raw_table_in_sync_streams') to move data from ps_untyped.
  • For dropped raw tables, before the table gets dropped in a migration. The table would not be part of the app's schema at this point, so it takes a JSON description of the table (necessary to describe local-only columns, which should not be moved to ps_untyped).

AI use disclaimer: Parts of the implementation and most tests were generated with Claude.

@simolus3 simolus3 marked this pull request as ready for review June 12, 2026 14:25
@simolus3 simolus3 requested a review from rkistner June 12, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant