Skip to content

feat: Use SSL and SSH config in LOG_BASED replication#717

Merged
edgarrmondragon merged 13 commits intoMeltanoLabs:mainfrom
danilofuchs:feat/ssl-and-ssh-for-log-based
Jan 13, 2026
Merged

feat: Use SSL and SSH config in LOG_BASED replication#717
edgarrmondragon merged 13 commits intoMeltanoLabs:mainfrom
danilofuchs:feat/ssl-and-ssh-for-log-based

Conversation

@danilofuchs
Copy link
Copy Markdown
Contributor

Fixes #469

Problem

There are two libraries used for connecting to the PostgreSQL database, depending on the replication mode:

  1. FULL_TABLE and INCREMENTAL: SQLAlchemy + psycopg2
  2. LOG_BASED: psycopg2 directly

The SQLAlchemy URL takes into account SSL and SSH configuration options, but the psycopg2 connection uses the raw config values.

Another inconsistency is that application_name=tap-postgres is only set if using the psycopg2 connection.

Solution

Creating a shared ConnectionParameters dataclass which can be built via either discrete config parameters, or via a sqlalchemy_url config. This dataclass can then be rendered as either a SQLAlchemy or a psycopg2 URL

The PostgresLogBasedStream uses the exact same ConnectionParameters as the other streams, via their shared PostgresConnector

Note: I opted to now allow configuring log_based replication with the sqlalchemy_url config, for more flexibility

Questions

  1. I was not able to run the SSH tests locally, and am not sure if another container would be necessary for integration tests.

Future improvements

  1. Add a new tap option called postgres_options, to allow adding arbitrary PostgreSQL options without bailing out to sqlalchemy_url
  2. Allow setting application_name via tap options, perhaps via the above suggested option

@edgarrmondragon edgarrmondragon added the enhancement New feature or request label Jan 10, 2026
Comment thread tap_postgres/client.py
Comment thread tap_postgres/connection_parameters.py Outdated
Copy link
Copy Markdown
Member

@edgarrmondragon edgarrmondragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @danilofuchs! Just one small nit.

Comment thread tap_postgres/client.py
Comment thread tap_postgres/connection_parameters.py
Comment thread tap_postgres/connection_parameters.py Outdated
Copy link
Copy Markdown
Member

@edgarrmondragon edgarrmondragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @danilofuchs!

@edgarrmondragon edgarrmondragon self-assigned this Jan 13, 2026
@edgarrmondragon edgarrmondragon added this pull request to the merge queue Jan 13, 2026
Merged via the queue into MeltanoLabs:main with commit 9a3db1d Jan 13, 2026
17 of 20 checks passed
edgarrmondragon added a commit that referenced this pull request Jan 13, 2026
…nd port

- Bug introduced by #717

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
@edgarrmondragon
Copy link
Copy Markdown
Member

edgarrmondragon added a commit that referenced this pull request Jan 13, 2026
…nd port

- Bug introduced by #717

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
edgarrmondragon added a commit that referenced this pull request Jan 14, 2026
…nd port

- Bug introduced by #717

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
edgarrmondragon added a commit that referenced this pull request Jan 14, 2026
…and port

- Bug introduced by #717

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
edgarrmondragon added a commit that referenced this pull request Jan 14, 2026
…nd port (#721)

- I missed this when reviewing #717
- #469 

cc @danilofuchs

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Log Based Replication not working with SSH Tunnel

2 participants