Skip to content

[Feature] [python] Support query auth (row filter & column masking) for REST catalog #8135

@MgjLLL

Description

@MgjLLL

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Apache Paimon's Java REST catalog already supports server-side query authorization, where the catalog can return per-query row-level filters and column masking rules so that fine-grained access control is enforced at read time. The Python client (pypaimon) currently has no equivalent: when reading via REST catalog, any filter / columnMasking returned by the server is ignored, so RBAC / column-level security policies are not honored from Python.
This issue tracks adding query-auth support to pypaimon so that Python readers behave consistently with the JVM client when the catalog enables query auth.

Solution

Introduce an opt-in query-auth.enabled option (default false). When enabled, before building a plan the client calls a new REST endpoint:

  • POST /v1/{prefix}/databases/{db}/tables/{tb}/auth
  • request: { "select": [] }
  • response: { "filter": [...], "columnMasking": { "": , ... } }

Then, on the read path:

  1. The returned predicate JSON list is parsed into a PyArrow compute filter and applied via an AuthFilterReader wrapped around the inner record-batch reader.
  2. The returned columnMasking map (column → transform JSON) is applied via an AuthMaskingReader, supporting transforms NULL / FIELD_REF / CAST / UPPER / LOWER / CONCAT / CONCAT_WS.
  3. A ColumnProjectReader projects the result back to the columns the user originally requested (extra fields pulled in only to satisfy the auth filter are dropped).

The contract matches the Java side, so no server-side change is required for catalogs that already implement query auth.

Anything else?

  • Wire-format compatibility is the main risk surface. Tests cover the predicate / transform JSON shapes against representative cases.
  • Backwards compatible by default: with query-auth.enabled=false (the default), behavior is unchanged.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions