Skip to content

[#10669] feat(iceberg-rest): Add pagination support for list endpoints#10671

Open
laserninja wants to merge 1 commit intoapache:mainfrom
laserninja:fix/10669-iceberg-rest-pagination-support
Open

[#10669] feat(iceberg-rest): Add pagination support for list endpoints#10671
laserninja wants to merge 1 commit intoapache:mainfrom
laserninja:fix/10669-iceberg-rest-pagination-support

Conversation

@laserninja
Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

Add pageToken and pageSize query parameter support to the Iceberg REST server's listNamespaces, listTables, and listViews endpoints, threading them through the full call chain (REST → Event/Hook Dispatchers → Executors → CatalogWrapper).

Pagination is implemented as offset-based in-memory pagination in IcebergCatalogWrapper. When pageSize is provided, results are sliced and nextPageToken is populated with the offset of the next batch. When no pagination parameters are provided, existing behavior (return all results) is preserved.

Why are the changes needed?

The Iceberg REST spec defines pageToken and pageSize query parameters on list endpoints, but Gravitino's Iceberg REST server did not accept or handle them. Clients with large catalogs cannot paginate results — everything was returned in a single response.

Fix: #10669

Does this PR introduce any user-facing change?

Yes. The following REST endpoints now accept optional pageToken and pageSize query parameters:

  • GET /v1/{prefix}/namespaces
  • GET /v1/{prefix}/namespaces/{namespace}/tables
  • GET /v1/{prefix}/namespaces/{namespace}/views

How was this patch tested?

  • Added testListNamespaceWithPagination in TestIcebergNamespaceOperations
  • Added testListTablesWithPagination in TestIcebergTableOperations
  • Added testListViewsWithPagination in TestIcebergViewOperations
  • Updated existing tests in TestIcebergViewHookDispatcher and TestIcebergViewOperationExecutor
  • All existing unit tests pass (./gradlew :iceberg:iceberg-rest-server:test -PskipITs)

@laserninja laserninja requested review from bharos and roryqi April 18, 2026 23:50
@roryqi
Copy link
Copy Markdown
Contributor

roryqi commented Apr 19, 2026

  1. Iceberg REST server will use Hive backend, too. How does Hive backend support paging? Do we need to consider the backend? How did other Iceberg REST like Polaris, Lakekeeper, UnityCatalog to support paging operations?
  2. Could u resolve the conflicts?

@laserninja
Copy link
Copy Markdown
Collaborator Author

Thanks for checking, @roryqi!

1. Backend considerations & how other Iceberg REST catalogs handle pagination:

The Iceberg Catalog Java API (listNamespaces(), listTables(), listViews()) returns complete List results with no built-in pagination support — this is true regardless of the backend (Hive, JDBC, etc.). The Hive metastore APIs (getAllDatabases, getAllTables) also return all results at once without native pagination. So in-memory pagination at the REST layer is the only practical approach here.

This is consistent with how other Iceberg REST catalogs handle it:

  • Polaris went through an extensive pagination effort (PRs #1528, #1938). Their initial implementation also did in-memory pagination at the REST/service layer. They later worked on pushing pagination down to the persistence layer, but that was for Polaris's own entity management — the Iceberg catalog operations themselves (via CatalogHandlers.listNamespaces) still use in-memory listing. They also noted this is a limitation when federating to external catalogs.
  • Lakekeeper and UnityCatalog similarly handle pageToken/pageSize at the service layer.

Since Gravitino's IcebergCatalogWrapper wraps the standard Iceberg Catalog interface, in-memory pagination is the appropriate approach at this layer. If backend-native pagination is desired in the future, it would require changes to the Iceberg Catalog interface itself.

2. Will resolve the conflicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Missing pagination support on Iceberg REST list endpoints (listNamespaces, listTables, listViews)

2 participants