Skip to content

docs: add BigLake Iceberg metastore integration guide #2325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
GuyEshdat wants to merge 1 commit into
base: docs
Choose a base branch
from
Open

docs: add BigLake Iceberg metastore integration guide #2325

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions docs/cloud/integrations/metadata-layer/biglake.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
---
title: "Google BigLake (Iceberg)"
badge: "Context Engine"
---

The Google BigLake integration in Elementary allows you to automatically monitor volume & freshness anomalies in your Iceberg tables managed by a BigLake Metastore catalog, by continuously syncing metadata about Iceberg snapshots.

<Note>
BigLake Iceberg is only supported for BigQuery environments. The same service account used for your BigQuery warehouse connection is reused — no separate credentials are required.
</Note>

## GCP Setup

### 1. Grant additional IAM roles

Elementary reuses the service account that is already connected to your BigQuery environment. You only need to grant three additional roles to that service account:

**On the GCP project that owns the BigLake catalog:**

| Role | Purpose |
|---|---|
| `roles/biglake.metadataViewer` | Allows listing namespaces and tables in the BigLake Metastore catalog |
| `roles/serviceusage.serviceUsageConsumer` | Required for the BigLake REST API to attribute quota usage to your project |
| `roles/storage.objectViewer` (with condition — explained below) | Grants read access to Iceberg metadata files in GCS |

### 2. Scope the Storage Object Viewer role to metadata files only

For security, restrict the `Storage Object Viewer` role to Iceberg metadata files rather than granting access to all objects in the bucket.

When assigning the role on the GCS bucket, add a **condition**:

```json
{
"expression": "resource.name.endsWith(\".metadata.json\")",
"title": "Iceberg metadata files only"
}
```

This ensures the service account can only read `*.metadata.json` files — the Iceberg snapshot metadata — and nothing else in the bucket.

## Elementary Configuration

Navigate to **Account settings > Environments**, choose your BigQuery environment, and scroll to the **Metadata Layer** section. Select **BigLake**.

### Connection Settings

| Field | Required | Description |
|---|---|---|
| **Connection Name** | Yes | A descriptive name for this connection (e.g. "Data Lake"). Must be unique if adding multiple integrations. |
| **Catalog name** | Yes | The name of your BigLake Metastore catalog. |
| **GCP project** | No | Only required if the BigLake catalog is in a different GCP project than your connected BigQuery warehouse. |
| **Catalog location** | No | Only required if the BigLake catalog is in a different GCP region than your connected BigQuery dataset. |
3 changes: 2 additions & 1 deletion docs/docs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -304,7 +304,8 @@
{
"group": "Iceberg catalog",
"pages": [
"cloud/integrations/metadata-layer/glue"
"cloud/integrations/metadata-layer/glue",
"cloud/integrations/metadata-layer/biglake"
]
},
{
Expand Down
Loading