Skip to content

Init agents context. Add read access logs script #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merteg merged 1 commit into from
Aug 13, 2026
Merged

Init agents context. Add read access logs script #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
174 changes: 174 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
# Delta Sharing – Agent Instructions

Virtana fork of [delta-io/delta-sharing](https://github.com/delta-io/delta-sharing), extending the reference server with GCS-backed telemetry and multi-environment Kubernetes deployments.

Deeper background lives in the [memory bank](memory-bank/README.md) — overview, build/test, Virtana divergence, deployment, and GCP egress pricing.

## Rules for Every Task

1. **Update [CHANGELOG-VIRTANA.md](CHANGELOG-VIRTANA.md).** Any change to behaviour, configuration, build setup, or deployment gets reflected in the relevant area section (add a new area if none fits). It's an informal running overview organized by area, not a dated/versioned log — no need to track release status. That file is the record of how this fork diverges from upstream.
2. **Run the tests** for whatever you touched (see [Build & Test](#build--test)). `server/test` runs scalastyle first and fails on style violations.
3. **Verify the image builds** when you change `build.sbt`, dependencies, or anything packaging-related:
```bash
DOCKER_DEFAULT_PLATFORM=linux/amd64 make image
```
The env var is required — the deployment target is amd64 and building on an arm64 Mac without it produces an unusable image.

## Branching

**`master` is the working branch** — do development and target PRs here. CI/CD builds and deploys from it.

`master` was forked from upstream [`delta-io/delta-sharing` `branch-1.3`](https://github.com/delta-io/delta-sharing/tree/branch-1.3) at commit `793cc19b2a3434227ebdc7f34bb2141047a67925`.

| Branch | Purpose |
|--------|---------|
| `master` | Working + release branch — CI/CD deploys from here |
| `virtana-1.3` | Historical fork branch |
| `main` | Upstream delta-io/delta-sharing — **not** the deployment branch |

Never target `main` for Virtana changes.

## Divergence from Upstream

See [CHANGELOG-VIRTANA.md](CHANGELOG-VIRTANA.md) for the full inventory of Virtana-modified and Virtana-authored files, plus the merge risks to watch when pulling from upstream. When fixing a bug, check that file first to tell whether the code is ours or upstream's.

## Build & Test

Use the wrapper script — plain `sbt` may not be in PATH:

```bash
./build/sbt server/compile # Scala 2.12 (server only)
./build/sbt client/compile # Cross-compiled 2.12 + 2.13
./build/sbt spark/compile # Scala 2.13 only

./build/sbt server/test # Runs scalastyle before tests
./build/sbt client/test
./build/sbt spark/test

# Image build — always pin the platform, the deploy target is amd64
DOCKER_DEFAULT_PLATFORM=linux/amd64 make image
DOCKER_DEFAULT_PLATFORM=linux/amd64 ./build/sbt server/docker:publishLocal
```

**Scala versions**: server → 2.12.18 + Spark 3.5.3 (Java 8); spark connector → 2.13.13 + Spark 4.0.0 (Java 17).

scalastyle runs as part of `server/test` and fails the build on violations. Config: [scalastyle-config.xml](scalastyle-config.xml). Use `org.scalatest.FunSuite` (not `AnyFunSuite`) for test suites.

Python tests:
```bash
python/dev/pytest
```

## Project Structure

```
server/src/main/scala/io/delta/sharing/server/ REST endpoints (Armeria), main entry point
server/src/main/scala/.../server/config/ ServerConfig, AccessLoggingConfig, Share/Table config
server/src/main/scala/.../server/common/ CloudFileSigner (GCS + S3), JsonPredicates
server/src/main/scala/.../server/telemetry/ Access log emission, GCP pricing tier, Delta writer
server/src/main/scala/.../kernel/ Delta Lake kernel integration
manifests/ Kustomize overlays per environment
ci/ Jenkins pipeline, deploy scripts
python/delta_sharing/ Python client library
```

## Server Configuration

The server takes `--config <path-to-yaml>`. Key settings (from [manifests/base/configmap.yaml](manifests/base/configmap.yaml)):

```yaml
host: "0.0.0.0"
port: 8080
endpoint: "/delta-sharing"
preSignedUrlTimeoutSeconds: 3600
deltaTableCacheSize: 100
evaluateJsonPredicateHints: true
evaluateJsonPredicateHintsV2: true
requestTimeoutSeconds: 180
idleTimeoutSeconds: 120 # Must exceed the proxy's IdleConnTimeout or clients see EOF errors
queryTablePageSizeLimit: 10000
perfLoggingEnabled: true

authorization:
bearerToken: "<token>" # Injected from $BEARER_TOKEN at deploy time

shares:
- name: "share_name"
schemas:
- name: "schema_name"
tables:
- name: "table_name"
location: "gs://bucket/path"
cdfEnabled: false
```

## GCS Integration

**Authentication**: Set `GOOGLE_APPLICATION_CREDENTIALS` to a service account JSON path. In Kubernetes, Workload Identity is used via the `dl-sharing` service account — no key file needed in production.

**Dependencies** (in [build.sbt](build.sbt)):
- `com.google.cloud:google-cloud-storage` — GCS SDK
- `com.google.cloud.bigdataoss:gcs-connector:hadoop2-2.2.4` — Hadoop FS integration

**GCS signing**: `server/src/main/scala/.../server/common/CloudFileSigner.scala` — generates pre-signed GCS URLs using `GoogleHadoopFileSystem` and `StorageResourceId`.

Use `gs://` table paths (not `s3://` or `s3a://`) for GCS-backed tables.

**GCS environments**:

| Environment | Bucket | Region |
|------------|--------|--------|
| zing-dev | `gs://zing-dev-197522-dl-v1/` | us-central1 |
| zing-preview | `gs://zing-preview-dl-v1/` | us-central1 |
| zcloud-prod | `gs://zcloud-prod-dl-v1/` | us-central1 |
| zcloud-prod2 | `gs://zcloud-prod2-dl-v1/` | us-west4 |
| zcloud-prod3 | `gs://zcloud-prod3-dl-v1/` | australia-southeast1 |
| zcloud-emea | GCS bucket | europe-west3 |

## Access Logging (Virtana Extension)

Virtana-added feature that writes structured access log entries to a Delta table on GCS after each query/CDF request. See [memory-bank/06-egress-monitoring.md](memory-bank/06-egress-monitoring.md) and [memory-bank/07-access-log-table-reference.md](memory-bank/07-access-log-table-reference.md).

**Config block**:
```yaml
accessLogging:
enabled: true
sourceRegion: "us-central1" # GCP region of this server's data bucket
detectGcpTraffic: true # Classify inter-GCP traffic for pricing tier
clientRegionHeader: "x-client-region"
clientIpHeader: "x-forwarded-for"
deltaTablePath: "gs://bucket/path/tenant/_system"
deltaFlushIntervalSeconds: 60
deltaFlushBatchSize: 1000
```

**Key classes**:
- `AccessLogEmitter` / `DeltaAccessLogWriter` — buffered async writer to GCS Delta table
- `GcpPricingTier` — classifies egress by region pair (e.g. `internet_to_na_eu`, `same_region`); refreshes GCP IP ranges from gstatic.com every 24h
- `GcpIpRangeLookup` — IP range → GCP region detection

**Delta table** (`access_log_br__system`): a single consolidated, unpartitioned table with `tenantId` as a column for per-tenant filtering. Protocol (1,2) for Delta Standalone compatibility; pre-created by `deltalake-admin` — the writer does not create the schema.

## Kubernetes Deployment

Manifests use Kustomize overlays at [manifests/](manifests/). Each environment overlay extends `manifests/base/`.

Deployment pattern:
- Init container merges base config + shares config using `envsubst` (substitutes `$BEARER_TOKEN`, `$GCP_PROJECT_ID`)
- Sidecar container `zc-api-proxy` handles JWT/Auth0 authentication in front of the sharing server on `localhost:8080`
- `dl-sharing` Kubernetes SA is bound to a GCP SA via Workload Identity

```bash
make deploy-dev # Deploy to zing-dev
make deploy-preview # Deploy to zing-preview
make deploy-prod # Deploy to zcloud-prod
```

## Environment Variables

| Variable | Where used |
|----------|-----------|
| `BEARER_TOKEN` | Kubernetes secret → injected into server config |
| `GCP_PROJECT_ID` | Kubernetes ConfigMap → injected into server config |
| `GOOGLE_APPLICATION_CREDENTIALS` | Local dev / test — path to service account JSON |
| `AZURE_TEST_ACCOUNT_KEY` | Optional — Azure blob storage integration tests |
68 changes: 68 additions & 0 deletions CHANGELOG-VIRTANA.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Virtana Changelog

Everything Virtana has changed on top of the upstream [delta-io/delta-sharing](https://github.com/delta-io/delta-sharing) reference server.

Fork point: upstream [`branch-1.3`](https://github.com/delta-io/delta-sharing/tree/branch-1.3) at commit `793cc19b2a3434227ebdc7f34bb2141047a67925`.

```bash
git diff --stat 793cc19b2a3434227ebdc7f34bb2141047a67925..HEAD
```

**Keep this file current.** This isn't a formal historical log — just a running, informal overview of
everything Virtana has changed in this fork, organized by area. Every task that changes behaviour,
configuration, build setup, or deployment updates the relevant area below (or adds a new one if it
doesn't fit). Upstream files not listed here are unmodified — when fixing a bug, first check whether
it lives in Virtana code or upstream code.

## Changes

### Telemetry package (largest addition)

`server/src/main/scala/io/delta/sharing/server/telemetry/`, entirely Virtana-authored:

| File | Purpose |
|------|---------|
| `AccessLogEmitter.scala` | `AccessLogEntry` / `PricingContextLogEntry` models; JSON emitter |
| `DeltaAccessLogWriter.scala` | Async buffered writer to the GCS Delta table |
| `GcpPricingTier.scala` | Egress cost classification by source→destination region pair |
| `GcpIpRangeLookup.scala` | GCP IP range → region detection (refreshed from `cloud.json`) |

Each has a matching suite under `server/src/test/scala/.../telemetry/`.

### Upstream server files, modified

| File | Virtana change |
|------|----------------|
| `DeltaSharingService.scala` | ~600 added lines: access log emission per query/CDF request, client region + IP header extraction, egress byte accounting, idle timeout config |
| `config/ServerConfig.scala` | New `AccessLoggingConfig` case class; new `perfLoggingEnabled` and `idleTimeoutSeconds` options |
| `DeltaSharedTableProtocol.scala` | New `CdfQueryTimings` / `TableQueryTimings` / `QueryResultTimings` observability models; `QueryResult` gained a `timings` field |
| `DeltaSharedTableLoader.scala` | `loadTableWithUpdateCost` returns `deltaLog.update()` elapsed time for perf logging |
| `standalone/internal/DeltaSharedTable.scala` | Per-phase timing instrumentation (snapshot resolve, replay, signing); near-timeout warnings |
| `standalone/internal/DeltaSharingCDCReader.scala` | CDF stream timing instrumentation |

### Build & infrastructure

| File | Virtana change |
|------|----------------|
| `build.sbt` | Global slf4j binding exclusions (`slf4j-log4j12`, `slf4j-reload4j`); per-dependency `ExclusionRule("org.slf4j")`; swapped `slf4j-simple` → `logback-classic`; `delta-standalone` changed from `provided` to a compile dependency (required by the Delta log writer); pinned `dockerBaseImage := "eclipse-temurin:8-jre"` |
| `server/src/main/resources/logback.xml` | New — preserves log severity in GCP Cloud Console |
| `scalastyle-config.xml` | License header regex relaxed to allow any copyright year |

### Deployment & docs

- `manifests/` — Kustomize base + 6 environment overlays
- `ci/` — Jenkinsfile, Makefile, `deploy.sh`, per-environment deployment YAML
- `Makefile` — image build and `deploy-*` targets
- [memory-bank/](memory-bank/README.md) — committed shared context for humans and agents: overview,
build/test, Virtana divergence, deployment, and GCP egress pricing reference. Linked from
[AGENTS.md](AGENTS.md). Replaces `docs/PER_SHARE_EGRESS_MONITORING.md` (now
`memory-bank/06-egress-monitoring.md`) and `docs/Notes.md` (now
`memory-bank/07-access-log-table-reference.md`); the old `docs/` is gone. The access log table
description was corrected there: `access_log_br__system` is unpartitioned (matches
`DeltaAccessLogWriter`), not partitioned by `year`/`month`/`day`.

## Known divergence risks when merging upstream

- `DeltaSharingService.scala` is heavily modified — expect conflicts on any upstream change there.
- The `delta-standalone` scope change (`provided` → compile) must be preserved or `DeltaAccessLogWriter` fails at runtime.
- slf4j exclusions must be preserved or the server emits "multiple SLF4J bindings" and loses Cloud Console severity.
7 changes: 4 additions & 3 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,16 @@
REPO_ROOT := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
SERVER_VERSION := $(shell grep 'version in ThisBuild' $(REPO_ROOT)version.sbt | sed 's/.*"\(.*\)".*/\1/')
SERVICE_IMAGE := $(shell grep '^SERVICE_IMAGE' $(REPO_ROOT).env | cut -d'=' -f2)
# Resolve IMAGE_TAG from ci/.env (used for push and deploy)
# IMAGE_TAG := $(shell grep '^IMAGE_TAG' $(REPO_ROOT)ci/.env | cut -d':' -f2 | tr -d ' =')
# IMAGE_TAG is normally supplied by the caller (e.g. Jenkins export). ?= leaves an
# already-set/exported value alone and only supplies "dev" for local builds -
IMAGE_TAG ?= dev

.PHONY: image
image:
@echo "Building Docker image from fork (version $(SERVER_VERSION))"
@cd $(REPO_ROOT) && build/sbt server/docker:publishLocal
@echo "Image built: deltaio/delta-sharing-server:$(SERVER_VERSION)"
@docker tag deltaio/delta-sharing-server:$(SERVER_VERSION) ${SERVICE_IMAGE}:${IMAGE_TAG:-dev}
@docker tag deltaio/delta-sharing-server:$(SERVER_VERSION) ${SERVICE_IMAGE}:${IMAGE_TAG}

.PHONY: push-dev
push-dev:
Expand Down
114 changes: 114 additions & 0 deletions dev/READ_AL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Read Access Logs

Query Delta Sharing access logs via Delta Sharing protocol or directly from GCS.

Access logs are stored in per-tenant Delta tables named `access_log_{tenant_id}`, where
`tenant_id` is extracted from the share name (pattern: `{tenant_id}_share`).

## Installation

```bash
# For direct GCS access (recommended)
pip3 install deltalake pandas pyarrow gcsfs

# For Delta Sharing mode
pip3 install delta-sharing pandas
```

## Modes

### Direct GCS Mode (`--direct`)

Reads directly from GCS bucket. Requires `gcloud auth application-default login`.

```bash
# Read _system tenant logs (default)
python3 read_access_logs.py --direct --env zing-dev

# Read specific tenant logs
python3 read_access_logs.py --direct --env zing-dev --tenant ipa7l25ufagwjfmv

# Filter last 7 days
python3 read_access_logs.py --direct --env zing-dev --tenant _system --days 7

# Filter by share name
python3 read_access_logs.py --direct --env zing-dev --tenant mytenantid --share myshare

# Combine filters
python3 read_access_logs.py --direct --env zcloud-prod --tenant _system --days 30 --limit 100
```

### Delta Sharing Mode (default)

Requires `_system_share` to be configured on the server.

```bash
# Using default profile
python3 read_access_logs.py --profile profile-dev.json

# List available shares/tables
python3 read_access_logs.py --profile profile-dev.json --list-tables

# Custom table URL
python3 read_access_logs.py --table-url "profile-prod.json#_system_share.SystemData_v0_1.access_log__system"
```

## Per-Tenant Table Structure

| Share accessed | Table name | GCS path |
|----------------|------------|----------|
| `_system_share` | `access_log__system` | `gs://.../tenant/_system/access_log__system` |
| `ipa7l25ufagwjfmv_share` | `access_log_ipa7l25ufagwjfmv` | `gs://.../tenant/_system/access_log_ipa7l25ufagwjfmv` |
| `hhgp5t6oz3nvczk7_share` | `access_log_hhgp5t6oz3nvczk7` | `gs://.../tenant/_system/access_log_hhgp5t6oz3nvczk7` |

## Environments

| Environment | GCS Base Path |
|-----------------|----------------------------------------------------|
| `zing-dev` | gs://zing-dev-197522-dl-v1/datalake/data/tenant/_system |
| `zing-preview` | gs://zing-preview-dl-v1/datalake/data/tenant/_system |
| `zcloud-prod` | gs://zcloud-prod-dl-v1/datalake/data/tenant/_system |
| `zcloud-prod2` | gs://zcloud-prod2-dl-v1/datalake/data/tenant/_system |
| `zcloud-prod3` | gs://zcloud-prod3-dl-v1/datalake/data/tenant/_system |

## Output Formats

```bash
# Summary with stats (default)
python3 read_access_logs.py --direct --env zing-dev

# Full table
python3 read_access_logs.py --direct --env zing-dev --output table

# Export to CSV
python3 read_access_logs.py --direct --env zcloud-prod --output csv --output-file logs.csv

# Export to JSON
python3 read_access_logs.py --direct --env zing-preview --limit 100 --output json --output-file logs.json
```

## Options Reference

| Option | Description |
|------------------|--------------------------------------------------|
| `--direct` | Read directly from GCS (requires gcloud auth) |
| `--env` | Environment for direct mode |
| `--tenant` | Tenant ID for direct mode (default: `_system`) |
| `--profile` | Delta Sharing profile file |
| `--table-url` | Full Delta Sharing table URL |
| `--list-tables` | List available shares/schemas/tables |
| `--days N` | Filter to last N days |
| `--share NAME` | Filter by share name |
| `--limit N` | Limit number of records |
| `--output` | Format: `summary`, `table`, `csv`, `json` |
| `--output-file` | Write output to file (for csv/json) |

## Output Columns

- `timestampMs` - Request timestamp (milliseconds)
- `share` - Share name accessed
- `schema` - Schema name
- `table` - Table name
- `egressBytes` - Bytes transferred
- `pricingTier` - GCS pricing tier
- `clientRegion` - Client's region
Loading
Loading