Skip to content

chore: allowlist Arrow IPC base64 false positives in .gitleaksignore#408

Merged
msrathore-db merged 1 commit into
mainfrom
mani/gitleaksignore-arrow-fixtures
Jul 16, 2026
Merged

chore: allowlist Arrow IPC base64 false positives in .gitleaksignore#408
msrathore-db merged 1 commit into
mainfrom
mani/gitleaksignore-arrow-fixtures

Conversation

@msrathore-db

Copy link
Copy Markdown
Contributor

Problem

The Databricks global pre-push secret scanner runs gitleaks detect over each new branch's commit range. The aws-access-token rule (which targets AKIA…/ASIA…-style keys) collides with high-entropy base64 substrings inside the Apache Arrow IPC payloads stored in the internal/rows/arrowbased/testdata fixtures ("batch": "/////5gCAAAU…").

This produces 24 false positives across 8 fixture files (diamonds.json, all_types*.json, arrays_native.json, structs_native.json). They are binary Arrow result data, not credentials — entropy ~1.5 vs ~4.3+ for a real AWS key.

All were introduced by f27a47b ("Fetch results in arrow format", #109). Because gitleaks detect fingerprints are commit-SHA-prefixed and the scan walks back through history, these re-surface on every new branch — even branches that never touch these files — forcing contributors to push with SKIP_SECRET_SCAN=1.

Fix

Commit a repo-root .gitleaksignore listing the 24 SHA-prefixed fingerprints. gitleaks detect reads this file automatically, so the false positives are suppressed permanently for all branches with no per-push workaround. This mirrors the approach already used in databricks-driver-test.

Verification

Re-running the exact pre-push scan against the introducing commit with the file present:

gitleaks detect -c gitleaks.toml --exit-code 3 --log-opts="-p -1 f27a47b…"
→ no leaks found (exit 0)

The push of this branch itself passed the pre-push secret scan with no block.

This pull request and its description were written by Isaac.

The Databricks global pre-push secret scanner runs `gitleaks detect` over
each new branch's commit range. The `aws-access-token` rule (which targets
AKIA…/ASIA…-style keys) collides with high-entropy base64 substrings inside
the Apache Arrow IPC payloads stored in the arrowbased testdata fixtures
("batch": "/////..."), producing 24 false positives across 8 files.

These are binary result data, not credentials (entropy ~1.5 vs ~4.3+ for a
real key). All were introduced by f27a47b ("Fetch results in arrow format",
#109), so they re-surface on every new branch whose scan range walks back
through history and re-includes that commit — even for commits that never
touch these files, forcing SKIP_SECRET_SCAN=1 on unrelated pushes.

Committing a repo-root .gitleaksignore (the same fix databricks-driver-test
uses) suppresses them permanently for all branches.

Co-authored-by: Isaac
Signed-off-by: Madhavendra Rathore <madhavendra.rathore@databricks.com>
@msrathore-db msrathore-db force-pushed the mani/gitleaksignore-arrow-fixtures branch from f14d216 to 3d81412 Compare July 16, 2026 12:14
@msrathore-db msrathore-db merged commit dbba827 into main Jul 16, 2026
11 checks passed
@msrathore-db msrathore-db deleted the mani/gitleaksignore-arrow-fixtures branch July 16, 2026 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants