feat(cubestore): GCS Workload Identity Federation (WIF/ADC) support

[KrishnaRMaddikara]

Problem

CubeStore's GCS remote storage implementation uses the cloud_storage crate,
which requires SERVICE_ACCOUNT or SERVICE_ACCOUNT_JSON environment variables
and panics when they are absent. This makes CubeStore completely non-functional
on GKE clusters that use Workload Identity Federation (WIF) — the recommended
GCP authentication model for Kubernetes workloads — where no service account key
file is needed or available.

Affects:

• All GKE users with Workload Identity configured (issues Support Workload Identity Federation for GCS without requiring explicit credentials in cube/cubestore Docker image #9837, Can router and workers can support GKE Workload Identity ? #7279)
• Any deployment using GOOGLE_APPLICATION_CREDENTIALS (ADC file path)
• Dev machines using gcloud auth application-default login

Solution

Replace cloud_storage with object_store crate for GCS operations.
object_store is already a transitive dependency of CubeStore via Apache
Arrow/DataFusion, so this adds no new dependencies.

GoogleCloudStorageBuilder::from_env() supports the full ADC credential chain
in priority order:

1. CUBESTORE_GCP_KEY_FILE — existing env var, backward compatible
2. CUBESTORE_GCP_CREDENTIALS — existing env var, backward compatible
3. GKE Workload Identity via metadata server at 169.254.169.254 ← main fix
4. GOOGLE_APPLICATION_CREDENTIALS file path (standard ADC)
5. gcloud CLI credentials (developer machines)

Backward Compatibility

Backward compatible for existing CUBESTORE_GCP_KEY_FILE,
CUBESTORE_GCP_CREDENTIALS, and common ADC-based environments.
Legacy JSON aliases (SERVICE_ACCOUNT_JSON,
GOOGLE_APPLICATION_CREDENTIALS_JSON) are mapped with deprecation warnings.

Testing

Tested on GKE with:

• Workload Identity binding: serviceAccount@project.iam.gserviceaccount.com
• IAM role: roles/storage.objectAdmin on the export bucket
• CubeStore successfully reads/writes pre-aggregation files without any key file

Changes in v2 (follow-up commit)

• Upload/download changed from full in-memory buffer to streaming
  (prevents OOM on workers with large pre-aggregation files)
• Legacy env var aliases mapped explicitly for full backward compatibility
• Fixed page count ceiling division
• Fixed misleading GCS consistency comment

Addressed all review feedback — pushed 2 follow-up commits:

Commit 2 (fix: streaming + legacy aliases):

• Upload: replaced fs::read() full-buffer with PutPayload::from_stream() — prevents OOM on large pre-aggregation files
• Download: replaced bytes().await full-buffer with chunked into_stream() write to disk
• Added explicit mapping for legacy SERVICE_ACCOUNT_JSON / GOOGLE_APPLICATION_CREDENTIALS_JSON aliases with deprecation warnings
• Fixed page count ceiling division: (len + 999) / 1000
• Fixed head() comment — workaround is for object_store path normalization, not GCS consistency

Commit 3 (fix: compile blockers):

• Removed stale second store.put(data) call in upload_file (leftover from old buffered implementation, data was undefined)
• Removed get_result.bytes().await before into_stream() — both consume GetResult, only streaming path remains
• Softened "GCS consistency error" wording to "post-upload verification"

PR description updated: "Fully backward compatible" → scoped to commonly used credential env vars.

Closes #9837
Closes #7279

[KrishnaRMaddikara]

Both blockers are resolved in commit 59e7538:

• upload_file: removed the stale second store.put(data) call —
  only the streaming put(&obj_path, payload) path remains
• download_file: removed get_result.bytes().await entirely —
  only get_result.into_stream() is used now
• check_upload_file error message changed to
  "not found during post-upload verification"
• PR description updated: softened "Fully backward compatible" to
  "backward compatible for existing CUBESTORE_GCP_KEY_FILE,
  CUBESTORE_GCP_CREDENTIALS, and common ADC-based environments"

Ready for review.

[KrishnaRMaddikara]

I’ve addressed the follow-up review points in the latest commits:
• removed the stale duplicate put(...) path in upload_file
• removed get_result.bytes().await; download_file now uses only streaming via into_stream()
• changed the post-upload verification message to neutral wording
• improved compatibility handling and updated the PR description wording

Latest commits on the PR branch:
• 59e7538 fix(cubestore/gcs): remove stale second put, fix double-consume of GetResult
• aa6b4c1 fix(cubestore/gcs): streaming transfers, legacy env aliases, page count, comments

This should now be ready for another review.