403 Errors with GCS Bazel Cache when using GitHub Action on VM inside GCP

[alsutton]

Describe the bug

We have a GitHub Action workflow for our CI checks, and the runners are self hosted, and split between a mac "rent-a-machine" company and GCP. As part of our checks we use the google-github-actions/auth@v3 action to authenticate each run to Google and obtain a credentials file, the location of which is passed to subsequence steps via the well known environment variable GOOGLE_APPLICATION_CREDENTIALS.

Subsequent steps, that use gcloud storage cp are able to copy data, from both the macs and GCP VMs, to the GCS bucket that we use for our Bazel cache, which, to us, indicates the authentication has succeeded, and valid credentials are available, and that the GCP VMs have all the permissions needed to write to the bucket.

When we run a bazel build, however, we see numerous HTTP 403 errors stating;

<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Provided scope(s) are not authorized</Details></Error>

which appears to correlate with when we would expect cache writes to be sent to the bucket.

To confirm the problem is with this credential helper we replaced it with a two step process; Create a bearer token using gcloud auth print-access-token, which is stored in an environment variable, and then the script returns the appropriately formatted response, with that token, to Bazel, in place of the credential helper. When we do this the 403 errors do not occur.

We have also tried running this helper in standalone mode, by setting CREDENTIAL_HELPER_STANDALONE to 1 in the environment. This did not resolve the issue.

To Reproduce

Setup a GCS bucket as the bazel remote cache.

Setup a GHA workflow that runs on GCP hosted virtual machines, that perform the auth step mentioned above.

Run workflow.

Expected behavior

No 403 errors, and cache writes succeeding.

Environment

• Ubuntu 24.04 on X86_64 and ARM.
• 0.0.9

Additional context
Seems to be a GCP VM related issue. GHA may, or may not, be a relevant factor.

[malt3]

Here is how we configure Google with GitHub Actions:

credential-helper/.github/workflows/ci.yml

Line 65 in 4ea7815

uses: google-github-actions/auth@v1

Maybe some of those options are relevant?
I remember that Google differentiates between gcloud credentials and Applications Default Credentials. The latter is what tweag-credential-helper consumes, while the Google CLI may give you the former.

[alsutton]

Here's our snippet;

      - uses: "google-github-actions/auth@v3"
        id: "gcp-auth"
        with:
          credentials_json: "${{ secrets.GCP_SERVICE_ACCOUNT_KEYS_JSON }}"
          create_credentials_file: true
          token_format: access_token
          access_token_scopes: https://www.googleapis.com/auth/cloud-platform

The script, which works, does this;

cat << EOF
{
  "headers": {
    "Authorization": ["Bearer ${CACHED_BEARER_TOKEN}"]
  }
}
EOF

and we run Bazel like this;

      - name: Build
        run: |
          bazel build //... \
            --config=ci
        env:
          CACHED_BEARER_TOKEN: ${{ steps.gcp-auth.outputs.access_token }}

(where the ci config in .bazelrc tells bazel to use the script instead of this helper)

I think the problem is due to the self hosted VM running in GCP, and so isn't something that a GitHub Actions hosted runner would replicate. GCP VMs have a their own, machine level, gcloud service account, which is permanently logged into gcloud, so gcloud auth list gives you;

 * {VM-Service-Account}

and so the account authenticated into the workflow becomes a second gcloud account.

Our developers, who are using macs' with their own gcloud accounts, don't see the problem, which is why I'm strongly confident the only place you'll replicate it is with a GCP VM self hosted runner.

[malt3]

I think the problem is due to the self hosted VM running in GCP, and so isn't something that a GitHub Actions hosted runner would replicate. GCP VMs have a their own, machine level, gcloud service account, which is permanently logged into gcloud, so gcloud auth list gives you;

That's a good observation, and I agree that this could have something to do with the problem you are experiencing. Sadly, that also makes it difficult to replicate on my end.

According to Google's docs, while the standard auth mechanism for "Application Default Credentials" does take service account attached to a GCP VM into account, it should be tried last:

Search order
ADC searches for credentials in the following locations:

• GOOGLE_APPLICATION_CREDENTIALS environment variable
• A credential file created by using the gcloud auth application-default login command
• The attached service account, returned by the metadata server

(source)

I'd go so far as to say that this is potentially a bug in the Go package golang.org/x/oauth2/google which we use for this purpose.

[alsutton]

Do you have any thoughts on producing an isolated test case that mimics the credential helper behaviour.

I can stand up a GCP VM as a test area of you have a way of reproducing the potentially problematic flow.