secret-manager v6 + google-auth-library v10 + gaxios v7: Could not refresh access token in Cloud Build

[IchordeDionysos]

Library Name

google-auth-library

Environment

• @google-cloud/secret-manager: v6.1.2
• google-gax: v5.0.6
• google-auth-library: v10.6.2
• gaxios: v7.1.4 (uses node-fetch v3.3.2)
• Node.js: 20
• Runtime: Google Cloud Build with custom service account
• Container image: Custom Node 20 Docker image

Description

After upgrading @google-cloud/secret-manager from v5 to v6, we get the following error when SecretManagerServiceClient attempts to access the GCE metadata server for token refresh inside a Cloud Build step:

2 UNKNOWN: Getting metadata from plugin failed with error: Could not refresh access token: Unexpected Gaxios Error
    at callErrorFromStatus (node_modules/@grpc/grpc-js/src/call.ts:84:17)
    at Object.onReceiveStatus (node_modules/@grpc/grpc-js/src/client.ts:360:55)
    ...
    at node_modules/@google-cloud/secret-manager/build/src/v1/secret_manager_service_client.js:227:29
    at node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
    at repeat (node_modules/google-gax/build/src/normalCalls/retries.js:114:25)

The key difference between the working (v5) and broken (v6) dependency chains:

• v5 (works): secret-manager → google-gax v4 → google-auth-library v9 → gaxios v6 → node-fetch v2
• v6 (fails): secret-manager → google-gax v5 → google-auth-library v10 → gaxios v7 → node-fetch v3

This is related to the previously reported #6276, which was closed without resolution.

Workaround

Pinning @google-cloud/secret-manager to v5 (which pulls google-gax v4 → google-auth-library v9 → gaxios v6 → node-fetch v2) resolves the issue.

Expected behavior

google-auth-library v10 should be able to reach the GCE metadata server and refresh tokens in Cloud Build environments, just as v9 does.

Hypothesis

The core issue seems to be:

TypeError: A dynamic import callback was invoked without --experimental-vm-modules
Cause:
  ...
  at Function.#getFetch (node_modules/gcp-metadata/node_modules/gaxios/src/gaxios.ts:675:10)

[IchordeDionysos]

Been investigating the issue a bit more and after some digging the root cause turned out to be Jest's VM runtime rejecting the dynamic import('node-fetch') that gaxios@^7 does in Gaxios.#getFetch, not anything network- or token-related.

Posting a minimal, fully portable reproduction here in case it's useful.

Root cause

gaxios@^7 lazy-loads node-fetch via:

// node_modules/gaxios/src/gaxios.ts (v7)
static async #getFetch() {
  const hasWindow = typeof window !== 'undefined' && !!window;
  this.#fetch ||= hasWindow
    ? window.fetch
    : (await import('node-fetch')).default;
  return this.#fetch;
}

When Jest runs without NODE_OPTIONS=--experimental-vm-modules, its per-file vm.SourceTextModule runtime has no dynamic-import callback registered, so await import('node-fetch') throws:

TypeError: A dynamic import callback was invoked without --experimental-vm-modules

The TypeError originates in Node's VM runtime (a different realm), so by the time it reaches gaxios' catch block, e instanceof Error returns false and gaxios falls through to:

} else {
  err = new GaxiosError('Unexpected Gaxios Error', opts, undefined, e);
}

That's where the misleading Unexpected Gaxios Error / Could not refresh access token message originates. The actual TypeError is preserved as err.cause and you can see it in the stack trace.

Any code path that does a token refresh through gcp-metadata → google-auth-library@^10 → gaxios@^7 is affected. With the current dependency graph, that includes:

• @google-cloud/secret-manager@^6 → google-gax@^5 → google-auth-library@^10 → gaxios@^7
• @google-cloud/bigquery@^8 → google-gax@^5 → google-auth-library@^10 → gaxios@^7
• firebase-admin@^13.10 → google-auth-library@^10 → gaxios@^7

Minimal reproduction

Single Cloud Build config, runs the same Jest test twice — once without the flag (bug), once with (fix).

`cloudbuild-reproduce.yaml`:

timeout: 600s
options:
  logging: CLOUD_LOGGING_ONLY
  env:
    - 'PROJECT_ID=$PROJECT_ID'
steps:
  - name: 'node:20'
    id: 'install'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        set -e
        corepack enable

        cat > /workspace/package.json << 'PKGJSON'
        {
          "name": "gaxios-vm-context-repro",
          "private": true,
          "packageManager": "yarn@4.10.3",
          "dependencies": {
            "@google-cloud/secret-manager": "^6.1.2",
            "@google-cloud/bigquery": "^8.3.1",
            "firebase-admin": "^13.10.0",
            "jest": "^29.7.0",
            "ts-jest": "^29.3.2",
            "typescript": "^5.8.3",
            "@types/jest": "^29.5.14",
            "@types/node": "^20.19.41"
          }
        }
        PKGJSON

        cat > /workspace/.yarnrc.yml << 'YARNRC'
        nodeLinker: node-modules
        enableScripts: false
        YARNRC

        cd /workspace
        CI=false yarn install

        cat > /workspace/jest.config.js << 'JESTCFG'
        module.exports = {
          roots: ['<rootDir>/test/'],
          preset: 'ts-jest',
          testEnvironment: 'node',
          transform: {
            '^.+\\.(ts|tsx)$': ['ts-jest', {isolatedModules: true}],
          },
          testTimeout: 2 * 60 * 1000,
        };
        JESTCFG

        cat > /workspace/tsconfig.json << 'TSCFG'
        {
          "compilerOptions": {
            "target": "ES2020",
            "module": "commonjs",
            "esModuleInterop": true,
            "skipLibCheck": true
          }
        }
        TSCFG

        mkdir -p /workspace/test
        cat > /workspace/test/gaxios-repro.test.ts << 'TESTFILE'
        import {SecretManagerServiceClient} from '@google-cloud/secret-manager';
        import {BigQuery} from '@google-cloud/bigquery';

        const PROJECT_ID = process.env.PROJECT_ID || (() => {
          throw new Error('PROJECT_ID is not set');
        })();

        let secretClient: SecretManagerServiceClient;
        let bqClient: BigQuery;

        describe('gaxios v7 + Jest VM context', () => {
          beforeAll(() => {
            secretClient = new SecretManagerServiceClient({projectId: PROJECT_ID});
            bqClient = new BigQuery({projectId: PROJECT_ID});
          });

          it('Secret Manager v6 listSecrets (gaxios v7 surface)', async () => {
            const iterable = secretClient.listSecretsAsync({
              parent: `projects/$${PROJECT_ID}`,
              pageSize: 1,
            });
            for await (const _ of iterable) {
              break;
            }
          });

          it('BigQuery v8 query (gaxios v7 surface)', async () => {
            const [rows] = await bqClient.query({
              query: 'SELECT CURRENT_TIMESTAMP() as ts',
            });
            expect(rows.length).toBe(1);
          });
        });
        TESTFILE

  # ARM A: WITHOUT --experimental-vm-modules → expected to FAIL with the gaxios error.
  - name: 'node:20'
    id: 'bug-no-flag'
    waitFor: ['install']
    allowFailure: true
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        set +e
        cd /workspace
        echo "=== ARM A: WITHOUT --experimental-vm-modules (expecting FAILURE) ==="
        ./node_modules/.bin/jest --runInBand --config jest.config.js --forceExit
        echo "=== ARM A exit_code=$? (non-zero = bug reproduced) ==="

  # ARM B: WITH --experimental-vm-modules → token refresh now succeeds.
  - name: 'node:20'
    id: 'fix-with-flag'
    waitFor: ['bug-no-flag']
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        set -e
        cd /workspace
        echo "=== ARM B: WITH --experimental-vm-modules (expecting SUCCESS) ==="
        NODE_OPTIONS="--experimental-vm-modules" ./node_modules/.bin/jest --runInBand --config jest.config.js --forceExit

Submit it with:

gcloud builds submit --no-source \
  --config=cloudbuild-reproduce.yaml \
  --project=<your-project-id> \
  --region=<your-region>

Observed output

ARM A — bug-no-flag (gaxios bug reproduced):

=== ARM A: WITHOUT --experimental-vm-modules (expecting FAILURE) ===
FAIL test/gaxios-repro.test.ts
  gaxios v7 + Jest VM context
    ✕ Secret Manager v6 listSecrets (gaxios v7 surface)
    ✕ BigQuery v8 query (gaxios v7 surface)

  ● gaxios v7 + Jest VM context › Secret Manager v6 listSecrets (gaxios v7 surface)
    2 UNKNOWN: Getting metadata from plugin failed with error:
    Could not refresh access token: Unexpected Gaxios Error

  ● gaxios v7 + Jest VM context › BigQuery v8 query (gaxios v7 surface)
    Could not refresh access token: Unexpected Gaxios Error
      at Gaxios._request (node_modules/gcp-metadata/node_modules/gaxios/src/gaxios.ts:229:15)
      at metadataAccessor (node_modules/gcp-metadata/src/index.ts:178:15)
      at Compute.refreshTokenNoCache (node_modules/google-auth-library/build/src/auth/computeclient.js:57:20)
      ...

    Cause:
    TypeError: A dynamic import callback was invoked without --experimental-vm-modules
      at Function.#getFetch (node_modules/gcp-metadata/node_modules/gaxios/src/gaxios.ts:675:10)
      at Gaxios._defaultAdapter (node_modules/gcp-metadata/node_modules/gaxios/src/gaxios.ts:153:30)
      at Gaxios._request (node_modules/gcp-metadata/node_modules/gaxios/src/gaxios.ts:194:41)
      ...
=== ARM A exit_code=1 (non-zero = bug reproduced) ===

ARM B — fix-with-flag (gaxios bug gone, token refresh succeeds):

=== ARM B: WITH --experimental-vm-modules (expecting SUCCESS) ===
(node:1) ExperimentalWarning: VM Modules is an experimental feature and might change at any time

Either both tests pass, or — if your default Cloud Build SA doesn't have secretmanager.secrets.list / bigquery.jobs.create on the project — you'll instead see legitimate API errors:

● Secret Manager v6 listSecrets (gaxios v7 surface)
    7 PERMISSION_DENIED: Permission 'secretmanager.secrets.list' denied on resource ...

● BigQuery v8 query (gaxios v7 surface)
    Access Denied: User does not have bigquery.jobs.create permission ...

That's the key signal: the gaxios "Could not refresh access token: Unexpected Gaxios Error" is gone the moment --experimental-vm-modules is in place. Same dependency set, same service account, same metadata server — the only difference between the two arms is the Node flag.

Workaround

For anyone hitting this in CI: run Jest with

NODE_OPTIONS="--experimental-vm-modules" jest ...

In Cloud Build, the cleanest way is to set it once globally:

options:
  env:
    - 'NODE_OPTIONS=--experimental-vm-modules'