Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ jobs:
uses: actions/checkout@v6
- name: Install dependencies
run: npm ci
- name: Verify no deep serverless/lib imports (guards #632)
run: npm run verify:imports
- name: Lint
run: npm run lint
- name: Tests
Expand All @@ -47,6 +49,13 @@ jobs:
name: CFN Synthesis Tests
runs-on: ubuntu-latest
needs: tests
# Serverless Framework v4 requires authentication for every CLI invocation,
# including non-interactive CI. The e2e harness spawns `serverless package`,
# so a license/access key must be present or the run will fail/prompt.
# Add SERVERLESS_ACCESS_KEY (Dashboard > Settings > Access Keys) as a repo
# secret. Free for individual developers / orgs under the revenue threshold.
env:
SERVERLESS_ACCESS_KEY: ${{ secrets.SERVERLESS_ACCESS_KEY }}
Comment on lines +52 to +58
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Gate e2e on secret availability for fork PRs.

pull_request workflows from forks do not receive repository secrets, so this job will fail there by default. Add a job condition so fork PRs are not hard-failed by missing SERVERLESS_ACCESS_KEY.

Suggested job gate
   e2e:
     name: CFN Synthesis Tests
     runs-on: ubuntu-latest
     needs: tests
+    if: ${{ github.event_name != 'pull_request' || secrets.SERVERLESS_ACCESS_KEY != '' }}
     # Serverless Framework v4 requires authentication for every CLI invocation,
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci.yml around lines 52 - 58, The e2e job currently always
expects SERVERLESS_ACCESS_KEY but pull_request workflows from forked repos don't
have repo secrets; add a job-level condition to skip/gate the e2e job when the
PR is from a fork (e.g., use a condition based on github.event_name and
github.event.pull_request.head.repo.fork) so the job only runs when secrets are
available; update the job that defines env: SERVERLESS_ACCESS_KEY to include
that if-condition (or equivalent) to prevent hard-failing on forked PRs.

steps:
- name: Checkout code
uses: actions/checkout@v6
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,6 @@ users.json
*.swp
.vscode/settings.json
lib

# transient config written by e2e synthesize helper
examples/**/serverless.e2e.yml
34 changes: 31 additions & 3 deletions e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,15 @@ runs `serverless package` to produce a CloudFormation template, and
asserts on the generated resources.

Critically, **these tests do not deploy anything to AWS** — they only
exercise the synthesis path. That makes them fast (~3s per fixture),
fork-safe (no AWS credentials required), and suitable to run on every
PR.
exercise the synthesis path. That makes them fast (~3s per fixture)
and suitable to run on every PR. No AWS credentials are required.

> **Serverless Framework v4 requires authentication.** As of v4, every
> CLI invocation (including the `serverless package` these tests run)
> must be authenticated, even though nothing is deployed. You need a
> Serverless **Access Key** or **License Key** — free for individual
> developers and organizations under the revenue threshold. See
> [Authentication](#authentication-serverless-v4) below.

## Running locally

Expand All @@ -38,6 +44,24 @@ npm run test:all
npx jest --config jest.e2e.config.ts basic-api-key
```

### Authentication (Serverless v4)

Set a Serverless Access Key (or License Key) before running, otherwise
the spawned `serverless package` will fail / prompt for login:

```bash
# One-time interactive login (writes a key to your machine)
npx serverless login

# …or set an explicit key for this shell / CI
export SERVERLESS_ACCESS_KEY=... # from Dashboard > Settings > Access Keys
# or
export SERVERLESS_LICENSE_KEY=...
```

The harness passes the current environment through to the CLI, so any
of the above is sufficient.

## Adding a new test

1. Create a new example under `examples/<feature>/`:
Expand Down Expand Up @@ -90,3 +114,7 @@ A dedicated `e2e` job in `.github/workflows/ci.yml` runs the full
synthesis test suite on every PR and push to `master` or `alpha`. It
runs after the unit-test matrix completes successfully (no point
running E2E if unit tests already failed).

The job reads `SERVERLESS_ACCESS_KEY` from repository secrets to
authenticate the Serverless v4 CLI. Add it under
**Settings > Secrets and variables > Actions** before this job can pass.
89 changes: 75 additions & 14 deletions e2e/helpers/synthesize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,20 +65,60 @@ export function synthesize(exampleDir: string): SynthesizeResult {

const packageDir = fs.mkdtempSync(path.join(os.tmpdir(), 'sls-appsync-e2e-'));

// Serverless v4's `package` resolves a deployment bucket and, when none is
// configured, calls getOrCreateDefaultBucket() — which reads/writes a shared
// SSM parameter on AWS. That makes `package` require AWS connectivity and
// race across parallel workers (`TooManyUpdates` on the shared parameter).
// Setting `provider.deploymentBucket` short-circuits that path
// (getServerlessDeploymentBucketName() returns early, before any AWS call),
// keeping synthesis fully offline and deterministic. We inject it into a
// temporary config copy via `--config` so the committed examples stay clean
// and deployable.
const baseConfig = fs.readFileSync(
path.join(absoluteExampleDir, 'serverless.yml'),
'utf8',
);
if (!/^provider:[ \t]*\n/m.test(baseConfig)) {
fs.rmSync(packageDir, { recursive: true, force: true });
throw new Error(
`Expected a block-style "provider:" in ${absoluteExampleDir}/serverless.yml ` +
`to inject a deploymentBucket for offline synthesis.`,
);
}
const e2eConfigName = 'serverless.e2e.yml';
const e2eConfigPath = path.join(absoluteExampleDir, e2eConfigName);
fs.writeFileSync(
Comment on lines +88 to +90
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Avoid shared temporary config filename across parallel synth runs.

Line 88 uses a fixed serverless.e2e.yml; concurrent synth calls for the same example can overwrite/delete each other’s config and make packaging flaky.

Suggested fix
-  const e2eConfigName = 'serverless.e2e.yml';
+  const e2eConfigName = `serverless.e2e.${process.pid}.${Date.now()}-${Math.random()
+    .toString(36)
+    .slice(2)}.yml`;
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const e2eConfigName = 'serverless.e2e.yml';
const e2eConfigPath = path.join(absoluteExampleDir, e2eConfigName);
fs.writeFileSync(
const e2eConfigName = `serverless.e2e.${process.pid}.${Date.now()}-${Math.random()
.toString(36)
.slice(2)}.yml`;
const e2eConfigPath = path.join(absoluteExampleDir, e2eConfigName);
fs.writeFileSync(
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/helpers/synthesize.ts` around lines 88 - 90, The code uses a fixed
temporary config name (e2eConfigName / e2eConfigPath) which causes races when
synth runs in parallel; change creation to generate a unique filename (e.g.,
append process.pid + timestamp or a random/UUID suffix or use fs.mkdtemp in
os.tmpdir) when building e2eConfigName/e2eConfigPath, update any cleanup logic
to remove that same unique path, and ensure fs.writeFileSync and any subsequent
reads/packaging reference the new unique variable so concurrent runs won't
clobber each other.

e2eConfigPath,
baseConfig.replace(
/^(provider:[ \t]*\n)/m,
'$1 deploymentBucket: serverless-appsync-e2e\n',
),
);

try {
execFileSync(SERVERLESS_BIN, ['package', '--package', packageDir], {
cwd: absoluteExampleDir,
env: {
...process.env,
// Suppress framework prompts and analytics
SLS_NOTIFICATIONS_MODE: 'off',
SLS_INTERACTIVE_SETUP_ENABLE: '0',
// Set a stable region so tests are deterministic
AWS_REGION: 'us-east-1',
AWS_DEFAULT_REGION: 'us-east-1',
execFileSync(
SERVERLESS_BIN,
['package', '--config', e2eConfigName, '--package', packageDir],
{
cwd: absoluteExampleDir,
env: {
...process.env,
// Suppress framework prompts and analytics
SLS_NOTIFICATIONS_MODE: 'off',
SLS_INTERACTIVE_SETUP_ENABLE: '0',
// Stable region for deterministic output
AWS_REGION: 'us-east-1',
AWS_DEFAULT_REGION: 'us-east-1',
// Dummy credential fallback so CI (no real creds) can satisfy v4's
// credentials check. With deploymentBucket set above, `package` makes
// no real AWS calls, so these are never used against AWS.
AWS_ACCESS_KEY_ID: process.env.AWS_ACCESS_KEY_ID ?? 'e2e-dummy',
AWS_SECRET_ACCESS_KEY:
process.env.AWS_SECRET_ACCESS_KEY ?? 'e2e-dummy',
},
stdio: ['ignore', 'pipe', 'pipe'],
},
stdio: ['ignore', 'pipe', 'pipe'],
});
);
} catch (err) {
// Surface stderr in test output so failures are diagnosable
const e = err as { stderr?: Buffer; stdout?: Buffer; message: string };
Expand All @@ -91,16 +131,37 @@ export function synthesize(exampleDir: string): SynthesizeResult {
`STDERR:\n${stderr}\n` +
`MESSAGE: ${e.message}`,
);
} finally {
fs.rmSync(e2eConfigPath, { force: true });
}

const templatePath = path.join(
// Serverless writes the synthesized template as
// `cloudformation-template-update-stack.json`. We look it up defensively
// (preferring the canonical name, then falling back to any
// `cloudformation-template-*-stack.json`) so the harness is resilient to
// any path/name nuance across Serverless Framework major versions.
const canonical = path.join(
packageDir,
'cloudformation-template-update-stack.json',
);
let templatePath = canonical;
if (!fs.existsSync(templatePath)) {
const fallback = fs
.readdirSync(packageDir)
.find(
(f) =>
/^cloudformation-template-.*-stack\.json$/.test(f) &&
f.endsWith('.json'),
);
if (fallback) {
templatePath = path.join(packageDir, fallback);
}
}
if (!fs.existsSync(templatePath)) {
fs.rmSync(packageDir, { recursive: true, force: true });
throw new Error(
`CloudFormation template was not produced at ${templatePath}.`,
`CloudFormation template was not produced in ${packageDir} ` +
`(expected ${canonical} or a cloudformation-template-*-stack.json).`,
);
}

Expand Down
Loading