ci: migrate pipeline to Twilio locked-down agents (Chromium CI image)#850
Open
AnkitSegment wants to merge 9 commits into
Open
ci: migrate pipeline to Twilio locked-down agents (Chromium CI image)#850AnkitSegment wants to merge 9 commits into
AnkitSegment wants to merge 9 commits into
Conversation
Contributor
There was a problem hiding this comment.
⚠️ Not ready to approve
The upload/publish steps are moved onto general-039 but still depend on Segment agent hooks (SEGMENT_LIB_PATH/run-with-role), which will likely break master/staging pipelines.
Pull request overview
Migrates the Buildkite CI pipeline to run on Twilio locked-down general-039 agents by introducing a custom Chromium-capable CI image, switching steps to the docker-compose plugin, and pointing npm/yarn at Twilio’s Artifactory registry to avoid Segment-only infrastructure dependencies.
Changes:
- Add a custom CI Docker image (
.buildkite/Dockerfile.ci) that installs Chromium (via Twilio’s Alpine mirror) and provides aphantomjsstub to bypassphantomjs-prebuiltdownloads in no-egress environments. - Introduce a CI-only compose file (
docker-compose-ci.yml) and update Buildkite pipeline steps to run test/SauceLabs jobs via the docker-compose plugin ongeneral-039. - Update Karma to use a
ChromeHeadlessNoSandboxlauncher and add an.npmrcpointing to Twilio’s virtual npm registry.
File summaries
| File | Description |
|---|---|
karma.conf.js |
Switches default launcher to ChromeHeadlessNoSandbox and defines the custom launcher/flags. |
docker-compose-ci.yml |
Adds a CI-only compose definition to build/run the new Chromium test image and propagate needed env vars. |
.npmrc |
Sets npm registry to Twilio Artifactory virtual registry. |
.buildkite/pipeline.yml |
Migrates steps to general-039 and uses the docker-compose plugin for build/test and SauceLabs jobs. |
.buildkite/Dockerfile.ci |
Adds a locked-down CI image (Node + Chromium + build toolchain + phantomjs stub) suitable for no-egress agents. |
Copilot's findings
- Files reviewed: 5/6 changed files
- Comments generated: 3
Note
Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+18
to
+28
| // Run headless Chrome with --no-sandbox: the CI container runs as root, | ||
| // where Chrome's sandbox refuses to start. CHROME_BIN points at the | ||
| // Chromium installed in .buildkite/Dockerfile.ci. | ||
| browsers: ['ChromeHeadlessNoSandbox'], | ||
|
|
||
| customLaunchers: { | ||
| ChromeHeadlessNoSandbox: { | ||
| base: 'ChromeHeadless', | ||
| flags: ['--no-sandbox', '--disable-gpu', '--disable-dev-shm-usage'] | ||
| } | ||
| }, |
| branches: master staging | ||
| agents: | ||
| queue: v1 | ||
| queue: general-039 |
| branches: master | ||
| agents: | ||
| queue: v1 | ||
| queue: general-039 |
Contributor
There was a problem hiding this comment.
⚠️ Not ready to approve
The repo-wide .npmrc registry override can break non-Twilio/local workflows and the pipeline’s auth-token setting should be safely quoted to avoid shell parsing issues.
Copilot's findings
- Files reviewed: 7/8 changed files
- Comments generated: 5
Note
Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.
| queue: general-039 | ||
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" $${NPM_TOKEN} |
| queue: general-039 | ||
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" $${NPM_TOKEN} |
| queue: general-039 | ||
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" $${NPM_TOKEN} |
| queue: general-039 | ||
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" $${NPM_TOKEN} |
| @@ -0,0 +1 @@ | |||
| registry=https://npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/ | |||
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
a0e576d to
653a916
Compare
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
653a916 to
87cba5a
Compare
Contributor
There was a problem hiding this comment.
⚠️ Not ready to approve
CI will fail as submitted due to missing yarn.lock updates for new dependencies and inconsistent Docker Compose service naming between docker-compose-ci.yml, Buildkite config, and scripts.
Copilot's findings
Comments suppressed due to low confidence (1)
.buildkite/pipeline.yml:46
- The stated goal is migrating off Segment-only Buildkite infrastructure for locked-down agents, but this pipeline still includes
segmentio/cache-buildkite-pluginin the SauceLabs and asset upload steps and still runs SauceLabs inside the Segment ECR image (528451384384.dkr.ecr.../analytics.js-integrations-ci). Those dependencies are called out in the PR description as failing on locked-down agents (S3 403 / missing tooling), so those steps are likely to remain broken unless they’re also migrated/removed.
- label: "SauceLabs"
key: "sauce_labs"
branches: "!master"
soft_fail: true
command:
- npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN}
- yarn install --ignore-engines
- yarn test:ci
plugins:
- ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0:
key: "v1-cache-dev-{{ checksum 'yarn.lock' }}"
paths: ["node_modules/"]
s3_bucket_name: "segment-buildkite-cache"
- docker#v3.3.0:
image: 528451384384.dkr.ecr.us-west-2.amazonaws.com/analytics.js-integrations-ci
user: root
environment:
- Files reviewed: 163/164 changed files
- Comments generated: 4
Note
Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.
| "eslint-plugin-prettier": "^3.1.4", | ||
| "fs-extra": "^9.0.1", | ||
| "husky": "^4.3.8", | ||
| "karma-webdriver-launcher": "^1.0.8", |
Comment on lines
+10
to
+12
| test: | ||
| image: 018537234677.dkr.ecr.us-east-1.amazonaws.com/docker.io/library/node:22.18.0-bookworm | ||
| working_dir: /app |
Comment on lines
+11
to
+14
| - docker-compose#v5.12.1: | ||
| run: e2e-runner | ||
| config: docker-compose-ci.yml | ||
| mount-buildkite-agent: true |
Comment on lines
18
to
31
| customLaunchers: { | ||
| ChromeRemote: { | ||
| base: 'WebDriver', | ||
| config: { | ||
| hostname: process.env.SELENIUM_HOST || 'localhost', | ||
| port: 4444 | ||
| }, | ||
| browserName: 'chrome', | ||
| pseudoActivityInterval: 30000 | ||
| } | ||
| }, | ||
|
|
||
| browsers: ['ChromeRemote'], | ||
|
|
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
87cba5a to
b7bac2f
Compare
Contributor
There was a problem hiding this comment.
⚠️ Not ready to approve
There are CI-breaking configuration mismatches (non-existent compose service names) and multiple verified discrepancies with the PR description (missing CI Dockerfile/.npmrc and remaining incompatible cache plugin usage).
Copilot's findings
- Files reviewed: 163/164 changed files
- Comments generated: 11
Note
Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.
Comment on lines
+25
to
+28
| - docker-compose#v5.12.1: | ||
| run: e2e-runner | ||
| config: docker-compose-ci.yml | ||
| mount-buildkite-agent: true |
Comment on lines
+11
to
+14
| - docker-compose#v5.12.1: | ||
| run: test | ||
| config: docker-compose-ci.yml | ||
| mount-buildkite-agent: true |
Comment on lines
75
to
79
| - label: ":cloud: Upload Assets to stage bucket" | ||
| branches: master staging | ||
| agents: | ||
| queue: v1 | ||
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/upload.sh |
Comment on lines
86
to
90
| - label: ":cloud: Upload Assets to production bucket" | ||
| branches: master | ||
| agents: | ||
| queue: v1 | ||
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/publish.sh |
| --volume "$PWD:/workdir" --workdir /workdir \ | ||
| -e NPM_TOKEN -e NODE_ENV \ | ||
| -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \ | ||
| app sh -e -c ' |
| } | ||
| }, | ||
|
|
||
| browsers: ['ChromeRemote'], |
Comment on lines
4
to
6
| env: | ||
| COMPOSE_PROFILES: "ci" | ||
| branches: "!master" |
Comment on lines
+10
to
+12
| test: | ||
| image: 018537234677.dkr.ecr.us-east-1.amazonaws.com/docker.io/library/node:22.18.0-bookworm | ||
| working_dir: /app |
| -e NPM_TOKEN -e NODE_ENV \ | ||
| -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \ | ||
| app sh -e -c ' | ||
| npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" "${NPM_TOKEN}" |
| -e NPM_TOKEN -e NODE_ENV \ | ||
| -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \ | ||
| app sh -e -c ' | ||
| npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" "${NPM_TOKEN}" |
These dependencies were unused — no karma config ever sets PhantomJS as a browser (all use ChromeRemote/SauceLabs). The phantomjs-prebuilt post-install script downloads a binary from GitHub which times out in CI, breaking builds. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add healthcheck to chrome service so test container waits until Selenium Grid is ready before starting (fixes race condition) - Fix karma.conf.js ChromeRemote config to use `url` instead of hostname/port — karma-webdriver-launcher requires a `url` key, the old config caused Selenium to receive "url" as a session ID Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
⚠️ Not ready to approve
The Buildkite pipeline and upload/publish scripts reference docker-compose services that don’t exist in docker-compose-ci.yml, which will break CI execution.
Copilot's findings
- Files reviewed: 164/165 changed files
- Comments generated: 6
Note
Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.
Comment on lines
+24
to
+28
| plugins: | ||
| - ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0: | ||
| key: "v1-cache-dev-{{ checksum 'yarn.lock' }}" | ||
| paths: [ "node_modules/" ] | ||
| s3_bucket_name: "segment-buildkite-cache" | ||
| save: true | ||
| - docker#v3.3.0: | ||
| image: circleci/node:12.18-browsers | ||
| user: root | ||
| environment: | ||
| - NPM_TOKEN | ||
| - CHROME-BIN=google-chrome | ||
| - docker-compose#v5.12.1: | ||
| run: e2e-runner | ||
| config: docker-compose-ci.yml | ||
| mount-buildkite-agent: true |
| queue: v1 | ||
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/upload.sh |
Comment on lines
+29
to
+33
| docker compose -f docker-compose-ci.yml run --rm \ | ||
| --volume "$PWD:/workdir" --workdir /workdir \ | ||
| -e NPM_TOKEN -e NODE_ENV \ | ||
| -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \ | ||
| app sh -e -c ' |
Comment on lines
+29
to
+33
| docker compose -f docker-compose-ci.yml run --rm \ | ||
| --volume "$PWD:/workdir" --workdir /workdir \ | ||
| -e NPM_TOKEN -e NODE_ENV \ | ||
| -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \ | ||
| app sh -e -c ' |
Comment on lines
+18
to
+22
| customLaunchers: { | ||
| ChromeRemote: { | ||
| base: 'WebDriver', | ||
| config: { | ||
| url: 'http://' + (process.env.SELENIUM_HOST || 'localhost') + ':4444/wd/hub' |
Comment on lines
+11
to
+14
| - docker-compose#v5.12.1: | ||
| run: test | ||
| config: docker-compose-ci.yml | ||
| mount-buildkite-agent: true |
The docker-compose plugin runs 'docker compose run' and only passes env vars explicitly listed under its environment key — the compose file's environment block is not automatically forwarded. Add SELENIUM_HOST=chrome explicitly so karma-webdriver-launcher connects to the chrome service instead of falling back to localhost:4444 (ECONNREFUSED). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The b7bac2f migration replaced the working Chromium-in-container setup (from 4e9fe51) with a Selenium standalone-chrome:126 + karma-webdriver-launcher approach that cannot work: wd@1.14.0 speaks JSON Wire Protocol but Selenium 4 only speaks W3C WebDriver, causing every session to get session ID "url" and fail with NoSuchSessionException. Restore 4e9fe51's approach: - .buildkite/Dockerfile.ci: build image with node + Chromium from Twilio ECR/apk mirrors (no public egress needed) - docker-compose-ci.yml: single 'app' service built from Dockerfile.ci - karma.conf.js: ChromeHeadlessNoSandbox (no Selenium, no WebDriver) - .buildkite/pipeline.yml: docker-compose plugin with mount-checkout Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
⚠️ Not ready to approve
The repo’s Karma config runs ChromeHeadless but karma-chrome-launcher has been removed from dependencies (package.json/yarn.lock), so test execution will fail until the Chrome launcher dependency is restored.
Copilot's findings
- Files reviewed: 165/166 changed files
- Comments generated: 4
Note
Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.
| "eslint-plugin-prettier": "^3.1.4", | ||
| "fs-extra": "^9.0.1", | ||
| "husky": "^4.3.8", | ||
| "karma-webdriver-launcher": "^1.0.8", |
| "karma": "^4.1.0", | ||
| "karma-browserify": "^6.0.0", | ||
| "karma-chrome-launcher": "^2.2.0", | ||
| "karma-webdriver-launcher": "^1.0.8", |
Comment on lines
+41
to
+42
| - wait: ~ | ||
| depends_on: ["build_master", "sauce_labs_master"] | ||
| depends_on: ["build_master"] |
Comment on lines
+38
to
+47
| # Several integrations still list phantomjs-prebuilt as a devDependency. Its | ||
| # postinstall downloads a binary from a public CDN, which is unreachable on the | ||
| # no-egress agents and fails `yarn install`. phantomjs-prebuilt has no | ||
| # skip-download flag, but its installer (tryPhantomjsOnPath) skips the download | ||
| # when a `phantomjs` already on PATH reports the expected version (2.1.1). The | ||
| # tests only ever launch ChromeHeadless (karma.conf.js) -- the phantom launcher | ||
| # is an unused devDependency -- so a stub that just answers `--version` is | ||
| # enough to satisfy the installer without ever running as a browser. | ||
| RUN printf '#!/bin/sh\necho 2.1.1\n' > /usr/local/bin/phantomjs && \ | ||
| chmod +x /usr/local/bin/phantomjs |
ChromeHeadlessNoSandbox in karma.conf.js requires karma-chrome-launcher but it was missing from the workspace root, causing all integrations to fail with "No provider for launcher:ChromeHeadless". Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
b7bac2f replaced karma-chrome-launcher with karma-webdriver-launcher in boomtrain. Since we now use ChromeHeadlessNoSandbox everywhere, boomtrain needs karma-chrome-launcher back to launch the browser correctly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
3599902 to
228055c
Compare
Comment on lines
+23
to
+27
| - docker-compose#v5.12.1: | ||
| run: app | ||
| config: docker-compose-ci.yml | ||
| mount-checkout: true | ||
| propagate-environment: true |
Comment on lines
15
to
18
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" $${NPM_TOKEN} | ||
| - yarn install --frozen-lockfile | ||
| - yarn test |
Comment on lines
32
to
35
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" $${NPM_TOKEN} | ||
| - yarn install --frozen-lockfile | ||
| - yarn test-master |
Comment on lines
+18
to
+20
| // Run headless Chrome with --no-sandbox: the CI container runs as root, | ||
| // where Chrome's sandbox refuses to start. CHROME_BIN points at the | ||
| // Chromium installed in .buildkite/Dockerfile.ci. |
| @@ -0,0 +1,13 @@ | |||
| FROM registry.twilio.com/library/base-node/22:22 | |||
Comment on lines
+33
to
+37
| app sh -e -c ' | ||
| npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" "${NPM_TOKEN}" | ||
| yarn install --frozen-lockfile | ||
| make build-and-upload | ||
| ' No newline at end of file |
Comment on lines
+28
to
+30
| echo "--- Build and publish assets inside the CI image" | ||
| docker compose -f docker-compose-ci.yml run --rm \ | ||
| --volume "$PWD:/workdir" --workdir /workdir \ |
Comment on lines
+33
to
+37
| app sh -e -c ' | ||
| npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" "${NPM_TOKEN}" | ||
| yarn install --frozen-lockfile | ||
| make build-and-publish | ||
| ' No newline at end of file |
Comment on lines
88
to
94
| - label: ":cloud: Upload Assets to stage bucket" | ||
| branches: master staging | ||
| agents: | ||
| queue: v1 | ||
| queue: general-039 | ||
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/upload.sh |
Comment on lines
101
to
107
| - label: ":cloud: Upload Assets to production bucket" | ||
| branches: master | ||
| agents: | ||
| queue: v1 | ||
| queue: general-039 | ||
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/publish.sh |
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
228055c to
6687093
Compare
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
6687093 to
72f9b66
Compare
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
72f9b66 to
d136149
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 164 out of 165 changed files in this pull request and generated 11 comments.
Comments suppressed due to low confidence (2)
.buildkite/pipeline.yml:124
- Same issue as the stage upload step: this production upload still uses Segment’s cache-buildkite-plugin on
general-039agents, which is expected to fail and is no longer needed now that publish.sh runs installs inside the CI image.
- NODE_ENV=production ./.buildkite/publish.sh
plugins:
- ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0:
key: "v1-cache-dev-{{ checksum 'yarn.lock' }}"
paths: ["node_modules/"]
s3_bucket_name: "segment-buildkite-cache"
.buildkite/pipeline.yml:56
- This SauceLabs step is still configured to install from the public npm registry and uses Segment’s cache-buildkite-plugin/S3 bucket. Because this pipeline runs on
general-039agents, that combination is expected to fail (no public egress + cache plugin root-cause issues). Remove the cache plugin here and configure npm/Yarn to use the Twilio Artifactory registry.
key: "sauce_labs"
branches: "!master"
Comment on lines
+35
to
+39
| - docker-compose#v5.12.1: | ||
| run: app | ||
| config: docker-compose-ci.yml | ||
| mount-checkout: true | ||
| propagate-environment: true |
Comment on lines
27
to
30
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" $${NPM_TOKEN} | ||
| - yarn install --frozen-lockfile | ||
| - yarn test |
Comment on lines
44
to
47
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" $${NPM_TOKEN} | ||
| - yarn install --frozen-lockfile | ||
| - yarn test-master |
Comment on lines
84
to
88
| plugins: | ||
| - ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0: | ||
| key: "v1-cache-dev-{{ checksum 'yarn.lock' }}" | ||
| paths: [ "node_modules/" ] | ||
| paths: ["node_modules/"] | ||
| s3_bucket_name: "segment-buildkite-cache" |
Comment on lines
102
to
111
Comment on lines
+33
to
+37
| app sh -e -c ' | ||
| npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" "${NPM_TOKEN}" | ||
| yarn install --frozen-lockfile | ||
| make build-and-upload | ||
| ' No newline at end of file |
Comment on lines
+28
to
+31
| echo "--- Build and publish assets inside the CI image" | ||
| docker compose -f docker-compose-ci.yml run --rm \ | ||
| --volume "$PWD:/workdir" --workdir /workdir \ | ||
| -e NPM_TOKEN -e NODE_ENV \ |
Comment on lines
+33
to
+37
| app sh -e -c ' | ||
| npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" "${NPM_TOKEN}" | ||
| yarn install --frozen-lockfile | ||
| make build-and-publish | ||
| ' No newline at end of file |
Comment on lines
+18
to
+28
| // Run headless Chrome with --no-sandbox: the CI container runs as root, | ||
| // where Chrome's sandbox refuses to start. CHROME_BIN points at the | ||
| // Chromium installed in .buildkite/Dockerfile.ci. | ||
| browsers: ['ChromeHeadlessNoSandbox'], | ||
|
|
||
| customLaunchers: { | ||
| ChromeHeadlessNoSandbox: { | ||
| base: 'ChromeHeadless', | ||
| flags: ['--no-sandbox', '--disable-gpu', '--disable-dev-shm-usage'] | ||
| } | ||
| }, |
Comment on lines
+56
to
+57
| ### Publishing node-browser image (Segment Internal) | ||
| Follow the instructions [here](https://twilio-productivity.atlassian.net/wiki/spaces/DA1/pages/1738014994/Segment+to+Twilio+Buildkite+Migration#Instructions-for-building-test-image-in-Analytics.js-Integrations) to build and publish the test image (if required). No newline at end of file |
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
d136149 to
37ae2f5
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 165 out of 166 changed files in this pull request and generated 10 comments.
Comments suppressed due to low confidence (1)
.buildkite/pipeline.yml:44
- The SauceLabs step still depends on Segment-only infrastructure (Segment cache plugin + old docker plugin/image), which contradicts the PR description’s goal of working on locked-down Twilio agents and likely reintroduces the original failures (S3 403 / missing plugin). Consider aligning SauceLabs with the same locked-down Docker image/plugin used by Build/Test and dropping the Segment cache plugin.
plugins:
- ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0:
key: "v1-cache-dev-{{ checksum 'yarn.lock' }}"
paths: ["node_modules/"]
s3_bucket_name: "segment-buildkite-cache"
Comment on lines
15
to
+19
| plugins: | ||
| - ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0: | ||
| key: "v1-cache-dev-{{ checksum 'yarn.lock' }}" | ||
| paths: ["node_modules/"] | ||
| s3_bucket_name: "segment-buildkite-cache" | ||
| save: true | ||
| - docker#v3.3.0: | ||
| image: circleci/node:12.18-browsers | ||
| user: root | ||
| environment: | ||
| - NPM_TOKEN | ||
| - CHROME-BIN=google-chrome | ||
| - docker#v5.12.0: | ||
| image: '${NODE_BROWSER_IMAGE}' | ||
| mount-ssh-agent: true | ||
| propagate-environment: true |
Comment on lines
27
to
+31
| plugins: | ||
| - ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0: | ||
| key: "v1-cache-dev-{{ checksum 'yarn.lock' }}" | ||
| paths: [ "node_modules/" ] | ||
| s3_bucket_name: "segment-buildkite-cache" | ||
| save: true | ||
| - docker#v3.3.0: | ||
| image: circleci/node:12.18-browsers | ||
| user: root | ||
| environment: | ||
| - NPM_TOKEN | ||
| - CHROME-BIN=google-chrome | ||
| - docker#v5.12.0: | ||
| image: '${NODE_BROWSER_IMAGE}' | ||
| mount-ssh-agent: true | ||
| propagate-environment: true |
Comment on lines
60
to
64
| plugins: | ||
| - ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0: | ||
| key: "v1-cache-dev-{{ checksum 'yarn.lock' }}" | ||
| paths: [ "node_modules/" ] | ||
| paths: ["node_modules/"] | ||
| s3_bucket_name: "segment-buildkite-cache" |
Comment on lines
76
to
80
| - label: ":cloud: Upload Assets to stage bucket" | ||
| branches: master staging | ||
| agents: | ||
| queue: v1 | ||
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/upload.sh |
Comment on lines
87
to
91
| - label: ":cloud: Upload Assets to production bucket" | ||
| branches: master | ||
| agents: | ||
| queue: v1 | ||
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/publish.sh |
| queue: general-039 | ||
|
|
||
| env: | ||
| SEGMENT_CONTEXTS: "snyk,npm,aws-credentials,ecr,saucelabs,npm-publish" |
Comment on lines
+29
to
+33
| docker compose -f docker-compose-ci.yml run --rm \ | ||
| --volume "$PWD:/workdir" --workdir /workdir \ | ||
| -e NPM_TOKEN -e NODE_ENV \ | ||
| -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \ | ||
| app sh -e -c ' |
Comment on lines
+29
to
+33
| docker compose -f docker-compose-ci.yml run --rm \ | ||
| --volume "$PWD:/workdir" --workdir /workdir \ | ||
| -e NPM_TOKEN -e NODE_ENV \ | ||
| -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \ | ||
| app sh -e -c ' |
| @@ -0,0 +1,14 @@ | |||
| #!/bin/bash | |||
| ### Releasing | ||
| All releases are handled by Segment engineers. Releases will be managed after a change has been approved and merged. | ||
|
|
||
| ### Publishing node-browser image (Segment Internal) |
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
37ae2f5 to
c143c0f
Compare
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
c143c0f to
994f02c
Compare
Comment on lines
15
to
+19
| plugins: | ||
| - ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0: | ||
| key: "v1-cache-dev-{{ checksum 'yarn.lock' }}" | ||
| paths: ["node_modules/"] | ||
| s3_bucket_name: "segment-buildkite-cache" | ||
| save: true | ||
| - docker#v3.3.0: | ||
| image: circleci/node:12.18-browsers | ||
| user: root | ||
| environment: | ||
| - NPM_TOKEN | ||
| - CHROME-BIN=google-chrome | ||
| - docker#v5.12.0: | ||
| image: '${NODE_BROWSER_IMAGE}' | ||
| mount-ssh-agent: true | ||
| propagate-environment: true |
Comment on lines
27
to
+31
| plugins: | ||
| - ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0: | ||
| key: "v1-cache-dev-{{ checksum 'yarn.lock' }}" | ||
| paths: [ "node_modules/" ] | ||
| s3_bucket_name: "segment-buildkite-cache" | ||
| save: true | ||
| - docker#v3.3.0: | ||
| image: circleci/node:12.18-browsers | ||
| user: root | ||
| environment: | ||
| - NPM_TOKEN | ||
| - CHROME-BIN=google-chrome | ||
| - docker#v5.12.0: | ||
| image: '${NODE_BROWSER_IMAGE}' | ||
| mount-ssh-agent: true | ||
| propagate-environment: true |
Comment on lines
12
to
14
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - yarn install --frozen-lockfile | ||
| - yarn test |
Comment on lines
24
to
26
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - yarn install --frozen-lockfile | ||
| - yarn test-master |
Comment on lines
37
to
40
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - yarn install --ignore-engines | ||
| - yarn test:ci | ||
| plugins: |
Comment on lines
78
to
80
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/upload.sh |
Comment on lines
89
to
91
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/publish.sh |
Comment on lines
+29
to
+33
| docker compose -f docker-compose-ci.yml run --rm \ | ||
| --volume "$PWD:/workdir" --workdir /workdir \ | ||
| -e NPM_TOKEN -e NODE_ENV \ | ||
| -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \ | ||
| app sh -e -c ' |
Comment on lines
+29
to
+33
| docker compose -f docker-compose-ci.yml run --rm \ | ||
| --volume "$PWD:/workdir" --workdir /workdir \ | ||
| -e NPM_TOKEN -e NODE_ENV \ | ||
| -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \ | ||
| app sh -e -c ' |
Comment on lines
+18
to
+20
| // Run headless Chrome with --no-sandbox: the CI container runs as root, | ||
| // where Chrome's sandbox refuses to start. CHROME_BIN points at the | ||
| // Chromium installed in .buildkite/Dockerfile.ci. |
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
994f02c to
6e07c43
Compare
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
6e07c43 to
882ec37
Compare
Comment on lines
34
to
38
| - label: "SauceLabs" | ||
| key: "sauce_labs" | ||
| branches: "!master" | ||
| soft_fail: true | ||
| command: |
Comment on lines
56
to
60
| branches: "master" | ||
| soft_fail: true | ||
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - yarn install --ignore-engines | ||
| - yarn test-master:ci |
Comment on lines
77
to
81
| - label: ":cloud: Upload Assets to stage bucket" | ||
| branches: master staging | ||
| agents: | ||
| queue: v1 | ||
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/upload.sh |
Comment on lines
88
to
92
| - label: ":cloud: Upload Assets to production bucket" | ||
| branches: master | ||
| agents: | ||
| queue: v1 | ||
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/publish.sh |
Comment on lines
+29
to
+33
| docker compose -f docker-compose-ci.yml run --rm \ | ||
| --volume "$PWD:/workdir" --workdir /workdir \ | ||
| -e NPM_TOKEN -e NODE_ENV \ | ||
| -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \ | ||
| app sh -e -c ' |
Comment on lines
+29
to
+33
| docker compose -f docker-compose-ci.yml run --rm \ | ||
| --volume "$PWD:/workdir" --workdir /workdir \ | ||
| -e NPM_TOKEN -e NODE_ENV \ | ||
| -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \ | ||
| app sh -e -c ' |
Comment on lines
+10
to
+14
| export AWS_ACCESS_KEY_ID=$(echo "$CREDS" | awk '{print $1}') | ||
| export AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | awk '{print $2}') | ||
| export AWS_SESSION_TOKEN=$(echo "$CREDS" | awk '{print $3}') | ||
| aws --region us-west-2 ecr get-login-password \ | ||
| | docker login --username AWS --password-stdin 528451384384.dkr.ecr.us-west-2.amazonaws.com |
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
882ec37 to
500f230
Compare
sayan-das-in
force-pushed
the
buildkite-migrations
branch
from
500f230 to
e7ab975
Compare
⛔ Snyk checks have failed. 25 issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 165 out of 166 changed files in this pull request and generated 12 comments.
Comments suppressed due to low confidence (1)
.buildkite/pipeline.yml:44
- The SauceLabs steps still depend on Segment-only infrastructure (
segmentio/cache-buildkite-plugin+segment-buildkite-cacheS3 bucket) even though the PR description says this was removed to work on locked-down agents. This will likely keep failing ongeneral-039and also leaves these steps running in a different image than the new${NODE_BROWSER_IMAGE}.
command:
- yarn install --ignore-engines
- yarn test:ci
plugins:
- ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0:
Comment on lines
16
to
+21
| - yarn test | ||
| plugins: | ||
| - ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0: | ||
| key: "v1-cache-dev-{{ checksum 'yarn.lock' }}" | ||
| paths: ["node_modules/"] | ||
| s3_bucket_name: "segment-buildkite-cache" | ||
| save: true | ||
| - docker#v3.3.0: | ||
| image: circleci/node:12.18-browsers | ||
| user: root | ||
| environment: | ||
| - NPM_TOKEN | ||
| - CHROME-BIN=google-chrome | ||
| - docker#v5.12.0: | ||
| image: '${NODE_BROWSER_IMAGE}' | ||
| mount-ssh-agent: true | ||
| propagate-environment: true |
Comment on lines
29
to
+34
| - yarn test-master | ||
| plugins: | ||
| - ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0: | ||
| key: "v1-cache-dev-{{ checksum 'yarn.lock' }}" | ||
| paths: [ "node_modules/" ] | ||
| s3_bucket_name: "segment-buildkite-cache" | ||
| save: true | ||
| - docker#v3.3.0: | ||
| image: circleci/node:12.18-browsers | ||
| user: root | ||
| environment: | ||
| - NPM_TOKEN | ||
| - CHROME-BIN=google-chrome | ||
| - docker#v5.12.0: | ||
| image: '${NODE_BROWSER_IMAGE}' | ||
| mount-ssh-agent: true | ||
| propagate-environment: true |
Comment on lines
60
to
65
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - yarn install --ignore-engines | ||
| - yarn test-master:ci | ||
| plugins: | ||
| - ssh://git@github.com/segmentio/cache-buildkite-plugin#v1.0.0: | ||
| key: "v1-cache-dev-{{ checksum 'yarn.lock' }}" |
Comment on lines
79
to
83
| - label: ":cloud: Upload Assets to stage bucket" | ||
| branches: master staging | ||
| agents: | ||
| queue: v1 | ||
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/upload.sh |
Comment on lines
90
to
94
| - label: ":cloud: Upload Assets to production bucket" | ||
| branches: master | ||
| agents: | ||
| queue: v1 | ||
| command: | ||
| - echo "--- Upload" | ||
| - NODE_ENV=production ./.buildkite/publish.sh |
Comment on lines
13
to
16
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" $${NPM_TOKEN} | ||
| - yarn install --frozen-lockfile | ||
| - yarn test |
Comment on lines
26
to
29
| command: | ||
| - npm config set "//registry.npmjs.org/:_authToken" $${NPM_TOKEN} | ||
| - npm config set "//npmjs.artifacts.twilio.com/artifactory/api/npm/virtual-npm-twilio/:_authToken" $${NPM_TOKEN} | ||
| - yarn install --frozen-lockfile | ||
| - yarn test-master |
Comment on lines
+18
to
+22
| // Run headless Chrome with --no-sandbox: the CI container runs as root, | ||
| // where Chrome's sandbox refuses to start. CHROME_BIN points at the | ||
| // Chromium installed in .buildkite/Dockerfile.ci. | ||
| browsers: ['ChromeHeadlessNoSandbox'], | ||
|
|
Comment on lines
+55
to
+57
|
|
||
| ### Publishing node-browser image (Segment Internal) | ||
| Follow the instructions [here](https://twilio-productivity.atlassian.net/wiki/spaces/DA1/pages/1738014994/Segment+to+Twilio+Buildkite+Migration#Instructions-for-building-test-image-in-Analytics.js-Integrations) to build and publish the test image (if required). No newline at end of file |
Comment on lines
+10
to
+14
| export AWS_ACCESS_KEY_ID=$(echo "$CREDS" | awk '{print $1}') | ||
| export AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | awk '{print $2}') | ||
| export AWS_SESSION_TOKEN=$(echo "$CREDS" | awk '{print $3}') | ||
| aws --region us-west-2 ecr get-login-password \ | ||
| | docker login --username AWS --password-stdin 528451384384.dkr.ecr.us-west-2.amazonaws.com |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the Buildkite pipeline on the Twilio locked-down
general-039agents. The previous pipeline still depended on Segment-only infrastructure and failed every run.Verified green on build #11 (Build and Test + SauceLabs both passed).
Root causes fixed
cache-buildkite-plugin→shasum: command not found+ S3403on the locked-down agents. Removed.ChromeHeadless, butnode:*-alpineships no browser. Added a CI image with Chromium.docker.iopulls TLS-timeout, soapk add chromiumcan't reach the public Alpine CDN. Chromium is installed from the Twiliobase-alpineapk mirror.phantomjs-prebuiltpostinstall download (the real blocker) — several integrations pull it as a devDependency; its install downloads a binary from a public CDN and failsyarn install. It has no skip flag, so a tinyphantomjsstub onPATHreporting2.1.1makes its installer skip the download. The phantom launcher is unused — tests only run ChromeHeadless.Changes
.buildkite/Dockerfile.cidocker-compose-ci.yml.buildkite/pipeline.ymlmount-checkoutkarma.conf.jsChromeHeadlessNoSandboxlauncher (container runs as root).npmrcNotes / follow-ups
yarn testruns--since masterand reports "no packages" — passes without exercising integration tests. The CI infra is verified working; a branch touching an integration will exercise real Chrome runs.Upload Assetssteps still call./.buildkite/upload.sh/publish.sh, which rely on Segment agent hooks (SEGMENT_LIB_PATH,run-with-role). Being verified separately onstaging.🤖 Generated with Claude Code