feat!: add generic e2e support to check-store workflow

[digitalrisedorset]

PR Checklist

Please check if your PR fulfills the following requirements:

• The commit message follows our guidelines: https://github.com/graycoreio/github-actions-magento2/blob/main/CONTRIBUTING.md#commit
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

• Bugfix
• Feature
• Code style update (formatting, local variables)
• Refactoring (no functional changes, no api changes)
• Build related changes
• CI related changes
• Documentation content changes
• Other... Please describe:

What is the current behavior?

check-store.yaml currently supports:

• coding-standard
• unit-test
• smoke-test

There is currently no generic mechanism for executing optional store-defined E2E test suites within the reusable workflow.

Fixes: N/A

What is the new behavior?

Adds optional e2e-test support to check-store.yaml.

The workflow now:

• provisions a Node runtime
• installs npm dependencies when a package.json exists
• executes npm run test:e2e

This keeps E2E implementation details owned by the consuming store while allowing Mage-OS CI orchestration to execute generic Node-based E2E suites.

A minimal TypeScript-based E2E fixture was also added to validate the orchestration flow.

Does this PR introduce a breaking change?

• Yes
• No

Other information

This intentionally keeps the E2E contract lightweight and framework-agnostic for now.

The workflow orchestrates execution but does not enforce:

• Playwright
• Cypress
• browser-specific tooling
• E2E file structure conventions

This allows consuming stores to define their own E2E implementation details while reusing the existing Mage-OS orchestration and Magento service bootstrapping.

[damienwebdev]

@herve

So, what I'd like to see is that we can run this against a store even if that store doesn't have the e2e tests installed.

So detect-e2e currently gates this against running if one isn't found, let's actually install the sample data and the e2e tests and attempt to run them.

[digitalrisedorset]

@herve

So, what I'd like to see is that we can run this against a store even if that store doesn't have the e2e tests installed.

So detect-e2e currently gates this against running if one isn't found, let's actually install the sample data and the e2e tests and attempt to run them.

@damienwebdev I've pushed a follow-up commit.

The workflow now:

• Continues normally when no E2E suite is present.
• Installs sample data when E2E is enabled.
• Installs the MageOS-adapted Playwright suite.
• Executes the Playwright tests.
• Removes the Node version pin.

I think this gets us much closer to the flow we discussed earlier today. Could you take another look when you have a moment? I suspect these changes might need tuning from where we are but this structure should be what we are aiming at. Once it works, I'll start 2 new activties:

• adding more Playwrights test on a weekly basis to start adding genuine tests coverage (this is something anyone can add to as well)
• start a E2E adapter so that any E2E tests suite can work. Lucas might want to join to add his test suite to work as an alternative

[digitalrisedorset]

@damienwebdev I've removed the guards you mentioned. I understand this is so that we can discover fully the E2E execution path and identify what still needs to be wired up. I imagine we may reinstate some of these guards once E2E tests are fully integrated.

[damienwebdev]

@Vinai asked two things:

• Is it possible, to use the built-in playwright version vs. installing our own?
• Is it possible to disable npm scripts to just lower the security surface concern?

[digitalrisedorset]

@Vinai asked two things:

• Is it possible, to use the built-in playwright version vs. installing our own?
• Is it possible to disable npm scripts to just lower the security surface concern?

@Vinai , @damienwebdev My thoughts were that MageOS would own the E2E tests and the execution environment (Node, Playwright version, dependency installation, etc.) would be a separate concern.
I understand the concern around keeping the Node/npm security surface area as small as possible, and I'm happy to investigate those options once we've validated the E2E flow itself.

[damienwebdev]

@digitalrisedorset

Thanks for the continued work on this. E2e test code is now running. Before I re-review, I need to see a green run of the Check Store Test workflow executed in your own fork, since fork PRs here don't run the full CI automatically.

Here's how to run it on digitalrisedorset/github-actions-magento2:

1. Enable Actions on your fork, go to the Actions tab and click "I understand my workflows, go ahead and enable them" if you haven't already.
2. Push your changes to feat/check-store-e2e-support on your fork.
3. Trigger the workflow, either:
   • Actions tab → "Check Store Test" → "Run workflow" → select branch feat/check-store-e2e-support, or
   • open a draft PR within your fork (feat/check-store-e2e-support → digitalrisedorset:main) so the pull_request trigger fires on every push.

Two notes:

• The pin in check-store.yaml pointing resolve-check-config at digitalrisedorset/github-actions-magento2/resolve-check-config@feat/check-store-e2e-support is correct for validation, it's what makes the fork run exercise your modified action instead of the published one on main. Keep it while you iterate, but it must be reverted to graycoreio/github-actions-magento2/resolve-check-config@main before this can merge.

Once you have a green run, link it here and I'll pick the review back up.

[digitalrisedorset]

@digitalrisedorset

Thanks for the continued work on this. E2e test code is now running. Before I re-review, I need to see a green run of the Check Store Test workflow executed in your own fork, since fork PRs here don't run the full CI automatically.

Here's how to run it on digitalrisedorset/github-actions-magento2:

1. Enable Actions on your fork, go to the Actions tab and click "I understand my workflows, go ahead and enable them" if you haven't already.

2. Push your changes to feat/check-store-e2e-support on your fork.

3. Trigger the workflow, either:

   • Actions tab → "Check Store Test" → "Run workflow" → select branch feat/check-store-e2e-support, or
   • open a draft PR within your fork (feat/check-store-e2e-support → digitalrisedorset:main) so the pull_request trigger fires on every push.

Two notes:

• The pin in check-store.yaml pointing resolve-check-config at digitalrisedorset/github-actions-magento2/resolve-check-config@feat/check-store-e2e-support is correct for validation, it's what makes the fork run exercise your modified action instead of the published one on main. Keep it while you iterate, but it must be reverted to graycoreio/github-actions-magento2/resolve-check-config@main before this can merge.

Once you have a green run, link it here and I'll pick the review back up.

thanks @damienwebdev for your support through this CI pipeline work. Today, I tracked down the issue preventing me from running the workflow in my fork. My fork was significantly behind upstream, so the Check Store workflow was not available. After syncing the fork, I was able to run Check Store Test successfully in my own fork as requested.
The latest run is green. I understand this is when you wanted to review again and start to investigate how the final use case (having CI to allow the merchant to run his/her own E2E test suite) can work

[michielgerritsen]

@digitalrisedorset a few points I noticed after looking at this PR.

Failing tests

The tests are failing because of this error:

    ╔══════════════════════════════════════════════════════╗
    ║ Host system is missing dependencies to run browsers. ║
    ║ Missing libraries:                                   ║
    ║     libgtk-4.so.1                                    ║
    ║     libgraphene-1.0.so.0                             ║
    ║     libopus.so.0                                     ║
    ║     libgstallocators-1.0.so.0                        ║
// etc

Playwright is a bit weird in the sense that an npm ci/npm install is not enough, you also need to run this command:

npx playwright install --with-deps

If you are only running Chromium browser, which is most cases is sufficient, you can also only install Chromium, which saves a bunch of seconds of not having to download and install webkit/firefox/safari/?:

npx playwright install --with-deps chromium

Artifact

I would recommend adding a step to upload the artifact:

      - uses: actions/upload-artifact@v4
        if: always() && steps.playwright.outcome == 'failure'
        with:
          name: playwright-report
          path: playwright-report/

Double-check the path, but that uploads a report where you can trace back exactly what the browser did, how the page looked, what was clicked on, etc, etc. As if you would run it locally.

Playwright defaults

I always update the playwright.config.ts with a few defaults that are off from the Playwright default:

The reporter:

reporter: process.env.CI
    ? [['github'], ['list', {printSteps: true}], ['html', {open: 'never'}]]
    : [['list', {printSteps: true}], ['html', {open: 'never'}]],

• GitHub: This outputs the failures back into overview of the current GitHub run.
• List: Output the steps it's currently doing. Otherwise, you have to wait until the full test suite has run to know that something has failed. This prints the result of each step as it is executed. Mainly for when watching the log go.
• Html: The report required for the previous step.

maxFailures: 2,

If there are two failed tests, quit running tests. When you have like 50 tests, and the first few are already failing, there is usually no value in running them all.

[digitalrisedorset]

@digitalrisedorset a few points I noticed after looking at this PR.

Failing tests

The tests are failing because of this error:

    ╔══════════════════════════════════════════════════════╗
    ║ Host system is missing dependencies to run browsers. ║
    ║ Missing libraries:                                   ║
    ║     libgtk-4.so.1                                    ║
    ║     libgraphene-1.0.so.0                             ║
    ║     libopus.so.0                                     ║
    ║     libgstallocators-1.0.so.0                        ║
// etc

Playwright is a bit weird in the sense that an npm ci/npm install is not enough, you also need to run this command:

npx playwright install --with-deps

If you are only running Chromium browser, which is most cases is sufficient, you can also only install Chromium, which saves a bunch of seconds of not having to download and install webkit/firefox/safari/?:

npx playwright install --with-deps chromium

Artifact

I would recommend adding a step to upload the artifact:

      - uses: actions/upload-artifact@v4
        if: always() && steps.playwright.outcome == 'failure'
        with:
          name: playwright-report
          path: playwright-report/

Double-check the path, but that uploads a report where you can trace back exactly what the browser did, how the page looked, what was clicked on, etc, etc. As if you would run it locally.

Playwright defaults

I always update the playwright.config.ts with a few defaults that are off from the Playwright default:

The reporter:

reporter: process.env.CI
    ? [['github'], ['list', {printSteps: true}], ['html', {open: 'never'}]]
    : [['list', {printSteps: true}], ['html', {open: 'never'}]],

• GitHub: This outputs the failures back into overview of the current GitHub run.
• List: Output the steps it's currently doing. Otherwise, you have to wait until the full test suite has run to know that something has failed. This prints the result of each step as it is executed. Mainly for when watching the log go.
• Html: The report required for the previous step.

maxFailures: 2,

If there are two failed tests, quit running tests. When you have like 50 tests, and the first few are already failing, there is usually no value in running them all.

Thanks @michielgerritsen for the suggestions, I have added these along with some other attempts today, I hope we have moved forward.

Here's where I am now:

• Added the Playwright reporters (github, list, html) and maxFailures.
• Uploading the Playwright report and test artifacts from the CI workflow.
• Added Playwright browser dependencies to be installed.
• Reduced the scope to a few smoke tests (healthcheck, homepage, category and product) rather than the full suite.

The browser now starts correctly and the tests execute, so we've moved past the infrastructure issues.

The current failure is a 404 on the PLP/PDP, which makes me think this is now an environment or sample data issue rather than a Playwright one. I'm currently installing the sample data using:

bin/magento sampledata:deploy
composer update
bin/magento setup:upgrade
bin/magento indexer:reindex

but the step appears to complete almost instantly, which doesn't seem right, so I suspect the sample data may not actually be available.

I'm going to pause here for now rather than continue debugging blindly. I think we've made good progress on getting the E2E framework into the pipeline, and I'd rather understand the environment issue properly before spending more time chasing it.

Thanks again for your help!

[digitalrisedorset]

@damienwebdev , @michielgerritsen thanks both for your support. The Playwright tests are failing because the sample data are not being installed. Now, this goes slightly beyond the initial scope that I was having in focus.

Since changing permissions and magento settings is security sensitive and likely requires other parts of the system to be adjusted, I was wondering if you could take a look at the error in the workflow and advise as to what may be the best place to change

[michielgerritsen]

Each time I get back to this issue I have to look up all moving parts, so for future reference an overview of all moving parts:

• The main (this) PR.
• That runs against the e2e branch of this fork. (Contains the tests + playwright.config.ts)
• The runs of this PR can be seen in this repository.

@digitalrisedorset Could you verify that I included all moving parts?

[digitalrisedorset]

Each time I get back to this issue I have to look up all moving parts, so for future reference an overview of all moving parts:

• The main (this) PR.
• That runs against the e2e branch of this fork. (Contains the tests + playwright.config.ts)
• The runs of this PR can be seen in this repository.

@digitalrisedorset Could you verify that I included all moving parts?

Yes, @michielgerritsen that's essentially it. The moving parts are currently:

1. This PR (github-actions-magento2) where I'm integrating the E2E workflow into check-store.
2. My fork of mageos-magento2 (ci/tests-e2e-playwright-elgentos-hyva), which currently contains the Playwright suite and related configuration. (ideally, we'd want to ensure our work is not coupled to this PR so that other e2e test suites can consume thie CI pipeline we are modying.
3. The workflow runs from this PR, which clone that E2E branch during execution.

At the moment the pipeline gets as far as running Playwright. The new blocker appears to be the Magento environment itself rather than the Playwright integration. In particular, the sample data don't seem to be available due to invalid permissions, so the PLP/PDP tests return 404s.

I think the next step is to understand the correct way for the CI environment to prepare a Magento instance with sample data rather than continuing to tweak the Playwright side.

[michielgerritsen]

Looking at the pipeline, it's currently failing on permission errors. @damienwebdev I think Docker is in the game here as well. Any insight on how to call bin/magento commands?

[damienwebdev]

@michielgerritsen we don't have any standard way to run bin/magento commands against the container.

We do something like this in https://github.com/graycoreio/github-actions-magento2/blob/main/setup-install/action.yml#L21

Perhaps a new action action for bin/magento with a container_id arg makes sense to solve this specific issue?

You can test out whether or not that's the issue by using docker exec containerId from the pipeline and seeing if your commands work.

[digitalrisedorset]

Hi both, thanks for jumping in to help move this E2E project forward.
Before we spend too much more time on the implementation, I think we should clarify one point that has started to concern me.

Even if we resolve the current environment issues, we should first agree on how we expect E2E to run in CI over the long term. My understanding is that running the full Playwright suite with all supported Magento versions will eventually hit CI timeout limits. It also means that debugging failures could become increasingly complex as different Magento versions, themes and sample data introduce different behaviours.

As I mentioned during our last weekly meetup, I wonder whether we should first focus on a much smaller Playwright focus to answer to some design questions:

• Can we run Playwright in chunks so that the CI remains reliable?
• How flexible should the Playwright configuration be to accommodate differences between supported Magento versions?
• Is running Playwright directly in CI the right long-term model for Mage-OS, given the number of Magento versions we want to support?

I think answering these questions first will give us a much clearer target and reduce the risk of investing time in a pipeline that we later need to redesign.

[michielgerritsen]

• Can we run Playwright in chunks so that the CI remains reliable?
  Yes, that's no issue and I have experience with that. See this action for example, it divides the tests over three runners and merges the results when done. But I have seen runners that go well over an hour; that ain't an issue.

• How flexible should the Playwright configuration be to accommodate differences between supported Magento versions?
  Let's focus on having it run on the latest version for now, then we can work on that later.

• Is running Playwright directly in CI the right long-term model for Mage-OS, given the number of Magento versions we want to support?
  It is. I also have experience with running in Cypress, Playwright is the better option by a wide margin.

Having said that, and I think I've stated this before, but the Elgentos suite isn't the best example to get started with. Let's start with a few basic tests before. As you can see, that's hard enough already. Once that is running, we can look into getting the test suite up on something better.

[digitalrisedorset]

Thanks, I read you have a strong experience with Playwright, I wonder if it makes sense for us to split the work slightly differently for now. Rather than both spending time on the Playwright side, I’m happy to focus on getting the CI environment into the right shape (nginx, running bin/magento commands in the container, permissions, sample data, etc.), so that have the hosting sorted Between me and Damien we can get this hosting cleared.

…

On Tue, 30 Jun 2026 at 20:51, Michiel Gerritsen ***@***.***> wrote: *michielgerritsen* left a comment (graycoreio/github-actions-magento2#279) <#279 (comment)> - *Can we run Playwright in chunks so that the CI remains reliable?* Yes, that's no issue and I have experience with that. See this action <https://github.com/mollie/magento2/actions/runs/28430606638> for example, it divides the tests over three runners and merges the results when done. But I have seen runners that go well over an hour; that ain't an issue. - *How flexible should the Playwright configuration be to accommodate differences between supported Magento versions?* Let's focus on having it run on the latest version for now, then we can work on that later. - *Is running Playwright directly in CI the right long-term model for Mage-OS, given the number of Magento versions we want to support?* It is. I also have experience with running in Cypress, Playwright is the better option by a wide margin. Having said that, and I think I've stated this before, but the Elgentos suite isn't the best example to get started with. Let's start with a few basic tests before. As you can see, that's hard enough already. Once that is running, we can look into getting the test suite up on something better. — Reply to this email directly, view it on GitHub <#279?email_source=notifications&email_token=BHVLMT5YPULENN2YNDX5KNL5CQK5NA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBUG4ZTQNRWGMZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4847386632>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BHVLMT3LCFR7SCIRPI2S2ML5CQK5NAVCNFSNUABFKJSXA33TNF2G64TZHM2TANZTGI2TOOBYHNEXG43VMU5TINBXGUZTMNZSHA32C5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/BHVLMT6BSAQ47TPEKLL5JAT5CQK5NA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBUG4ZTQNRWGMZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/BHVLMT5VCS2WI3VQ7VX7MD35CQK5NA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBUG4ZTQNRWGMZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>