Merging regression suites from engines

[titzer]

In some backroom discussions, I've raised the possibility of putting together the union of all the regression suites that are accumulating in various engines. For example, Wizard has thousands of .wast files that are either new tests, regression tests for old bugs, or expansions of spec tests. Due to the magic of .wast, all such tests pass on the Wasm 3.0 spec interpreter (yay!), so I think these could also be useful to other engines. Similarly, other engines' accumulated regression tests have been useful for Wizard as well.

I wonder if there is appetite for duplicating all of the .wast tests out there into this repo? We could do this manually at first, letting engine maintainers push new tests here via PRs. In the future we could also consider scripts to pull in said tests.

There are maybe some issues to discuss, such as making sure that tests mirrored or duplicated here actually do accurately represent the spec, but I think this could be good for the ecosystem.

Thoughts?

@fitzgen @eqrion @kmiller68 @tlively @cfallin

[eqrion]

I think this broadly makes sense and I support it.

Regression tests are often times odd and have a different flavor than normal hand-written tests. I also could imagine we'll end up with a lot of them (sounds like you already have thousands!). So I'd lean towards a dedicated subfolder for them.

We currently have automation that exports the spec tests her to WPT. We would need to be sure that WPT is okay with suddenly getting thousands of new tests. We could probably exclude them from the automation if that's an issue (SpiderMonkey directly imports the spec tests, not through WPT)

Do you have a review policy in mind? Something lightweight would be nice.

Also we don't have any regression tests to contribute because our regression tests are all written in JS using the wasm JS-API. We maybe could change that culturally, but that's what we do right now.

[tlively]

V8 doesn't use .wast tests, either. Binaryen does use .wast tests, but they don't have assertions, so it's unclear how useful they would be to other projects. (They instead use FileCheck from LLVM to check the output of running the input modules through Binaryen tools against expectations that live in comments alongside the input.)

[titzer]

@eqrion, yeah I think it makes sense to have subfolders. E.g. having a folder per engine contributor could make it easier to push/pull in tests from different engines periodically. From skimming the wasmtime code, it has a mix of internal tests written in Rust, API tests in .wat, a copy of the spec testsuite, and some of its own .wast regression tests.

@tlively I'm wondering if the Binaryen .wast tests are runnable. Are the expectations of static transformations, or do they execute the programs?

[alexcrichton]

Personally I'd love to be able to merge regression test suites across engines. For me though a critical part of that would be automation, both automatically pulling in tests into this repo and then pushing them back down to other repos. While I agree manually pushing tests would be a good incremental step I'd fear things would get stale pretty quickly.

I'd agree that there could be a worry that some tests that accidentally test engine-specific behavior that may or may not be spec-compliant, but I'd expect that to get loosely sorted out over time. Consumers of this test suite would likely need some sort of list of allowed and denied tests anyway so such tests could be lumped into there.

Given the spec tests are already in *.wast I'd naively expect engine-specific tests to also be in that format, but I understand that not all engines use this format. Another possible alternative, along the lines of #119, would be to use a different format if the *.wast format is the sticking point. If the sticking point is just moving the tests to a declarative format at all, however, then that's likely going to be difficult to overcome.

[redianthus]

As a datapoint, we do use .wast files in Owi and we are using this repo as a git submodule for our tests.

[rossberg]

Is there any particular reason not to add .wast tests to the upstream spec repo directly, in subfolders or otherwise?

That would also preclude any accidental merging of non-compliant tests, since spec CI runs all tests against the reference interpreter.

I honestly do not understand what would be gained by introducing a different format. All engines already need to be able to execute .wast to run the existing test suite (be it directly, converted to .bin.wast, or converted to .js). Introducing a new format would just shift the burden to also support that one to either a new shared conversion tool, or to every individual engine/tool maker.

[eqrion]

Is there any particular reason not to add .wast tests to the upstream spec repo directly, in subfolders or otherwise?

Speaking for myself, automation would lower the barrier to make this more worthwhile. I think there's a similar precedent with WPTs. It's much more convenient for engineers to just make one review/PR to their local codebase instead of having to duplicate it upstream later.

That would also preclude any accidental merging of non-compliant tests, since spec CI runs all tests against the reference interpreter.

Yeah that's tough. Another related issue is how to separate out regression tests for in-progress proposals. e.g. If we commit a 'stack-switching' regression test, it shouldn't get merged up here until stack-switching is merged.

I honestly do not understand what would be gained by introducing a different format. All engines already need to be able to execute .wast to run the existing test suite (be it directly, converted to .bin.wast, or converted to .js). Introducing a new format would just shift the burden to also support that one to either a new shared conversion tool, or to every individual engine/tool maker.

I don't have an opinion on a new format. At least for SpiderMonkey, we write our tests using the JS-API + 'wasmTextToBinary' because it lets us generate wasm modules programmatically to cover more cases (e.g. all valtypes) in a concise manner.

Our simple regression tests though could be written in wast, but we don't have a wast runner in SM and need to use an offline tool to convert them to a JS file which can be checked in. We probably could add a wast runner into SM and start writing regression tests in wast. We just haven't had a reason to do that work yet.

[titzer]

@rossberg I'd be OK pushing Wizard's tests upstream to spec in principle, but we generally prefer micro-tests rather than per-instruction combination files, unlike the current convention in the spec repo. E.g. there are thousands of Wizard tests that are individually very small. They're valuable to us to be split up this way as they are easier to debug.

[tlively]

@tlively I'm wondering if the Binaryen .wast tests are runnable. Are the expectations of static transformations, or do they execute the programs?

A few are meant to be executed (these live in https://github.com/WebAssembly/binaryen/tree/main/test/lit/exec), but that vast majority are just checking static transformations.

[rossberg]

@titzer, I think that's okay. We can organise secondary tests coming from engines in their own dedicated compartment of the suite, where they can follow their own conventions.