Test fixedErrorsDisappear is flaky

[PaulKlint]

Describe the bug

During builds the test fixedErrorsDisappear fails intermittently. Here is one of the many examples: https://github.com/usethesource/rascal/actions/runs/22952589966/job/66621434269

[DavyLandman]

it takes a while to run (like 20-30min), but on my machine I can checkout the main branch, and run:

while mvn -P compiler-tests test -Drascal.compile.skip -Drascal.tutor.skip -Dtest=org.rascalmpl.compiler.lang.rascalcore.check.tests.RunTests; do :; done

and it will stop after a certain amount of runs with this exception.

I suspect it has to do with the order of the test modules, because if I only run ChangeScenarioTests in a loop, I cannot trigger it yet. (but this needs a bit more runs)

[DavyLandman]

Correction. I managed to hit it with ChangeScenarioTests but it took 100+ tries.

this was the output:

[INFO] Running org.rascalmpl.compiler.lang.rascalcore.check.tests.RunTests
Rascal JUnit test runner uses Rascal version Not specified
Rascal JUnit project root: |file:///home/davy/swat.engineering/rascal/rascal/|
[INFO] Bootstrap |std:///| = |file:///home/davy/swat.engineering/rascal/rascal/target/classes/org/rascalmpl/library|
[INFO] Bootstrap runtime   = |file:///home/davy/swat.engineering/rascal/rascal/target/classes|
[INFO] pom.xml: Rascal version is Not specified
[INFO] pom.xml: Detected Rascal project self-application
[INFO] pom.xml: Project root is |file:///home/davy/swat.engineering/rascal/rascal/|
[INFO] pom.xml: Bin folder   is |file:///home/davy/swat.engineering/rascal/rascal/target/classes|
[INFO] pom.xml: Source module path is:
  - |std:///|
  - |file:///home/davy/swat.engineering/rascal/rascal/src/org/rascalmpl/compiler|
  - |file:///home/davy/swat.engineering/rascal/rascal/src/org/rascalmpl/tutor|
  - |file:///home/davy/swat.engineering/rascal/rascal/test/org/rascalmpl/test/data|
  - |file:///home/davy/swat.engineering/rascal/rascal/test/org/rascalmpl/benchmark|
  - |mvn://org.rascalmpl--typepal--0.16.6-RC2|
[INFO] pom.xml: Library module (and classes) path is:
  - |file:///home/davy/swat.engineering/rascal/rascal/target/classes|
Job: loading modules
Job: loading modules
Job: Generating parser;
Job: loading modules
Job: loading modules
Job: Testing lang::rascalcore::check::tests::ChangeScenarioTests
Job: Compiling C A B C
Job: Compiling D D
Job: Compiling C A
Job: Compiling C C D
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling B
Job: Compiling A A
Job: Compiling B B
Job: Compiling B A
Job: Compiling C C
Job: Compiling C A
Job: Compiling C B
Job: Compiling Top Exception Math List Map Set Top
Job: Compiling Top Exception
Job: Compiling Top Set
Job: Compiling Top Exception Set
Job: Compiling A
Job: Compiling B
Job: Compiling B
error("Cannot add grammar or tmodel since `A` is not found",|memory:///test-modules/B.rsc|(0,42,<1,0>,<1,42>))
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling B
Job: Compiling B
error("Cannot add grammar or tmodel since `A` is not found",|memory:///test-modules/B.rsc|(0,18,<1,0>,<1,18>))
Job: Compiling A
Job: Compiling B
Job: Compiling Scratch
Job: Compiling B
Job: Compiling Scratch
Job: Compiling D A B C D
Job: Compiling D A B C
Job: Compiling D C D
Job: Compiling B
error("Cannot add grammar or tmodel since `A` is not found",|memory:///test-modules/B.rsc|(0,18,<1,0>,<1,18>))
Job: Compiling C
Job: Compiling A
Job: Compiling B
Job: Compiling C
Job: Compiling A
Job: Compiling C
Job: Compiling List
CHANGE ["Exception"]
Job: Compiling List Exception
CHANGE ["Map"]
Job: Compiling List Map
Job: Compiling E A B C D E
Job: Compiling C B C
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling B
Job: Compiling B
error("Cannot add grammar or tmodel since `A` is not found",|memory:///test-modules/B.rsc|(0,18,<1,0>,<1,18>))
Job: Compiling C
Job: Compiling A
Job: Compiling B
Job: Compiling B
error("Cannot add grammar or tmodel since `A` is not found",|memory:///test-modules/B.rsc|(0,53,<1,0>,<1,53>))
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling B
Job: Compiling B
error("Cannot add grammar or tmodel since `A` is not found",|memory:///test-modules/B.rsc|(0,38,<1,0>,<1,38>))
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling B
pathConfig(
  bin=|memory:///test-modules/rascal-tests-bin-1|,
  generatedSources=|memory:///test-modules/generated-test-sources-1|,
  libs=[|mvn://org.rascalmpl--rascal--0.42.0-RC2|],
  srcs=[|memory:///test-modules/|],
  generatedResources=|memory:///test-modules/generated-test-resources-1|)
Job: Compiling ParserBase FileLocations Debugging FileUtility ListUtility StringUtility MathUtility
pathConfig(
  bin=|memory:///test-modules/rascal-tests-bin-2|,
  generatedSources=|memory:///test-modules/generated-test-sources-2|,
  libs=[|mvn://org.rascalmpl--rascal--0.42.0-RC2|],
  srcs=[|memory:///test-modules/|],
  generatedResources=|memory:///test-modules/generated-test-resources-2|)
Job: Compiling ParserBase FileLocations Debugging FileUtility ListUtility StringUtility MathUtility
Job: Compiling ParserBase
error("Cannot add grammar or tmodel since `vis::ParseTree` is not found",|memory:///test-modules/ParserBase.rsc|(9,245,<2,8>,<11,4>))
Job: Compiling ParserBase
pathConfig(
  bin=|memory:///test-modules/rascal-tests-bin-3|,
  generatedSources=|memory:///test-modules/generated-test-sources-3|,
  libs=[|mvn://org.rascalmpl--rascal--0.42.0-RC2|],
  srcs=[|memory:///test-modules/|],
  generatedResources=|memory:///test-modules/generated-test-resources-3|)
Job: Compiling ParserBase FileLocations Debugging FileUtility ListUtility StringUtility MathUtility
Job: Compiling ParserBase
error("Cannot add grammar or tmodel since `vis::ParseTree` is not found",|memory:///test-modules/ParserBase.rsc|(9,245,<2,8>,<11,4>))
Job: Compiling ParserBase
pathConfig(
  bin=|memory:///test-modules/rascal-tests-bin-4|,
  generatedSources=|memory:///test-modules/generated-test-sources-4|,
  libs=[|mvn://org.rascalmpl--rascal--0.42.0-RC2|],
  srcs=[|memory:///test-modules/|],
  generatedResources=|memory:///test-modules/generated-test-resources-4|)
Job: Compiling ParserBase FileLocations Debugging FileUtility ListUtility StringUtility MathUtility
Job: Compiling ParserBase
error("Cannot add grammar or tmodel since `vis::ParseTree` is not found",|memory:///test-modules/ParserBase.rsc|(9,245,<2,8>,<11,4>))
Job: Compiling ParserBase
pathConfig(
  bin=|memory:///test-modules/rascal-tests-bin-5|,
  generatedSources=|memory:///test-modules/generated-test-sources-5|,
  libs=[|mvn://org.rascalmpl--rascal--0.42.0-RC2|],
  srcs=[|memory:///test-modules/|],
  generatedResources=|memory:///test-modules/generated-test-resources-5|)
Job: Compiling ParserBase FileLocations Debugging FileUtility ListUtility StringUtility MathUtility
Job: Compiling ParserBase
error("Cannot add grammar or tmodel since `vis::ParseTree` is not found",|memory:///test-modules/ParserBase.rsc|(9,245,<2,8>,<11,4>))
Job: Compiling ParserBase
pathConfig(
  bin=|memory:///test-modules/rascal-tests-bin-6|,
  generatedSources=|memory:///test-modules/generated-test-sources-6|,
  libs=[|mvn://org.rascalmpl--rascal--0.42.0-RC2|],
  srcs=[|memory:///test-modules/|],
  generatedResources=|memory:///test-modules/generated-test-resources-6|)
Job: Compiling ParserBase FileLocations Debugging FileUtility ListUtility StringUtility MathUtility
Job: Compiling ParserBase
error("Cannot add grammar or tmodel since `vis::ParseTree` is not found",|memory:///test-modules/ParserBase.rsc|(9,245,<2,8>,<11,4>))
Job: Compiling ParserBase
pathConfig(
  bin=|memory:///test-modules/rascal-tests-bin-7|,
  generatedSources=|memory:///test-modules/generated-test-sources-7|,
  libs=[|mvn://org.rascalmpl--rascal--0.42.0-RC2|],
  srcs=[|memory:///test-modules/|],
  generatedResources=|memory:///test-modules/generated-test-resources-7|)
Job: Compiling ParserBase FileLocations Debugging FileUtility ListUtility StringUtility MathUtility
Job: Compiling ParserBase
error("Cannot add grammar or tmodel since `vis::ParseTree` is not found",|memory:///test-modules/ParserBase.rsc|(9,245,<2,8>,<11,4>))
Job: Compiling ParserBase
pathConfig(
  bin=|memory:///test-modules/rascal-tests-bin-8|,
  generatedSources=|memory:///test-modules/generated-test-sources-8|,
  libs=[|mvn://org.rascalmpl--rascal--0.42.0-RC2|],
  srcs=[|memory:///test-modules/|],
  generatedResources=|memory:///test-modules/generated-test-resources-8|)
Job: Compiling ParserBase FileLocations Debugging FileUtility ListUtility StringUtility MathUtility
Job: Compiling A
Job: Compiling B
Job: Compiling B
error("Cannot add grammar or tmodel since `A` is not found",|memory:///test-modules/B.rsc|(0,53,<1,0>,<1,53>))
Job: Compiling A
Job: Compiling B
Job: Compiling E A B C D E
Job: Compiling C B
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling A
Job: Compiling B
Job: Compiling B
error("Cannot add grammar or tmodel since `A` is not found",|memory:///test-modules/B.rsc|(0,18,<1,0>,<1,18>))
Job: Compiling A
Job: Compiling B
Job: Compiling A
Job: Compiling A A
[ERROR] Tests run: 41, Failures: 0, Errors: 1, Skipped: 5, Time elapsed: 238.7 s <<< FAILURE! -- in org.rascalmpl.compiler.lang.rascalcore.check.tests.RunTests
[ERROR] org.rascalmpl.compiler.lang.rascalcore.check.tests.RunTests.fixedErrorsDisappear: <15303,2267> -- Time elapsed: 0.005 s <<< ERROR!
org.rascalmpl.exceptions.Throw: /home/davy/swat.engineering/rascal/rascal/src/org/rascalmpl/compiler/lang/rascalcore/check/tests/StaticTestingUtils.rsc:160,24: "{}"
        at org.rascalmpl.interpreter.control_exceptions.ControlException.reallyFillInStackTrace(ControlException.java:42)
        at org.rascalmpl.exceptions.Throw.fillInStackTrace(Throw.java:138)
        at java.base/java.lang.Throwable.<init>(Throwable.java:270)
        at java.base/java.lang.Exception.<init>(Exception.java:66)
        at java.base/java.lang.RuntimeException.<init>(RuntimeException.java:62)
        at org.rascalmpl.interpreter.control_exceptions.ControlException.<init>(ControlException.java:25)
        at org.rascalmpl.exceptions.Throw.<init>(Throw.java:51)
        at org.rascalmpl.semantics.dynamic.Statement$Throw.interpret(Statement.java:1009)
        at org.rascalmpl.interpreter.result.RascalFunction.runBody(RascalFunction.java:429)
        at org.rascalmpl.interpreter.result.RascalFunction.call(RascalFunction.java:352)
        at org.rascalmpl.interpreter.result.OverloadedFunction.callWith(OverloadedFunction.java:409)
        at org.rascalmpl.interpreter.result.OverloadedFunction.call(OverloadedFunction.java:369)
        at org.rascalmpl.semantics.dynamic.Expression$CallOrTree.interpret(Expression.java:542)
        at org.rascalmpl.semantics.dynamic.Statement$Expression.interpret(Statement.java:390)
        at org.rascalmpl.semantics.dynamic.Statement$Return.interpret(Statement.java:851)
        at org.rascalmpl.interpreter.result.RascalFunction.runBody(RascalFunction.java:429)
        at org.rascalmpl.interpreter.result.RascalFunction.call(RascalFunction.java:352)
        at org.rascalmpl.interpreter.result.OverloadedFunction.callWith(OverloadedFunction.java:409)
        at org.rascalmpl.interpreter.result.OverloadedFunction.call(OverloadedFunction.java:369)
        at org.rascalmpl.semantics.dynamic.Expression$CallOrTree.interpret(Expression.java:542)
        at org.rascalmpl.semantics.dynamic.Statement$AssertWithMessage.interpret(Statement.java:178)
        at org.rascalmpl.interpreter.result.RascalFunction.runBody(RascalFunction.java:429)
        at org.rascalmpl.interpreter.result.RascalFunction.call(RascalFunction.java:320)
        at org.rascalmpl.interpreter.result.ICallableValue.call(ICallableValue.java:50)
        at org.rascalmpl.interpreter.result.AbstractFunction.call(AbstractFunction.java:208)
        at org.rascalmpl.values.functions.IFunction.call(IFunction.java:48)
        at org.rascalmpl.interpreter.TestEvaluator.lambda$runTests$0(TestEvaluator.java:125)
        at org.rascalmpl.test.infrastructure.QuickCheck.test(QuickCheck.java:112)
        at org.rascalmpl.interpreter.TestEvaluator.lambda$runTests$1(TestEvaluator.java:123)
        at org.rascalmpl.debug.IRascalMonitor.job(IRascalMonitor.java:68)
        at org.rascalmpl.interpreter.TestEvaluator.runTests(TestEvaluator.java:94)
        at org.rascalmpl.interpreter.TestEvaluator.test(TestEvaluator.java:54)
        at org.rascalmpl.test.infrastructure.RascalJUnitTestRunner.run(RascalJUnitTestRunner.java:236)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
        at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
        at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)

[DavyLandman]

After a day of running test, I'va managed to reduce the test to:

test bool fixedErrorsDisappear2() {  // ht @toinehartman
    clearMemory();
    pcfg = getDefaultTestingPathConfig();

    mlocs = writeModules("
        module ParserBase
    ");

    assert checkModulesOK(mlocs, pathConfig=pcfg) : "Precondition failed: no errors expected!";

    // Introduce a type error (import of module that does not exist)
    l = writeModule("
        module ParserBase

        import vis::ParseTree; // module does not exist -\> error

    ");

    assert missingModuleInModule(l, pathConfig=pcfg) : "Precondition failed: expected at least one error, but got none!";

    return true;
}

its on the branch fix/breaking-compiler-test-2710.

there is a java script to hit it (BreakTest) but I can now also trigger it in the repl:

rascal> import lang::rascalcore::check::tests::ChangeScenarioTests;
ok

rascal>for (i <- [0..40]) fixedErrorsDisappear2();

|file:///C:/Users/Davy/swat.engineering/rascal/rascal/src/org/rascalmpl/compiler/lang/rascalcore/check/tests/StaticTestingUtils.rsc|(5498,2,<162,24>,<162,26>): "{}"
        at checkModuleAndFilter(|file:///C:/Users/Davy/swat.engineering/rascal/rascal/src/org/rascalmpl/compiler/lang/rascalcore/check/tests/StaticTestingUtils.rsc|(4771,733,<143,0>,<163,1>))
        at missingModuleInModule(|file:///C:/Users/Davy/swat.engineering/rascal/rascal/src/org/rascalmpl/compiler/lang/rascalcore/check/tests/StaticTestingUtils.rsc|(18329,166,<492,0>,<493,74>))
        at fixedErrorsDisappear2(|file:///C:/Users/Davy/swat.engineering/rascal/rascal/src/org/rascalmpl/compiler/lang/rascalcore/check/tests/ChangeScenarioTests.rsc|(15303,613,<472,0>,<493,1>))
        at $(|prompt:///|(19,24,<1,19>,<1,43>))