CFE-3041: testall should fail a regular acceptance test if it prints errors

[sarakthon]

Ticket: CFE-3041

Running testall before changes (just a snippet from terminal):

./00_basics/01_compiler/string_contexts.cf Pass
./00_basics/01_compiler/string_contexts.x.cf Pass
./00_basics/01_compiler/trailing_comma_args FAIL (UNEXPECTED FAILURE)
2.cf FAIL (UNEXPECTED FAILURE)
./00_basics/01_compiler/trailing_comma_paramlist FAIL (UNEXPECTED FAILURE)
2.cf FAIL (UNEXPECTED FAILURE)
./00_basics/01_compiler/trailing_commas FAIL (UNEXPECTED FAILURE)
2.cf FAIL (UNEXPECTED FAILURE)
./00_basics/01_compiler/trailing_commas.cf Pass
./00_basics/01_compiler/with_iteration.cf Pass
./00_basics/02_switches/001.cf Pass
./00_basics/02_switches/002.cf Pass
./00_basics/02_switches/003.cf Pass
./00_basics/02_switches/004.cf Pass
./00_basics/02_switches/005.cf Pass
./00_basics/02_switches/006.cf Pass
./00_basics/02_switches/008.cf Pass
./00_basics/02_switches/010.cf Pass
./00_basics/02_switches/dry_run_perms_doesnt_lie.cf Pass
./00_basics/02_switches/show-evaluated-vars.cf Pass
./00_basics/02_switches/staging/007.cf Skipped (Staging tests are disabled)
./00_basics/02_switches/staging/009.cf Skipped (Staging tests are disabled)
./00_basics/02_switches/staging/negate_hard_class.cf Skipped (Staging tests are disabled)
./00_basics/03_bodies/001.cf Pass
./00_basics/03_bodies/002.cf Pass
./00_basics/03_bodies/003.cf Pass
./00_basics/03_bodies/101.cf Pass
./00_basics/03_bodies/102.cf Pass
./00_basics/03_bodies/103.cf Pass
./00_basics/03_bodies/104.cf Pass
./00_basics/03_bodies/109.cf Pass
./00_basics/03_bodies/112.cf Pass
./00_basics/03_bodies/113.cf Pass
./00_basics/03_bodies/114.cf Pass
./00_basics/03_bodies/115.cf Pass
./00_basics/03_bodies/116.cf Pass
./00_basics/03_bodies/117.cf Pass
./00_basics/03_bodies/118.cf Pass
./00_basics/03_bodies/119.cf Pass
./00_basics/03_bodies/120.cf Pass
./00_basics/03_bodies/121.cf Pass
...
...
Passed tests:    1632
Failed tests:    130
Skipped tests:   235
Soft failures:   75
Flakey failures: 0
Total tests:     2072

After changes:

./00_basics/01_compiler/string_contexts.cf FAIL (Test passed, but printed error message(s)) (UNEXPECTED FAILURE)
./00_basics/01_compiler/string_contexts.x.cf Pass
./00_basics/01_compiler/trailing_comma_args FAIL (UNEXPECTED FAILURE)
2.cf FAIL (UNEXPECTED FAILURE)
./00_basics/01_compiler/trailing_comma_paramlist FAIL (UNEXPECTED FAILURE)
2.cf FAIL (UNEXPECTED FAILURE)
./00_basics/01_compiler/trailing_commas FAIL (UNEXPECTED FAILURE)
2.cf FAIL (UNEXPECTED FAILURE)
./00_basics/01_compiler/trailing_commas.cf Pass
./00_basics/01_compiler/with_iteration.cf Pass
./00_basics/02_switches/001.cf Pass
./00_basics/02_switches/002.cf Pass
./00_basics/02_switches/003.cf Pass
./00_basics/02_switches/004.cf Pass
./00_basics/02_switches/005.cf Pass
./00_basics/02_switches/006.cf Pass
./00_basics/02_switches/008.cf Pass
./00_basics/02_switches/010.cf Pass
./00_basics/02_switches/dry_run_perms_doesnt_lie.cf Pass
./00_basics/02_switches/show-evaluated-vars.cf Pass
./00_basics/02_switches/staging/007.cf Skipped (Staging tests are disabled)
./00_basics/02_switches/staging/009.cf Skipped (Staging tests are disabled)
./00_basics/02_switches/staging/negate_hard_class.cf Skipped (Staging tests are disabled)
./00_basics/03_bodies/001.cf Pass
./00_basics/03_bodies/002.cf FAIL (Test passed, but printed error message(s)) (UNEXPECTED FAILURE)
./00_basics/03_bodies/003.cf Pass
./00_basics/03_bodies/101.cf Pass
./00_basics/03_bodies/102.cf Pass
./00_basics/03_bodies/103.cf Pass
./00_basics/03_bodies/104.cf Pass
./00_basics/03_bodies/109.cf Pass
./00_basics/03_bodies/112.cf FAIL (Test passed, but printed error message(s)) (UNEXPECTED FAILURE)
./00_basics/03_bodies/113.cf FAIL (Test passed, but printed error message(s)) (UNEXPECTED FAILURE)
./00_basics/03_bodies/114.cf Pass
./00_basics/03_bodies/115.cf Pass
./00_basics/03_bodies/116.cf Pass
./00_basics/03_bodies/117.cf Pass
./00_basics/03_bodies/118.cf FAIL (Test passed, but printed error message(s)) (UNEXPECTED FAILURE)
./00_basics/03_bodies/119.cf FAIL (Test passed, but printed error message(s)) (UNEXPECTED FAILURE)
./00_basics/03_bodies/120.cf FAIL (Test passed, but printed error message(s)) (UNEXPECTED FAILURE)
./00_basics/03_bodies/121.cf FAIL (Test passed, but printed error message(s)) (UNEXPECTED FAILURE)
...
...
Passed tests:    1319
Failed tests:    442
Skipped tests:   235
Soft failures:   75
Flakey failures: 0
Total tests:     2071

Next I'm going to look into the tests that now fail with these new changes, and investigate the error messages.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}