Skip to content

CFE-3041: change regular acceptance test to .error.cf - #6334

Open
sarakthon wants to merge 1 commit into
cfengine:masterfrom
sarakthon:acceptance-tests
Open

CFE-3041: change regular acceptance test to .error.cf#6334
sarakthon wants to merge 1 commit into
cfengine:masterfrom
sarakthon:acceptance-tests

Conversation

@sarakthon

@sarakthon sarakthon commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

As far as I understand, this test deliberately causes an error considering it tries to insert lines to a file that does not exist. Based on this I think the test is a potential error.cf candidate.

Before change:
After adding a check to testall that flags tests which report "Pass", but still print error: messages during execution, the following output was observed:

...
./00_basics/03_bodies/112.cf FAIL (Test passed, but printed error message(s)) (UNEXPECTED FAILURE)
...

Passed tests:    1432
Failed tests:    340
Skipped tests:   238
Soft failures:   39
Flakey failures: 0
Total tests:     2049

The output in error.log before changing the filename to error.cf was as follows:

----------------------------------------------------------------------
./00_basics/03_bodies/112.cf 
----------------------------------------------------------------------
2026-08-11T21:13:11+0200    error: File '/home/ubuntu/core/tests/acceptance/workdir/__00_basics_03_bodies_112_cf/tmp/TEST.cfengine' was marked for editing but could not be opened
2026-08-11T21:13:11+0200    error: Errors encountered when actuating files promise '/home/ubuntu/core/tests/acceptance/workdir/__00_basics_03_bodies_112_cf/tmp/TEST.cfengine'
2026-08-11T21:13:11+0200    error: Method 'init' failed in some repairs
R: ok: class 'repair_failed' was set (should be)
R: ok: class 'cancel_kept' was set (should be)
R: ok: class 'cancel_repaired' was set (should be)
R: ok: class 'promise_kept' was not set (should not be)
R: ok: class 'promise_repaired' was not set (should not be)
R: ok: class 'repair_denied' was not set (should not be)
R: ok: class 'repair_timeout' was not set (should not be)
R: ok: class 'cancel_notkept' was not set (should not be)
R: /home/ubuntu/core/tests/acceptance/./00_basics/03_bodies/112.cf Pass
2026-08-11T21:13:11+0200    error: Method 'test_run' failed in some repairs

Return code is 0.

  ==> FAIL (Test passed, but printed error message(s)) (UNEXPECTED FAILURE)

After change:
Output when running testall after changing filename to error.cf:

...
./00_basics/03_bodies/112.error.cf Pass
...
Passed tests:    1433
Failed tests:    339
Skipped tests:   238
Soft failures:   39
Flakey failures: 0
Total tests:     2049

Output in error.log after changing filename to error.cf:

----------------------------------------------------------------------
./00_basics/03_bodies/112.error.cf (should log error(s), but exit with 0)
----------------------------------------------------------------------
2026-08-11T22:02:56+0200    error: File '/home/ubuntu/core/tests/acceptance/workdir/__00_basics_03_bodies_112_error_cf/tmp/TEST.cfengine' was marked for editing but could not be opened
2026-08-11T22:02:56+0200    error: Errors encountered when actuating files promise '/home/ubuntu/core/tests/acceptance/workdir/__00_basics_03_bodies_112_error_cf/tmp/TEST.cfengine'
2026-08-11T22:02:56+0200    error: Method 'init' failed in some repairs
R: ok: class 'repair_failed' was set (should be)
R: ok: class 'cancel_kept' was set (should be)
R: ok: class 'cancel_repaired' was set (should be)
R: ok: class 'promise_kept' was not set (should not be)
R: ok: class 'promise_repaired' was not set (should not be)
R: ok: class 'repair_denied' was not set (should not be)
R: ok: class 'repair_timeout' was not set (should not be)
R: ok: class 'cancel_notkept' was not set (should not be)
R: /home/ubuntu/core/tests/acceptance/./00_basics/03_bodies/112.error.cf Pass
2026-08-11T22:02:56+0200    error: Method 'test_run' failed in some repairs

Return code is 0.

  ==> Pass

@cf-bottom

Copy link
Copy Markdown

Thank you for submitting a PR! Maybe @larsewi can review this?

@sarakthon sarakthon changed the title CFE-3041: testall script should fail a regular acceptance test if it prints errors CFE-3041: change regular acceptance test to .error.cf Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants