|
| 1 | +# Feature proposal: `expectFailure` label & matcher |
| 2 | + |
| 3 | +## Summary |
| 4 | + |
| 5 | +Update the `expectFailure` option in `test()` to accept different types of values, enabling both **custom label** and **error matching**. This proposal integrates the requirements from [nodejs/node#61570](https://github.com/nodejs/node/issues/61570), ensuring consistency with `skip`/`todo` while adding robust validation capabilities. |
| 6 | + |
| 7 | +## API & Behavior |
| 8 | + |
| 9 | +The behavior of `expectFailure` is strictly determined by the type of value provided: |
| 10 | + |
| 11 | +### String: Failure label |
| 12 | + |
| 13 | +When a **non-empty** `string` is provided, it acts as an output label (reason) for that test-case (identical to `skip` and `todo` options). |
| 14 | + |
| 15 | +```js |
| 16 | +test('fails with a specific reason', { |
| 17 | + expectFailure: 'Bug #123: Feature not implemented yet' |
| 18 | +}, () => { |
| 19 | + throw new Error('boom'); |
| 20 | +}); |
| 21 | +``` |
| 22 | +- **Behavior**: The test is expected to fail. The string is treated as a label/reason. |
| 23 | +- **Validation**: None. It accepts _any_ error. |
| 24 | +- **Output**: The reporter displays the string literally, prefixed by a TAP comment character (e.g., `# EXPECTED FAILURE Bug #123…`). |
| 25 | + |
| 26 | +### Matcher: Error constructor, Error-like object, RegExp, or validation function |
| 27 | + |
| 28 | +When an `Error` constructor, `Error`-like object, `RegExp`, or validation function is provided, it is treated as validation to match against, mirroring the behaviour of [`assert.throws`](https://nodejs.org/docs/latest-v25.x/api/assert.html#assertthrowsfn-error-message) (possibly leveraging it under the hood). |
| 29 | + |
| 30 | +```js |
| 31 | +test('fails with matching error (RegExp)', { |
| 32 | + expectFailure: /expected error message/ |
| 33 | +}, () => { |
| 34 | + throw new Error('this is the expected error message'); |
| 35 | +}); |
| 36 | + |
| 37 | +test('fails with matching error (Class)', { |
| 38 | + expectFailure: RangeError |
| 39 | +}, () => { |
| 40 | + throw new RangeError('Index out of bounds'); |
| 41 | +}); |
| 42 | +``` |
| 43 | + |
| 44 | +### Configuration Object: Reason & Validation |
| 45 | + |
| 46 | +When a **Plain Object** with specific properties (`match`, `label`) is provided, it allows specifying both a failure reason and validation logic simultaneously. |
| 47 | + |
| 48 | +```js |
| 49 | +test('fails with reason and specific error', { |
| 50 | + expectFailure: { |
| 51 | + label: 'Bug #123: Edge case behavior', // Reason |
| 52 | + match: /Index out of bounds/ // Validation |
| 53 | + } |
| 54 | +}, () => { |
| 55 | + throw new RangeError('Index out of bounds'); |
| 56 | +}); |
| 57 | +``` |
| 58 | +- **Properties**: |
| 59 | + - `label` (String): The failure reason/label (displayed in reporter). |
| 60 | + - `match` (RegExp | Object | Function | Class): Validation logic. This is passed directly to `assert.throws` validation argument, supporting all its capabilities. |
| 61 | +- **Requirement**: The object must contain at least one of `label` or `match`. |
| 62 | +- **Behavior**: The test passes **only if** the error matches the `match` criteria. |
| 63 | +- **Output**: The reporter displays the `label`. |
| 64 | + |
| 65 | +### Equivalence |
| 66 | + |
| 67 | +The following configurations are equivalent in behavior: |
| 68 | + |
| 69 | +**1. Reason only:** |
| 70 | +```js |
| 71 | +expectFailure: 'reason'; |
| 72 | +``` |
| 73 | + |
| 74 | +**2. Validation only:** |
| 75 | +```js |
| 76 | +expectFailure: /error/ |
| 77 | +expectFailure: { match: /error/ } |
| 78 | +``` |
| 79 | + |
| 80 | +**3. Catch-all (Any Error):** |
| 81 | +```js |
| 82 | +expectFailure: true |
| 83 | +``` |
| 84 | + |
| 85 | +## Ambiguity Resolution |
| 86 | +Potential ambiguity between a **Matcher Object** and a **Configuration Object** is resolved as follows: |
| 87 | + |
| 88 | +1. **String** → Reason. |
| 89 | +2. **RegExp** or **Function** → Matcher (Validation). |
| 90 | +3. **Object**: |
| 91 | + * **Empty Object** (`{}`) → **Error**: throws `ERR_INVALID_ARG_VALUE`. |
| 92 | + ```js |
| 93 | + // Uses Node.js standard error code |
| 94 | + throw new ERR_INVALID_ARG_VALUE( |
| 95 | + 'expectFailure', |
| 96 | + expectFailure, |
| 97 | + 'must not be an empty object' |
| 98 | + ); |
| 99 | + ``` |
| 100 | + * If the object contains **only** `match` and/or `label` properties → **Configuration Object**. |
| 101 | + * Otherwise → **Matcher Object** (passed to `assert.throws` for property matching). |
| 102 | + |
| 103 | +## Activation & Truthiness |
| 104 | + |
| 105 | +To maintain strict consistency with `todo` and `skip` options: |
| 106 | +* The feature is **disabled** only if `expectFailure` is `undefined` or `false`. |
| 107 | +* **All other values** enable the feature (treat as truthy). |
| 108 | + * `expectFailure: ''` (Empty String) → **Enabled** (treats as generic failure expectation). |
| 109 | + * `expectFailure: 0` → **Enabled** (treated as a Matcher Object unless specific logic excludes numbers, but per consistency it enables the feature). |
| 110 | + |
| 111 | +### Flat Options (`expectFailureError`) |
| 112 | + |
| 113 | +It was proposed to split the options into `expectFailure` (reason) and `expectFailureError` (validation). |
| 114 | +This was rejected in favor of the nested/polymorphic structure using `match` and `label` properties. This syntax was selected as the preferred choice for its readability and clarity: |
| 115 | +* `match`: Clearly indicates "fails **matching** this error" (Validation). |
| 116 | +* `label`: Clearly indicates the **label** or reason for the expected failure. |
| 117 | +This approach keeps related configuration grouped without polluting the top-level options namespace. |
| 118 | + |
| 119 | +## Implementation Details |
| 120 | + |
| 121 | +### Validation Logic |
| 122 | + |
| 123 | +The implementation leverages `assert.throws` internally to perform error validation. |
| 124 | +- If `expectFailure` is a Matcher (RegExp, Class, Object), it is passed as the second argument to `assert.throws(fn, expectFailure)`. |
| 125 | +- If `expectFailure` is a Configuration Object, `expectFailure.match` is passed to `assert.throws`. |
0 commit comments