Skip to content

Improve assertions error messages with structured format#7444

Open
Evangelink wants to merge 62 commits intomainfrom
dev/amauryleve/rework-assert
Open

Improve assertions error messages with structured format#7444
Evangelink wants to merge 62 commits intomainfrom
dev/amauryleve/rework-assert

Conversation

@Evangelink
Copy link
Copy Markdown
Member

@Evangelink Evangelink commented Feb 20, 2026

Fix #7421

Copilot AI review requested due to automatic review settings February 20, 2026 21:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request modernizes assertion error messages by introducing a structured, multi-line format that improves readability and developer experience. The changes replace old-style concatenated messages with formatted parameter displays using raw string literals.

Changes:

  • Introduced structured error message formatting with parameter names, expressions, and values on separate lines
  • Added helper methods for value formatting, expression truncation, and redundancy detection
  • Updated all assertion methods to use the new formatting approach
  • Removed obsolete resource strings and added new simplified ones
  • Updated all test expectations to match the new message format

Reviewed changes

Copilot reviewed 41 out of 41 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
Assert.cs Core formatting infrastructure: FormatValue, FormatParameter, truncation logic
Assert.ThrowsException.cs Updated to new structured format for exception type mismatches
Assert.StartsWith/EndsWith/Matches.cs String assertion methods using new format
Assert.IsTrue/IsFalse.cs Boolean assertion methods with condition parameter display
Assert.IsNull/IsNotNull.cs Null checking assertions with value display
Assert.IsInstanceOfType/IsExactInstanceOfType.cs Type checking with structured type comparison
Assert.IComparable.cs Comparison assertions (greater/less than, positive/negative)
Assert.Count/Contains.cs Collection assertions with expression parameters
Assert.AreSame.cs Reference equality with hash code display for disambiguation
FrameworkMessages.resx Resource string simplification and new message keys
xlf files Localization files marked with untranslated entries
Test files Updated expectations for all assertion error messages

@Evangelink Evangelink force-pushed the dev/amauryleve/rework-assert branch 2 times, most recently from a55c762 to 49018c6 Compare February 22, 2026 10:31
Copilot AI review requested due to automatic review settings February 22, 2026 10:31
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated 7 comments.

@Evangelink Evangelink force-pushed the dev/amauryleve/rework-assert branch from 49018c6 to 63e8257 Compare February 22, 2026 10:38
Copilot AI review requested due to automatic review settings February 22, 2026 12:56
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated no new comments.

@nohwnd
Copy link
Copy Markdown
Member

nohwnd commented Feb 23, 2026

looks great

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 53 out of 53 changed files in this pull request and generated 3 comments.

- OutputTests: Update expected AreEqual message to new callsite format
- TrxReportTests: Update expected AreEqual message with aligned spacing
- RetryTests: Split Assert.Fail message check (no longer 'Assert.Fail failed.')
- GenericTestMethodTests: Update regex to match new Assert.Fail format (callsite on separate line)
- Fix FormatCollectionPreview: Use failedEnumeration flag instead of forcing truncated=true, avoiding misleading '... 0+ more' suffix on exception
- Fix CollectionAssert.AreEquivalent null check: Use dedicated 'Expected collections to be equivalent.' message with expected/actual nullness instead of generic AreEqualFailNew
- Fix FormatValue culture: Use Convert.ToString with InvariantCulture for primitive types to ensure consistent output across locales
Copilot AI review requested due to automatic review settings March 23, 2026 15:43
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 55 out of 55 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

src/TestFramework/TestFramework/Assertions/Assert.cs:120

  • FormatValue treats any IEnumerable as a collection preview and will enumerate it to build the message. This can consume one-shot / lazy / side-effecting enumerables (and may be inconsistent with FormatCollectionParameter, which avoids enumerating non-ICollection sequences). Consider only previewing ICollection (or materialized snapshots) and falling back to type name for non-ICollection enumerables to avoid changing runtime behavior during message formatting.
        // For collections, show a preview with element values
        if (value is IEnumerable enumerable)
        {
            return FormatCollectionPreview(enumerable);
        }

- Fix ReplaceNulls: Return 'null' instead of empty string for null input
- Move AreSame equality hints to resx (AreSameObjectsAreEqualHint/AreSameObjectsAreNotEqualHint)
- Move AreEquivalent null mismatch message to resx (AreEquivalentNullMismatchFailNew)
- Move Assert.That comparison/bool messages to resx (AssertThatEqualFailNew, AssertThatNotEqualFailNew, AssertThatGreaterThanFailNew, etc.)
- All user-facing strings now go through FrameworkMessages for localization
- ContainsSingle predicate tests: Add wildcard for new collection preview line
- Items interpolated string tests: Remove collection lines, add wildcards
- IsInRange DateTime: Use wildcards for InvariantCulture date format
- AreEqual culture: Add ... to expected callsite for culture overload
- Assert.That char: Update expected values for InvariantCulture char formatting
Copilot AI review requested due to automatic review settings March 23, 2026 16:51
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 56 out of 56 changed files in this pull request and generated 3 comments.

Copilot AI review requested due to automatic review settings March 23, 2026 17:02
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 56 out of 56 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (4)

src/TestFramework/TestFramework/Assertions/Assert.Contains.cs:309

  • The non-generic Contains(object?, IEnumerable) eagerly materializes non-ICollection inputs via ToList() before searching. This changes passing behavior (a match near the start no longer short-circuits) and can cause large/infinite/side-effecting sequences to be fully enumerated unnecessarily. Consider scanning the enumerable directly and only materializing/buffering elements when an assertion failure requires a stable snapshot for message formatting.
        // Materialize non-ICollection enumerables to prevent multiple enumeration.
        ICollection snapshot = collection as ICollection ?? collection.Cast<object?>().ToList();

        foreach (object? item in snapshot)
        {
            if (object.Equals(item, expected))
            {
                return;
            }
        }

        ThrowAssertContainsItemFailed(message, expectedExpression, collectionExpression, snapshot);

src/TestFramework/TestFramework/Assertions/Assert.Contains.cs:391

  • Contains<T>(Func<T,bool>, IEnumerable<T>) snapshots non-ICollection enumerables ([.. collection]) before calling Any. That prevents Any from short-circuiting and can introduce significant overhead/side effects on passing assertions. Consider enumerating once and stopping at the first match, buffering only what’s needed for the failure message.
    public static void Contains<T>(Func<T, bool> predicate, IEnumerable<T> collection, string? message = "", [CallerArgumentExpression(nameof(predicate))] string predicateExpression = "", [CallerArgumentExpression(nameof(collection))] string collectionExpression = "")
    {
        ICollection<T> snapshot = collection as ICollection<T> ?? [.. collection];
        if (!snapshot.Any(predicate))
        {
            ThrowAssertContainsPredicateFailed(message, predicateExpression, collectionExpression, snapshot);
        }

src/TestFramework/TestFramework/Assertions/Assert.Contains.cs:524

  • DoesNotContain<T> snapshots non-ICollection enumerables ([.. collection]) before checking Contains. This prevents early-exit on failures (when notExpected appears early) and can fully enumerate expensive/side-effecting sequences unnecessarily. Consider iterating and failing immediately on the first match; only buffer/materialize when needed for formatting the failure message.
    public static void DoesNotContain<T>(T notExpected, IEnumerable<T> collection, string? message = "", [CallerArgumentExpression(nameof(notExpected))] string notExpectedExpression = "", [CallerArgumentExpression(nameof(collection))] string collectionExpression = "")
    {
        ICollection<T> snapshot = collection as ICollection<T> ?? [.. collection];
        if (snapshot.Contains(notExpected))
        {
            ThrowAssertDoesNotContainItemFailed(message, notExpectedExpression, collectionExpression, snapshot);
        }

src/TestFramework/TestFramework/Assertions/Assert.Contains.cs:633

  • DoesNotContain<T>(Func<T,bool>, IEnumerable<T>) snapshots non-ICollection enumerables ([.. collection]) before calling Any. This removes Any’s ability to short-circuit and can fully enumerate sequences even when the predicate matches early (the failing case). Consider scanning the enumerable directly and stopping at the first match, buffering elements only when producing the failure message requires it.
    public static void DoesNotContain<T>(Func<T, bool> predicate, IEnumerable<T> collection, string? message = "", [CallerArgumentExpression(nameof(predicate))] string predicateExpression = "", [CallerArgumentExpression(nameof(collection))] string collectionExpression = "")
    {
        ICollection<T> snapshot = collection as ICollection<T> ?? [.. collection];
        if (snapshot.Any(predicate))
        {
            ThrowAssertDoesNotContainPredicateFailed(message, predicateExpression, collectionExpression, snapshot);
        }

- Add safety comment to TryEvaluateFormatted about re-evaluation side effects
- Fix FormatValue: Pass maxLength to FormatCollectionPreview for consistency
- Fix ThrowAssertIsInRangeFailed: Remove unused minValueExpression/maxValueExpression params
- Fix Contains/DoesNotContain: Avoid eager materialization when ICollection is available
- Fix AllItemsAreUnique: Use FormatValue instead of ReplaceNulls for proper escaping
- Fix IsSubsetOf: Use FormatValue instead of Convert.ToString for null-safe formatting
- Update IsSubsetOf tests for quoted string values
Update tests to match the structured assertion message format where
Assert.Fail no longer includes 'failed.' suffix and uses newline
separator between assertion name and user message.
Copilot AI review requested due to automatic review settings March 24, 2026 09:48
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 60 out of 60 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.cs:45

  • This assertion hard-codes a "\n" line break in the expected exception message, but Assert.ThrowAssertFailed composes messages using Environment.NewLine. This can cause the test to fail on platforms where the newline sequence differs. Consider composing the expected message with Environment.NewLine (or normalizing line endings) to keep it cross-platform.

Copilot AI review requested due to automatic review settings March 26, 2026 13:59
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 59 out of 59 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve AreSame error messages in case of null values

4 participants