Improve assertions error messages with structured format#7444
Improve assertions error messages with structured format#7444Evangelink wants to merge 62 commits intomainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request modernizes assertion error messages by introducing a structured, multi-line format that improves readability and developer experience. The changes replace old-style concatenated messages with formatted parameter displays using raw string literals.
Changes:
- Introduced structured error message formatting with parameter names, expressions, and values on separate lines
- Added helper methods for value formatting, expression truncation, and redundancy detection
- Updated all assertion methods to use the new formatting approach
- Removed obsolete resource strings and added new simplified ones
- Updated all test expectations to match the new message format
Reviewed changes
Copilot reviewed 41 out of 41 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| Assert.cs | Core formatting infrastructure: FormatValue, FormatParameter, truncation logic |
| Assert.ThrowsException.cs | Updated to new structured format for exception type mismatches |
| Assert.StartsWith/EndsWith/Matches.cs | String assertion methods using new format |
| Assert.IsTrue/IsFalse.cs | Boolean assertion methods with condition parameter display |
| Assert.IsNull/IsNotNull.cs | Null checking assertions with value display |
| Assert.IsInstanceOfType/IsExactInstanceOfType.cs | Type checking with structured type comparison |
| Assert.IComparable.cs | Comparison assertions (greater/less than, positive/negative) |
| Assert.Count/Contains.cs | Collection assertions with expression parameters |
| Assert.AreSame.cs | Reference equality with hash code display for disambiguation |
| FrameworkMessages.resx | Resource string simplification and new message keys |
| xlf files | Localization files marked with untranslated entries |
| Test files | Updated expectations for all assertion error messages |
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsInstanceOfTypeTests.cs
Show resolved
Hide resolved
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsExactInstanceOfTypeTests.cs
Show resolved
Hide resolved
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreSame.cs
Outdated
Show resolved
Hide resolved
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsExactInstanceOfTypeTests.cs
Show resolved
Hide resolved
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsTrueTests.cs
Show resolved
Hide resolved
a55c762 to
49018c6
Compare
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.cs
Outdated
Show resolved
Hide resolved
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.cs
Outdated
Show resolved
Hide resolved
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hans.xlf
Show resolved
Hide resolved
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pt-BR.xlf
Show resolved
Hide resolved
49018c6 to
63e8257
Compare
|
looks great |
- OutputTests: Update expected AreEqual message to new callsite format - TrxReportTests: Update expected AreEqual message with aligned spacing - RetryTests: Split Assert.Fail message check (no longer 'Assert.Fail failed.') - GenericTestMethodTests: Update regex to match new Assert.Fail format (callsite on separate line)
- Fix FormatCollectionPreview: Use failedEnumeration flag instead of forcing truncated=true, avoiding misleading '... 0+ more' suffix on exception - Fix CollectionAssert.AreEquivalent null check: Use dedicated 'Expected collections to be equivalent.' message with expected/actual nullness instead of generic AreEqualFailNew - Fix FormatValue culture: Use Convert.ToString with InvariantCulture for primitive types to ensure consistent output across locales
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 55 out of 55 changed files in this pull request and generated 5 comments.
Comments suppressed due to low confidence (1)
src/TestFramework/TestFramework/Assertions/Assert.cs:120
FormatValuetreats anyIEnumerableas a collection preview and will enumerate it to build the message. This can consume one-shot / lazy / side-effecting enumerables (and may be inconsistent withFormatCollectionParameter, which avoids enumerating non-ICollectionsequences). Consider only previewingICollection(or materialized snapshots) and falling back to type name for non-ICollectionenumerables to avoid changing runtime behavior during message formatting.
// For collections, show a preview with element values
if (value is IEnumerable enumerable)
{
return FormatCollectionPreview(enumerable);
}
- Fix ReplaceNulls: Return 'null' instead of empty string for null input - Move AreSame equality hints to resx (AreSameObjectsAreEqualHint/AreSameObjectsAreNotEqualHint) - Move AreEquivalent null mismatch message to resx (AreEquivalentNullMismatchFailNew) - Move Assert.That comparison/bool messages to resx (AssertThatEqualFailNew, AssertThatNotEqualFailNew, AssertThatGreaterThanFailNew, etc.) - All user-facing strings now go through FrameworkMessages for localization
- ContainsSingle predicate tests: Add wildcard for new collection preview line - Items interpolated string tests: Remove collection lines, add wildcards - IsInRange DateTime: Use wildcards for InvariantCulture date format - AreEqual culture: Add ... to expected callsite for culture overload - Assert.That char: Update expected values for InvariantCulture char formatting
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 56 out of 56 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (4)
src/TestFramework/TestFramework/Assertions/Assert.Contains.cs:309
- The non-generic
Contains(object?, IEnumerable)eagerly materializes non-ICollectioninputs viaToList()before searching. This changes passing behavior (a match near the start no longer short-circuits) and can cause large/infinite/side-effecting sequences to be fully enumerated unnecessarily. Consider scanning the enumerable directly and only materializing/buffering elements when an assertion failure requires a stable snapshot for message formatting.
// Materialize non-ICollection enumerables to prevent multiple enumeration.
ICollection snapshot = collection as ICollection ?? collection.Cast<object?>().ToList();
foreach (object? item in snapshot)
{
if (object.Equals(item, expected))
{
return;
}
}
ThrowAssertContainsItemFailed(message, expectedExpression, collectionExpression, snapshot);
src/TestFramework/TestFramework/Assertions/Assert.Contains.cs:391
Contains<T>(Func<T,bool>, IEnumerable<T>)snapshots non-ICollectionenumerables ([.. collection]) before callingAny. That preventsAnyfrom short-circuiting and can introduce significant overhead/side effects on passing assertions. Consider enumerating once and stopping at the first match, buffering only what’s needed for the failure message.
public static void Contains<T>(Func<T, bool> predicate, IEnumerable<T> collection, string? message = "", [CallerArgumentExpression(nameof(predicate))] string predicateExpression = "", [CallerArgumentExpression(nameof(collection))] string collectionExpression = "")
{
ICollection<T> snapshot = collection as ICollection<T> ?? [.. collection];
if (!snapshot.Any(predicate))
{
ThrowAssertContainsPredicateFailed(message, predicateExpression, collectionExpression, snapshot);
}
src/TestFramework/TestFramework/Assertions/Assert.Contains.cs:524
DoesNotContain<T>snapshots non-ICollectionenumerables ([.. collection]) before checkingContains. This prevents early-exit on failures (whennotExpectedappears early) and can fully enumerate expensive/side-effecting sequences unnecessarily. Consider iterating and failing immediately on the first match; only buffer/materialize when needed for formatting the failure message.
public static void DoesNotContain<T>(T notExpected, IEnumerable<T> collection, string? message = "", [CallerArgumentExpression(nameof(notExpected))] string notExpectedExpression = "", [CallerArgumentExpression(nameof(collection))] string collectionExpression = "")
{
ICollection<T> snapshot = collection as ICollection<T> ?? [.. collection];
if (snapshot.Contains(notExpected))
{
ThrowAssertDoesNotContainItemFailed(message, notExpectedExpression, collectionExpression, snapshot);
}
src/TestFramework/TestFramework/Assertions/Assert.Contains.cs:633
DoesNotContain<T>(Func<T,bool>, IEnumerable<T>)snapshots non-ICollectionenumerables ([.. collection]) before callingAny. This removesAny’s ability to short-circuit and can fully enumerate sequences even when the predicate matches early (the failing case). Consider scanning the enumerable directly and stopping at the first match, buffering elements only when producing the failure message requires it.
public static void DoesNotContain<T>(Func<T, bool> predicate, IEnumerable<T> collection, string? message = "", [CallerArgumentExpression(nameof(predicate))] string predicateExpression = "", [CallerArgumentExpression(nameof(collection))] string collectionExpression = "")
{
ICollection<T> snapshot = collection as ICollection<T> ?? [.. collection];
if (snapshot.Any(predicate))
{
ThrowAssertDoesNotContainPredicateFailed(message, predicateExpression, collectionExpression, snapshot);
}
- Add safety comment to TryEvaluateFormatted about re-evaluation side effects - Fix FormatValue: Pass maxLength to FormatCollectionPreview for consistency - Fix ThrowAssertIsInRangeFailed: Remove unused minValueExpression/maxValueExpression params - Fix Contains/DoesNotContain: Avoid eager materialization when ICollection is available - Fix AllItemsAreUnique: Use FormatValue instead of ReplaceNulls for proper escaping - Fix IsSubsetOf: Use FormatValue instead of Convert.ToString for null-safe formatting - Update IsSubsetOf tests for quoted string values
Update tests to match the structured assertion message format where Assert.Fail no longer includes 'failed.' suffix and uses newline separator between assertion name and user message.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 60 out of 60 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (1)
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.cs:45
- This assertion hard-codes a "\n" line break in the expected exception message, but Assert.ThrowAssertFailed composes messages using Environment.NewLine. This can cause the test to fail on platforms where the newline sequence differs. Consider composing the expected message with Environment.NewLine (or normalizing line endings) to keep it cross-platform.
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TrxReportTests.cs
Show resolved
Hide resolved
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/OutputTests.cs
Outdated
Show resolved
Hide resolved
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.ThrowsExceptionTests.cs
Show resolved
Hide resolved
…-assert # Conflicts: # test/IntegrationTests/MSTest.Acceptance.IntegrationTests/OutputTests.cs
…or cross-platform robustness
Fix #7421