RFC 016: Filter command-line option

[Evangelink]

Draft RFC for #4293.

Summary

Replaces --treenode-filter with a first-class --filter <expression> option, built on a small prefix-routed grammar (kind=value, kind~value, bare value) that aligns with the existing graph-query grammar and reads naturally for users coming from VSTest.

The RFC also:

• Introduces FilterMatchTextProperty as the dedicated default match target for bare filter values (with fallback to DisplayName), so frameworks decide what --filter Foo matches against without us conflating it with display text.
• Introduces ITestNodeFilterKindProvider / TestNodeFilterKind so test frameworks and bridges (first customer: the VSTest bridge with TestCategory, Priority, Owner, Trait) can register their own filter kinds.
• Keeps --treenode-filter as a deprecated alias for --filter Query=... for one release window.

Built-in kinds

DisplayName, Uid, Query (absorbs today's tree-node grammar), Namespace, ClassName, MethodName, FullyQualifiedName (synthesized).

Status

Draft for design review. Implementation work (TestNode property, parser, predicate evaluator, VSTest bridge wiring, MSTest rollout, --help/--info snapshot updates) is intentionally out of scope until the RFC is approved.

Open questions

There are 8 design questions explicitly called out in the RFC's "Open questions" section that I'd like reviewer input on before turning this into code — including bare-value fallback to DisplayName, default case sensitivity, wildcard support, Trait=name=value shape, and the mixed-framework solution scenario.

cc @nohwnd

[Copilot]

Pull request overview

Adds a new draft design RFC (RFC 016) describing a user-friendly --filter <expression> command-line option for Microsoft.Testing.Platform (MTP), intended to supersede --treenode-filter while preserving backward compatibility via a deprecated alias.

Changes:

• Introduces a proposed filter expression grammar with kind-prefixed predicates and a bare-value shorthand.
• Proposes new extensibility and public API surfaces (FilterMatchTextProperty, ITestNodeFilterKindProvider, TestNodeFilterKind, etc.) to let frameworks/bridges register filter kinds.
• Outlines help/info integration, validation rules, migration plan, and open design questions for review.

Show a summary per file

File	Description
docs/RFCs/016-Filter-Command-Line-Option.md	New RFC draft specifying the --filter option design, extensibility model, and migration/back-compat plan.

Copilot's findings

• Files reviewed: 1/1 changed files
• Comments generated: 3

[Evangelink]

On the question of inner alternation/conjunction (TestCategory=A|B instead of TestCategory=A|TestCategory=B):

Short answer: no, not for v1. I've added this as Open Question #9 in the RFC and propose deferring it.

Three approaches were considered:

1. Outer parser splits |/& inside values. Rejected. It forces every kind into the same RHS grammar. Trait=name=value already uses = inside its value, and a Query= value can legitimately contain | (e.g. Query=/asm/test|other/). Making | mean "OR within value" universally breaks both kinds.
2. Per-kind opt-in AllowInnerAlternation flag on TestNodeFilterKind. Workable but adds API surface to v1 and forces a precedence decision — does TestCategory=A|B&Priority=1 mean (TestCategory=A | TestCategory=B) & Priority=1, or TestCategory=A | (TestCategory=B & Priority=1)? Either choice will surprise some users.
3. A separate values-list syntax (e.g. TestCategory=[A,B,C] or TestCategory in (A,B,C)) added later. Non-breaking: the new punctuation is invalid in today's grammar, so we can adopt it later with zero migration cost.

For v1 I'd rather ship the smaller, unambiguous surface and match VSTest exactly (VSTest itself does not support inner | — it requires the verbose TestCategory=A|TestCategory=B form, so we lose no parity). If user feedback shows the verbose form is a real pain point, option 3 is the path I'd take in a follow-up RFC.

See Open Question #9 in 14b11e4 for the same rationale captured in the RFC.

[Evangelink]

@bradwilson @OsirisTerje @thomhurst — pinging you as test framework owners (xUnit / NUnit / TUnit). This RFC proposes a new --filter CLI option for Microsoft.Testing.Platform with a kind-based grammar (Kind=Value | Kind=Value) and an extensibility hook so each framework can register its own filter kinds and the DefaultFilterMatchTextProperty used for bare matches. Would love your feedback on the surface area, the extensibility model, and anything that wouldn't fit your framework's semantics.

@MarcoRossignoli @Youssef1313 — would also appreciate your eyes on the design before we move toward implementation.

[thomhurst]

A big thing to think about for this I think is llms. For TUnit, they get confused and try to use old vstest expressions a lot. I think good error messages should help them with this though.

[thomhurst]

Also are we opening up custom built extensions at all? I'd be happy to keep the existing treenode filter command line since users already use it, but would rather do it cleanly with actual filter types in the code, than workarounds

[bradwilson]

What about the source based --filter support package that you've already provided to us? Doesn't that conflict with this proposal? Does this mean we shouldn't ship with it?

[bradwilson]

A big thing to think about for this I think is llms. For TUnit, they get confused and try to use old vstest expressions a lot. I think good error messages should help them with this though.

I think you're just going to anger users if your --filter isn't compatible with the one that VSTest ships with, whether that knowledge and muscle memory comes from the developer themselves or via LLM.

[Evangelink]

The goal is to merge the various options all together with a small caveat for when no prefix is specified. Let's take as example --filter SomeTest which in VSTest case means TestCase.FQN contains "SomeTest", but we don't have the same FQN in MTP. We could "construct" it based of TestMethodIdentifierProperty (namespace.class.method) but I know this FQN implementation is already framework specific and doesn't always match this pattern.

[bradwilson]

Should anything be changed for xUnit.net v3 4.0 in the face of this proposal? Should I remove the --filter option?

I'm concerned not only with whether this new version of --filter actually serves the user's needs of a purely VSTest-compatible filter, but also whether some new version of .NET SDK is going to be broken with xUnit.net because we also bring a --filter option.

[bradwilson]

Reading through the whole RFC, it appears that --filter will be live whether the test framework opts in or not (because of Built-in Kinds).

If this is true, I need to remove --filter immediately from 4.0.

[thomhurst]

The goal is to merge the various options all together with a small caveat for when no prefix is specified.

I think before trying to smush everything together, we should support extensible filters and allowing extension authors to define their own.

I thought MTP was meant to be quite agnostic and I opinionated, but in terms of filters, it offers no extensibility and is completely opinionated currently.

Being opinionated isn't necessarily bad imo because it keeps test frameworks having similar expected filters. But I think not allowing us to offer more custom options is a real extensibility blocker

[bradwilson]

Should --treenode-filter keep working forever rather than being deprecated? Cost is small; "purity" is the only argument for removal.

I really dislike that MTP doesn't have a way to have a "hidden" option. We use this in xUnit.net frequently when transitioning from one syntax to another to preserve backward compatibility without listing the explicit back-compat option.

Example: when reporters became user configurable, and we switched all reporters from their short form (e.g., -json) to the longer form (e.g., -reporter json). We still support all the original built-in short form names for backward compatibility but they don't show up in the help. https://github.com/xunit/xunit/blob/1250120334242dfd4bca417c6317fb24147c6cf9/src/xunit.v3.runner.common/Parsers/CommandLineParserBase.cs#L235-L240

[bradwilson]

Being opinionated isn't necessarily bad imo because it keeps test frameworks having similar expected filters. But I think not allowing us to offer more custom options is a real extensibility blocker

This is explicitly called out in the Open Questions section (and deferred to be addressed later):

8. Per-project inconsistency in mixed-framework solutions. Different frameworks register different kind sets; running one filter expression across a solution may match in one project and error in another ("unknown kind"). Proposed mitigation: at solution entry, downgrade "unknown kind" from an error to a warning only when the kind is known by at least one project in the solution. Costly to implement; defer to a follow-up RFC.

What that potential implementation probably means is that things won't necessarily work when they're expected to, and the user is expected to see and interpret the warning to understand why. To be fair, there's no good answer here: Do you "hope for the best" and just run things, hoping the user knows that some of the filter is inapplicable to some of the test projects? Or do you refuse to run, making is hard or impossible for users to craft together a single query for solutions which target multiple test frameworks? Really only Microsoft knows (or could know) how often people are in such situations; I don't really field many questions at all about mixing xUnit.net with other test frameworks.

[Evangelink]

Should anything be changed for xUnit.net v3 4.0 in the face of this proposal? Should I remove the --filter option?

I haven't look at xUnit v4, I suppose you register --filter yourself (not through the bridge). Thinking about it, it may already be a problem for the bridge because if core platform "claims" --filter and vstest bridge does too it would cause issue. I can obviously update the bridge but if someone uses older version of bridge with newer core platform this would break.

I'll sleep on it and think about what I could do.

I really dislike that MTP doesn't have a way to have a "hidden" option. We use this in xUnit.net frequently when transitioning from one syntax to another to preserve backward compatibility without listing the explicit back-compat option.

Nothing is set in stone yet but I have tried to work on #8501 to help with that. I also did #7113 which we could probably combine. I am not against us all discussing and designing a different solution.

but in terms of filters, it offers no extensibility and is completely opinionated currently.

In this RFC, I am discussing the way to add prefix and the related handler logic which should in-part cover this need. I'll get back to your various open tickets around filter extensibility to see how to open it more.

To be fair, there's no good answer here

Mixed framework is more of a niche than mass thing but it's something that exist (especially for more mature products). I know this is a problem we have with the fact we don't have enough common filters. When we started design of MTP, I was hoping to the new filter to be "enforced" on all frameworks so we could have a filter covering most of the scenario for all frameworks but sadly because of the various internal pushbacks we are in some situation.

Maybe we should have claimed some more general options to cause less issues. Again I was naively hoping we would be allowed more time and we could all work together on the growing specs rather than us all moving on our side (not a blame, more of a reality check).

Overall maybe this design is interesting but would need to be applied only to a next breaking change version of MTP and for now we only move with adding more --filter-xxx options. I'll re-read it all (including your comments) tomorrow with fresh view to see if there is anything to move forward.

[Youssef1313]

I'm extremely opposed to creating yet a new --filter. This is going to be really confusing.

If any change is to be taken here, I would simply just drop our treenode filter (in next major) which is only used by TUnit.

The platform cannot do much about filtering anyways. It's up to test frameworks to do the filtering because the framework is where the knowledge of running the tests is.

Simply put, I would just keep the VSTest-based filtering in the VSTestBridge. For frameworks that don't use the bridge, they can use the VSTest source only package to provide the --filter support.

I think the best path forward is to just keep everything similar to VSTest filtering. That's really the least problematic approach to the ecosystem.

[Youssef1313]

That's not very true.

--filter is supported by MSTest and NUnit.

The support was missing in xunit.v3 but we are filling the gap now in xunit.v3 4.x. We shouldn't degrade this after we worked towards resolving it.

[Youssef1313]

The goal is to merge the various options all together with a small caveat for when no prefix is specified. Let's take as example --filter SomeTest which in VSTest case means TestCase.FQN contains "SomeTest", but we don't have the same FQN in MTP. We could "construct" it based of TestMethodIdentifierProperty (namespace.class.method) but I know this FQN implementation is already framework specific and doesn't always match this pattern.

MTP doesn't need to have the knowledge of FQN. It's test framework responsibility to filter.