Extensible Filters + AggregateFilter

[thomhurst]

WIP.

Fixes #3530
Fixes #3590

Changes:

• Remove ITestExecutionFilterFactory - Filters can just be registered directly on the test application builder via a delegate. They can access the service provider if they like.
• Test filters expose an IsAvailable property (I dunno what to name this? IsAvailable? IsActive?) to tell the framework whether it has been requested as a filter for this test run
• Filters are then selected only if IsAvailable is true
• The filter implementation is built just before being passed to the framework with the following logic:
  • If there are test nodes passed to the execute request, then we build a test node uid filter
  • If there is a single IsAvailable filter, we pass that to the framework
  • If there is no available filter, we pass a NopFilter to the framework
  • If there are two or more available filters, we construct an AggregateFilter, and the filters are passed in as inner filters which can be accessed via a ITestExecutionFilter[] array inside AggregateFilter

Framework authors can now also create their own filters and register them.

• They create a filter that implements ITestExecutionFilter.
• They then call TestApplicationBuilder.TestHost.RegisterTestExecutionFilter(...)
• If IsAvailable is true for that filter, it'll be passed into the framework

[thomhurst]

@Evangelink Hope you don't mind me having a go at tackling a couple of issues that have been sitting around for months?

Microsoft.Testing.Extensions.Retry isn't open-source/in this repo? Got failing tests because I've changed code, but that's in a package it seems so I can't make changes to it?

[thomhurst]

Same happening with integration tests

Unhandled exception. System.TypeLoadException: Could not load type 'Microsoft.Testing.Platform.Requests.ITestExecutionFilterFactory' from assembly 'Microsoft.Testing.Platform, Version=42.42.42.42,

Why does it reference packages instead of the local project's code?

[thomhurst]

@Evangelink @nohwnd @MarcoRossignoli Providing you were okay with implementing this, how would I go about being unblocked due to private/internal code that I can't edit?

[Evangelink]

@thomhurst I wasn't able to play a little with the PR but although I like the concept of the aggregate filter, I don't think that the extensibility as you are doing it is the right way forward as it would allow users to register any kind of filter including filters that are not handled by the framework. As of now, the filters have to be specifically supported by the platform or the framework.

[thomhurst]

Hmm that's a good point. The registration code is going to have to be surfaced publicly though in order for frameworks to register it.

Might be that filters work in conjunction with capabilities? So filters are only considered active/available if the framework has a capability for it? Kinda like the trx extension. That way we know a framework can handle that filter.

[thomhurst]

Another alternative is the filter knows how to perform the logic itself, so that users could actually add filters on the fly without specific support from the framework. This actually makes it freely extensible by the end-user rather than the framework author.

E.g.

public interface ITestExecutionFilter
{
    ...
    bool CanRun(TestNode testNode)
}

public class MyFilter : ITestExecutionFilter
{
    ...
    public bool CanRun(TestNode testNode) => testNode.Id == "1"; 
}

public class TreeNodeFilter : ITestExecutionFilter
{
    ...
    public bool CanRun(TestNode testNode) => MatchesFilter($"{testNode.Assembly.Name}/{testNode.Namespace}/{testNode.Class}/{testNode.Name}", testNode.Properties); 
}

[thomhurst]

@Evangelink did you have any opinions? I think the filter defining its own logic means users could make their own filters. Way more extensible then

[thomhurst]

@Evangelink @nohwnd @MarcoRossignoli @Youssef1313

Here's my latest idea on filtering. Each implementation implements its own MatchesFilter(TestNode) method which defines its own logic on how it should filter.

This means all frameworks need to do is call: ITestExecutionFilter.MatchesFilter(testNode) - They don't need to do any casting, or type checking, or implementing any logic, because it's defined by the filter itself.

Of course, they still could override the behaviour by doing what they do currently - Casting to the concrete type and implementing the filtering themselves, but I don't envision a lot of scenarios where you'd want to change the logic.

This also makes filtering more consistent between frameworks as they can all use centralised logic instead of recreating it themselves, with the risk of introducing bugs.

This also means users could create their own filters and register them, and they'd automatically flow through as the ITestExecutionFilter, or inside the AggregateFilter, and the filtering will work with their newly introduced logic without the framework even having to know about the specific type/logic.

Keen to know your thoughts. I think this works nicely with the Open/Closed principle - Closed for modification but highly open to extensibility 😄