'executor://mstestadapter/v2': The method 'RunCleanup' was not found on the interface/type

[sfranzen]

Describe the bug

After upgrading MSTest.TestAdapter and MSTest.TestFramework from 2.1.2 to 3.1.1 in one of our test projects, I ran into an issue similar to #799. An exception occurs on a call to "RunCleanup", see the stacktrace pasted under Additional Context. When run locally through Visual Studio, the exception is logged, but it does not fail the test run. The DevOps pipeline, using the Visual Studio Test task of version 2.220.0, fails due to the nonzero exit code. The project is targeting .NET v4.7.2.

Steps To Reproduce

1. Upgrade MSTest.TestAdapter and MSTest.TestFramework from 2.1.2 to 3.1.1;
2. Clean and recompile to ensure the new packages are used;
3. Run tests that require a "RunCleanup".

Expected behavior

Ideally, no exception should occur, but if it does, the pipeline task should not respond differently than the local test runner.

Actual behavior

The DevOps task fails:

2023-11-17T09:26:42.1551706Z Test Run Failed.
2023-11-17T09:26:42.1552022Z Total tests: 135
2023-11-17T09:26:42.1553156Z      Passed: 135
2023-11-17T09:26:42.1564233Z  Total time: 5,8879 Seconds
2023-11-17T09:26:42.1829408Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2023-11-17T09:26:42.2076630Z ##[error]Error: The process 'G:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe' failed with exit code 1

Additional context

The exact test adapter version and exception stacktrace are as follows:

2023-11-17T09:26:40.0439341Z Multiple versions of same extension found. Selecting the highest version.
2023-11-17T09:26:40.0439912Z   Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter : 14.0.4701.2
2023-11-17T09:26:40.0444875Z An exception occurred while invoking executor 'executor://mstestadapter/v2': The method 'RunCleanup' was not found on the interface/type 'Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner, Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
2023-11-17T09:26:40.0445772Z Stack trace:
2023-11-17T09:26:40.0450672Z 
2023-11-17T09:26:40.0516598Z Server stack trace: 
2023-11-17T09:26:40.0522551Z    at System.Runtime.Remoting.Messaging.MethodCall.ResolveMethod(Boolean bThrowIfNotResolved)
2023-11-17T09:26:40.0526724Z    at System.Runtime.Remoting.Messaging.MethodCall..ctor(SmuggledMethodCallMessage smuggledMsg, ArrayList deserializedArgs)
2023-11-17T09:26:40.0527501Z    at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoDispatch(Byte[] reqStmBuff, SmuggledMethodCallMessage smuggledMcm, SmuggledMethodReturnMessage& smuggledMrm)
2023-11-17T09:26:40.0528099Z    at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatchCallback(Object[] args)
2023-11-17T09:26:40.0528536Z 
2023-11-17T09:26:40.0530263Z Exception rethrown at [0]: 
2023-11-17T09:26:40.0530866Z    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
2023-11-17T09:26:40.0531410Z    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
2023-11-17T09:26:40.0531977Z    at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner.RunCleanup()
2023-11-17T09:26:40.0532562Z    at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunCleanup(ITestExecutionRecorder testExecutionRecorder, UnitTestRunner testRunner)
2023-11-17T09:26:40.0533249Z    at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.ExecuteTestsInSource(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, String source, Boolean isDeploymentDone)
2023-11-17T09:26:40.0533981Z    at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.ExecuteTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, Boolean isDeploymentDone)
2023-11-17T09:26:40.0540439Z    at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle, TestRunCancellationToken cancellationToken)
2023-11-17T09:26:40.0546218Z    at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle)
2023-11-17T09:26:40.0552145Z    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
2023-11-17T09:26:40.0558330Z    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.<>c__DisplayClass46_0.<RunTestInternalWithExecutors>b__0()
2023-11-17T09:26:40.0563240Z    at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.<>c__DisplayClass0_0.<Run>b__0()
2023-11-17T09:26:40.0564248Z --- End of stack trace from previous location where exception was thrown ---
2023-11-17T09:26:40.0564802Z    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2023-11-17T09:26:40.0565364Z    at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.Run(Action action, PlatformApartmentState apartmentState, Boolean waitForCompletion)
2023-11-17T09:26:40.0569430Z    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.TryToRunInStaThread(Action action, Boolean waitForCompletion)
2023-11-17T09:26:40.0574294Z    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

[Evangelink]

Hi @sfranzen,

Thanks for the feedback! Sadly this is a known hard limitation of VSTest that doesn't allow us to mix multiple versions of the same dll in a solution (it's selecting one version and use it for all). To fix that, you would have to bump all your test projects to v3.

I know this is far from ideal scenario but we are quite tighted on that side. I will move forward by closing the issue as "by design".

[sfranzen]

@Evangelink Understood, thanks for the response!