MSTestRunner does not integrate with "VS MSBuild"

[RuiAlias]

Describe the bug

An MSTest.Sdk project cannot be tested with "VS MSBuild" (or "Full MSBuild").

Steps To Reproduce

On a Windows machine with "VS MSBuild" and dotnet SDK installed:

dotnet new mstest
MSBuild.exe -restore -t:Build;Test mstest.csproj

You can also see the repro: https://github.com/RuiAlias/mtp-msbuild-repro. On this repro I created a GitHub Actions Workflow that runs the msbuild command and fails: https://github.com/RuiAlias/mtp-msbuild-repro/actions/runs/11858511987/job/33049304509.

Expected behavior

MSBuild.exe runs the tests successfully.

Actual behavior

MSBuild.exe fails with error:

D:\a\mtp-msbuild-repro\mtp-msbuild-repro\bin\Debug\net8.0\mtp-msbuild-repro.dll : error run failed: Tests failed: Possible reasons for this include: [D:\a\mtp-msbuild-repro\mtp-msbuild-repro\mtp-msbuild-repro.csproj]
D:\a\mtp-msbuild-repro\mtp-msbuild-repro\bin\Debug\net8.0\mtp-msbuild-repro.dll : error run failed: Could not execute because the specified command or file was not found. [D:\a\mtp-msbuild-repro\mtp-msbuild-repro\mtp-msbuild-repro.csproj]
D:\a\mtp-msbuild-repro\mtp-msbuild-repro\bin\Debug\net8.0\mtp-msbuild-repro.dll : error run failed:   * You misspelled a built-in dotnet command. [D:\a\mtp-msbuild-repro\mtp-msbuild-repro\mtp-msbuild-repro.csproj]
D:\a\mtp-msbuild-repro\mtp-msbuild-repro\bin\Debug\net8.0\mtp-msbuild-repro.dll : error run failed:   * You intended to execute a .NET program, but dotnet---internal-msbuild-node does not exist. [D:\a\mtp-msbuild-repro\mtp-msbuild-repro\mtp-msbuild-repro.csproj]
D:\a\mtp-msbuild-repro\mtp-msbuild-repro\bin\Debug\net8.0\mtp-msbuild-repro.dll : error run failed:   * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH. [D:\a\mtp-msbuild-repro\mtp-msbuild-repro\mtp-msbuild-repro.csproj]
D:\a\mtp-msbuild-repro\mtp-msbuild-repro\bin\Debug\net8.0\mtp-msbuild-repro.dll : error run failed:  [D:\a\mtp-msbuild-repro\mtp-msbuild-repro\mtp-msbuild-repro.csproj]

Additional context

I think I've narrowed the problem down to the name of the executable "MSBuild.exe".

testfx/src/Platform/Microsoft.Testing.Platform.MSBuild/Tasks/InvokeTestingPlatformTask.cs

Line 191 in 7eade67

if (dotnetRunnerName != MonoRunnerName && Path.GetFileName(_currentProcess.MainModule!.FileName!).Equals(dotnetRunnerName, StringComparison.OrdinalIgnoreCase))

On the repro there is a GitHub Action Workflow that proves this by: renaming the executable from "MSBuild.exe" to "dotnet.exe" and then running dotnet.exe -restore -t:Build;Test mstest.csproj, the tests run successfully.

• dotnet test mstest.csproj works
• dotnet msbuild -restore -t:Build;Test mstest.csproj works
• MSBuild.exe -restore -t:Build;Test mstest.csproj fails
• mv MSBuild.exe dotnet.exe; dotnet.exe -restore -t:Build;Test mstest.csproj works

https://github.com/RuiAlias/mtp-msbuild-repro/blob/main/.github/workflows/build-and-test.yml

[Evangelink]

Hi @RuiAlias,

I have done some manual checks and the Test target is also not working from a non-SDK project both for VSTest and the new runner:

VSTest:

...
Test : The term 'Test' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:28
+ msbuild  -restore -t:Build;Test .\ProjectWithVSTest\ProjectWithVSTest ...
+                            ~~~~
    + CategoryInfo          : ObjectNotFound: (Test:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

MSTest runner:

...
Test : The term 'Test' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:28
+ msbuild  -restore -t:Build;Test .\ProjectWithMSTestRunner\ProjectWithMSTestRunner ...
+                            ~~~~
    + CategoryInfo          : ObjectNotFound: (Test:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

I'll mark this as feature request and discuss with the team if we want to support it or not.

[RuiAlias]

Hi @Evangelink. That error is due to PowerShell syntax. AFAIK PowerShell interprets ; as command separator, so it's interpreting two commands from that line: msbuild -restore -t:Build and Test .\ProjectWithMSTestRunner\ProjectWithMSTestRunner. I invoke it as msbuild -restore '-t:Build;Test' mstest.csproj (quotes around the targets argument), when using PowerShell.

The "Test" targets exists and works. But it only works if the name of the invoked process is "dotnet.exe". Which is a problem for me because I need to use the "Full MSBuild", that has an executable named "msbuild.exe".

From what I've identified the issue has nothing to do with "MSTest.SDK" and is the responsibility of the "Microsoft.Testing.Platform.MSBuild".

[Evangelink]

Thanks! Indeed some mistake from me.

I have looked at the code and the Test target is part of dotnet and not MSBuild so I don't think there is anything we can do to achieve that but I'll try one more test to see if we can workaround that.

[RuiAlias]

Thanks! Indeed some mistake from me.

No problem. And thanks for looking into this issue.

I have looked at the code and the Test target is part of dotnet and not MSBuild so I don't think there is anything we can do to achieve that but I'll try one more test to see if we can workaround that.

MSBuild has a "Test" target since v17.9.5 according to this commit dotnet/msbuild@b9d3539. It's how "Microsoft.Testing.Platform.MSBuild", "Microsoft.Build.RunVSTest" and other test runners are able to integrate with MSBuild.

testfx/src/Platform/Microsoft.Testing.Platform.MSBuild/buildMultiTargeting/Microsoft.Testing.Platform.MSBuild.targets

Lines 262 to 268 in 077bf0e

	<!-- Plug with the new msbuild target
	https://github.com/dotnet/msbuild/blob/main/src/Tasks/Microsoft.Common.CurrentVersion.targets#L6807
	https://github.com/dotnet/msbuild/blob/main/src/Tasks/Microsoft.Common.Test.targets
	-->
	<Target Name="_MSBuildTest" AfterTargets="Test" Condition=" '$(UseMSBuildTestInfrastructure)' == 'True' AND '$(_MSBuildTestInfrastructureAvailable)' == 'True' " >
	<CallTarget Targets="_TestingPlatformTest" />
	</Target>

I believe the issue is just this line in "Microsoft.Testing.Platform.MSBuild", that depends on the name of the executable _currentProcess.MainModule!.FileName!:

testfx/src/Platform/Microsoft.Testing.Platform.MSBuild/Tasks/InvokeTestingPlatformTask.cs

Line 191 in 7eade67

if (dotnetRunnerName != MonoRunnerName && Path.GetFileName(_currentProcess.MainModule!.FileName!).Equals(dotnetRunnerName, StringComparison.OrdinalIgnoreCase))

Which basically evaluates to if ("dotnet.exe" != "mono.exe" && "msbuild.exe" == "dotnet.exe"), when running with "Full MSBuild". Now, I'm not sure why this comparison was implemented like this. I assume it was to have a different behavior when running under Mono; and MSBuild was forgotten or not taken into consideration. But it might have been for another reason.

What I do know is that renaming the executable from "msbuild.exe" to "dotnet.exe" is enough to make it work which means that there isn't a deeper issue going on, like that there are missing features in MSBuild, etc..

[Evangelink]

Ooooh I was convinced this code was on dotnet not MSBuild! Changing this ticket as bug then!

Thanks for the detailed answers and investigations!