The other week I got to spend a few hours trying to figure out, why some of our tests would silently crash the VSTest test host with no error message, log statement, and no Windows event log entry. All I got was:
The active test run was aborted. Reason: Test host process crashed
This then lead me down a rabbit hole of acquiring some knowledge on how to get VSTest to output more information than it usually does.
VSTest
The original command for the test run looked something like this:
vstest.console.exe SomeTest.dll /Settings:Some.runsettings /Logger:trx /Platform:x64
It will execute the tests in the SomeTest.dll with the Some.runsettings specified, log the output in the TRX format (assuming short for “Test Results XML”) and execute the whole test under the 64bits architecture.
To enable diagnostic logging you can add --diag:logs/log.txt, which will write multiple diagnostic logs under the specified logs/ directory. One file for the test host, one for the data collection, and one for the test runner. Unfortunately in my case all I saw in those logs was a forcibly closed connection and the test host simply stopping to log.
TestRequestSender: GetAbortErrorMessage: Exception: System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
To narrow down which test could be causing the issue you can use /TestCaseFilter which has many possibilities to filter by category, name, attributes, etc. for example like this: /TestCaseFilter:TestCategory!=GoodTests
This helped me narrow it down to one DLL in particular and at least one test that was always causing a failure.
Additionally, to find out what test potentially caused the issue, you can use /Blame which generates a sequence file listing all the tests in the order they have been or would have been executed.
If this all didn’t help, you can generate mini dumps of the crash to further analyze in Visual Studio, WinDbg, or similar. For me this somehow only worked with the Visual Studio 2022 runner and not with the one from Visual Studio 2019. You can enable it with: /Blame:CollectDump
Putting it all together the final command looked something like this:
vstest.console.exe SomeTest.dll /Settings:Some.runsettings /Logger:trx /Platform:x64 /TestCaseFilter:TestCategory!=GoodTests /Blame:CollectDump --diag:logs/log.txt
Mini Dump
I’m not really familiar working with dumps, so I can’t give you more advice than to point to the two tools that can help you:
- Visual Studio can open dumps and if you have the symbols and/or source code, it’s able to connect the information in the dump to the source code itself.
- WinDbg: Website / MSStore
The Solution?
After quite some time trying to get more information out of VSTest, I eventually took a closer look at the DLL that caused all the trouble. In our case it was an issue with a special package that is only used in a few tests. Unfortunately, it always lead to an immediate shutdown of the test host without giving it time to write any logs or indicate to the problem.
The mini dump and logs only partially helped this time, but maybe it can useful for you or my future self. 😊
References
- VSTest Console Options: https://learn.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2022
- TestCase Filter: https://github.com/Microsoft/vstest-docs/blob/main/docs/filter.md
- Issue Discussion: https://github.com/microsoft/vstest/issues/2952