Execution Order Issue with ReSharper Test Runner Impacting Tests Individually
I am reaching out to seek assistance regarding an execution order issue we have encountered when using the ReSharper Test Runner. In our C# project, we have implemented a comprehensive test suite that incorporates both unit and integration tests. To ensure proper data setup and deletion, we utilize the [ClassInitialize] attribute in some test classes to perform these tasks before any tests within the class are run. Additionally, for most test classes, we rely on the [TestInitialize] attribute to initialize the test-specific data. However, when running tests through the ReSharper Test Runner, the execution order is not respected. This leads to a critical problem where a test using the [TestInitialize] attribute can inadvertently delete the required data set up by the [ClassInitialize] method. Consequently, the test classes relying on [ClassInitialize] start to fail when subsequent tests run. This issue remains unnoticed when using the Visual Studio Test runner which respects the expected execution order. How can we resolve this execution order problem without having to rewrite many tests for the developers using R#? Is there any configuration or setting within ReSharper that can enforce the correct execution order of tests. Since this is a legacy project we are uncertain about the specific trigger that led to this problem. I have tried it with several older versions of R# but the problem remains.
Please sign in to leave a comment.
Hello Gabriel,
When using
AssemblyInitialize,ClassInitializeorTestInitializeattributes, MSTest has an established order in which it runs the methods marked with such attributes, it is listed in the docs:In turn, ReSharper runs tests in a class in the order these tests are written in the file, from top to bottom. Hopefully, this information will help point you in the right direction.
If you have any further questions, please feel free to ask. We would really appreciate an example of code that produces this behavior as it will enable us to investigate further. Thank you in advance. Have a great day!
Hello Ivan,
Thank you for your reaction. I have a small video clip (mp4) where you can see the ReSharper Test Runner jump from test to test in different classes. Unfortunately due to the sensitive nature of this project (federal government), I cannot publicly post the video clip. Is there another way for me to share the clip with you?
Thank you,
Hello,
you can share privately it via another ticket you have submitted earlier (#5223705) or you can use JetBrains Upload Service - upload the file and post the upload ID here.
I uploaded the video clip using the JetBrains Upload Service:
Upload id: 2023_06_13_DJ9gypvDQDgmXGFLh8UimS (file: 2023-06-13_12-04-06.mp4)
I indeed submitted a similar ticket a couple of days ago but I could no longer find it and therefore I was under the impression that it never got posted or that it got deleted.
Hello,
Thank you for the screencast, it sheds a little light on the behavior you're observing but still - we would really appreciate a sample solution or an example of a test class where the execution order is perceived wrong. Also, do all those tests that are executed in the order show in the video belong to the same test project?