Test Runner serial execution of tests

We have some unit tests that make calls to OTS libraries that we are using that we suspect are not thread safe. When we execute the unit tests through R# we experience some tests hang and it looks like a threading issue. If we execute the tests through a different test runner like Icarus then all tests pass.

I suspect R# executes tests concurrently... is it true? Can I configure R# to execute tests serially?

We will independantly work on the thread safety of the interface but I am just trying to diagnose the root cause right now. Thanks.

6 comments
Comment actions Permalink

Assuming you're using nunit or mstest, then tests are run serially. However, you can tell ReSharper to run multiple assemblies in parallel. As I understand it, each assembly is then run in a separate process, so you shouldn't see threading issues in that respect.

0
Comment actions Permalink

Thanks for your reply. Can you point to any reference for your advice or help how to run R# tests from multiple assemblies in parallel?
[Edit: sorry, i Googled this and found how to set multiple assemblies in parallel, however, i would like to hear from Jetbrains if possible that tests from a single assembly are not run concurrently].

[Edit: also found this discussion on Stack Overflow http://stackoverflow.com/questions/6427138/how-to-run-nunit-test-fixtures-serially]

I should have included some more specific information.
We are using Nunit included with R# 6.1
Visual Studio 2010 64 bit
Windows 7 64 bit
The OTS library in question is Intel's MKL

Another interesting note. If I execute testA alone in R#, it will execute and pass. If I execute all tests in my solution (multiple assemblies) using R#, testA will hang after an access violation exception. If I execute all tests in the solution from Gallio Icarus, testA will execute and pass.

0
Comment actions Permalink

So, as you found, you can enable the number of assemblies to run in parallel through the options dialog. Tests in a single assembly are run serially. ReSharper actually uses the underlying test frameworks (nunit, mstest) to run the tests, and attach as listeners to display the results. So, if the test framework runs the tests serially, ReSharper does too.

I don't know what's going on in the StackOverflow question, that's not what's supposed to happen, and if you read the comments, there are several where people can't reproduce this behaviour.

Thanks
Matt (at JetBrains)

0
Comment actions Permalink

Thanks Matt for the extra information. Unfortunately I can reproduce this behaviour only through R#. I just executed the tests in the NUnit GUI and they all execute serially. For now it's easier if we stop using R# :-(

0
Comment actions Permalink

Can you reproduce this in a small solution you can send me? Or can you make each test write the current thread id to a file, where the write method is wrapped in a lock?

Thanks
Matt

0
Comment actions Permalink

I tried but I could not reproduce this using a small managed example. If I get the time I will try to carve a small example out of the project that calls the native library.

0

Please sign in to leave a comment.