Unit test runner runs wrong tests

The unit test runner has always worked well for me, but latest version seems to be extremely buggy. All these issues are for an nUnit test project we have. We are running version 2017.2, maybe these issues are known and being worked on? I can't imagine we are the only ones experiencing this and there is nothing out of the ordinary about our test setup for nUnit.

 

Incorrect nesting of tests in the Unit Test Explorer where there is an Nunit SetupFixture, some TestFixtures seem to be nested under it, others not, there doesn't seem to be any reason why you would nest tests when a SetupFixture applies to all tests in a namespace anyway?

Unit Test Explorer not showing the correct list of tests, even after hitting Refresh, only option being to close the solutions and open it again

Unpredictable behaviour when trying to run tests at a Fixture level. This has worked fine for me in the past but is causing big problems at the moment. Unless I specifically select the unit tests I want to run in the unit test explorer I can't trust what will happen when I try to run all tests in a certain TestFixture, mostly all the Tests in the full project seem to start to run in some kind of strange mode where every test is it's own session with OneTimeSetUp being called once per test.

17 comments
Comment actions Permalink
Official comment

There was an issue in R# 2017.1 that affected execution of tests in a session when hitting the 'Run current session' button. But it is fixed and works as expected in 2017.1.2. See https://youtrack.jetbrains.com/issue/RSRP-464189. (There is no version 2017.2, by the way. Сould you please specify which version you meant?)

The incorrect nesting of the tests can indeed happen when there is a SetUpFixture. It is a long-standing issue which is unfortunately being pushed out out of the scope by more pressing matters. We will try to include it into the next release. There is a workaround: regrouping tests in session/explorer after running them will eliminate duplicated nodes. Also, incorrect grouping does not cause problems with execution.

Other issues you mentioned are unknown to us. It is especially strange since our last release was concentrated around making .netcorepp tests work. Out of the top of my head I don't remember any changes made to vanilla UT frameworks. (Are we right to assume that test projects in question are normal vanilla class libraries with some version of NUnit referenced?) So it would be very help if you could compile a sample for us.

 

Comment actions Permalink
  • Unpredictable behaviour when trying to run tests at a Fixture level. This has worked fine for me in the past but is causing big problems at the moment. Unless I specifically select the unit tests I want to run in the unit test explorer I can't trust what will happen when I try to run all tests in a certain TestFixture, mostly all the Tests in the full project seem to start to run

I confirm this behaviour in resharper version 2017.1.3. In the UI it is not visible this is happening. But as I am using integration tests, I see applications pop up from tests I did not select. If this happened when running normal unit tests, I would probably not have noticed the full project was being tested.

As version 2017.2 is available now, I will try that.

 

2
Comment actions Permalink

This is still happening in Resharper 2017.3.2. I have a TestFixtureSource attribute, and when I try to run one test all of them are executed. Besides that the nesting in both Unit Test Explorer and Unit Test Sessions is wrong. I have used this runner before and this didn't happened. I'm being forced now to use VS runner.

2
Comment actions Permalink

I can confirm that "Unpredictable behaviour when trying to run tests at a Fixture level" is still happening and I'm using Resharper 2017.2.

For some reason when I try to run the tests by right clicking on the SetUpFixture, Teardown method inside is triggered first, instead of tests.

1
Comment actions Permalink

Hello,

I am having an issue similar to the ones described above.

I am using NUnit3 with ReSharper 2018.2 and I have a SetUpFixture class in the namespace.

When I try to run a test case individually, all the other test cases from the same namespace are executed as well, even when they are not included in the test session.

I have lots of warnings like this :
"2018.08.23 13:39:03.982    WARN Run: 0587183c-ad47-46f2-8454-21f6afbb3caa - Element <AnyOtherTestCaseInsideTheNamespace> doesn't exist in session"

Eventual test errors from the other test cases are also thrown, which was supposed to happen only if I actually ran them.

Do you happen to know if there is a fix for that?

Thanks!

1
Comment actions Permalink

Hello Rvictor!

 

Could you please try the following build - https://drive.google.com/open?id=14gXwbD9gOGEZYnRtpcrEzL3eDiMU06Bt and check if it helps?

Thank you.

1
Comment actions Permalink

Thank you for letting us know.

The fix will be included in ReSharper 2018.2.1 version.

Thank you.

1
Comment actions Permalink

No answers? I can't be the only person that's seen this our test project is really basic NUnit. see the following examples with Customer information redacted. GlobalSetUp is a SetUpFixture that applies to all the unit tests. So why are two TestFixtures nested under it and one is not?

So I close Visual Studio and reopen it, no code changes but now a different nesting (still incorrect though)

Whatever is buggy about the Test Identification I imagine is also causing problems when coming to run tests. I should be able to right click on a Fixture, Project etc. and run all tests, but it's unpredictable what will happen in this case. I've resorted to hightlighting all tests individually if I want to run more than one at a time.

 

 

 

 

0
Comment actions Permalink

Yes sorry the version is 2017.1.2

Also it was probably a bit confusing saying the test runner worked fine before, that was for other projects in the past, this test project has been built from start in the last few months, so whatever strange behaviour i'm encountering may have been there from a number of versions back

If I have a chance I will try strip the test project down to just the basics and see if I can recreate the problem, they are integration tests that use Selenium so there's a lot of moving parts, the reason I really notice OneTimeSetup being called multiple times within a test run is because that's where we create the Selenium driver so a new browser instance gets opened for each test.

0
Comment actions Permalink

Still seeing issues with NUnit in 2018.1.2. Seems to be related to the TestFixtureSource issue. The issue i'm seeing though is that when i run a data driven test that uses TestCaseSource attribute it appears to execute unrelated tests in the background. If i try to debug a test exceptions for tests that i had not run are being caught. I can follow the call stack back to them. 

In addition to this i'm now constantly getting the message 

 

---------------------------
ReSharper Ultimate – Unit Test Runner
---------------------------
All unit tests finished, but test process still running. Abort?
---------------------------
Yes No
---------------------------

0
Comment actions Permalink

Hello Ryan!

 

Could you please  run Visual Studio with the following command line: 'devenv /ReSharper.LogFile C:\resharper_log.txt /ReSharper.LogLevel Verbose', reproduce the issue and send us a corresponding 'resharper_log.txt' file.

You can do it privately via 'Submit a request' form.

Thank you.

0
Comment actions Permalink

Hi Angelina!

This build seems to have it fixed, indeed!

Thank you

0
Comment actions Permalink

Hello,

I ran into the same issue than Rvictor and I also tried the build you gave him but for me I could not run any test after installing it. It fails with an "NUnit runner compilation failed".

I am just leaving this comment to let you know.

Regards,

0
Comment actions Permalink

Thanks for sharing a build containing the fix!

Is there any indication on when an official version with it is going to be released?

0
Comment actions Permalink

Hello Danny!

 

It's gonna be ReSharper 2018.2.1 version.

Thank you.

0
Comment actions Permalink

Hello Thomas!

 

Thank you for the feedback.

Could you please run Visual Studio with the following command line: 'devenv /ReSharper.LogFile C:\resharper_log.txt /ReSharper.LogLevel Verbose', reproduce the issue and send us a corresponding 'resharper_log.txt' file. You can do it privately via submit a request form.

Thank you.

0
Comment actions Permalink

Hello Angelina,

I created a specific issue with this bug here:

https://youtrack.jetbrains.com/issue/RSRP-471333

Regards,

0

Please sign in to leave a comment.