Resharper 10 test runner annoyances Follow
ReSharper 10. VS 12.0.40629.0. Windows 8.1. Intel i7 64-bit
Resharper's test runner window has some really annoying issues in the latest release:
(These occur with the Output/Coverage window arranged below the tree view. I haven't tried with the Output/Coverage window to the right of the tree view.)
1. Sometimes a test run ends with the progress bar not fully across the screen. Is this by design? In the previous version the bar would always end up fully on the right hand side of the window when the tests had completed. This allowed a user to use the progress bar to, er, track progress. Now you cannot do this. Instead you have to look for movement on the icons in the tree control.
2. With the runner window undocked, re-sizing the window via drag and drop on the RH border causes re-drawing issues with the result column in the tree control. This happens to a lesser extent when the window is docked.
3. From time to time (when running large suites of tests) the horizontal scroll bar at the bottom of the tree control (if there is one; depends on lengths of test names, lengths of result texts etc) goes mad, flashing wildly.
4. Often the results in the tree control do not match the results according to the progress bar and the other icons across the top of the tree control. See the attached screen-dump. The progress bar shows four failed tests, but the whole of the tree-control is green.
5. The contents of the Output pane are often inconsistent with the selection within the tree.
In my opinion you (JetBrains) have taken your eyes off the quality ball with respect to the test runner since release 8. Version 9 had a whole raft of different issues with the test runner concerning usability. All of the new features, standardised platforms, single installers, unified licensing, continuous testing, faster building etc elsewhere in the product are very nice, but the little day to day glitches in the parts of the product that I use many times each day mount up to produce a very frustrating user experience.
David Razzetti
Attachment(s):
ResharperScreenDump.png
Please sign in to leave a comment.
Today I updated to version 10.1.
Within a few minutes, the test runner was showing inconsistent results. See the attached screen-shot.
On the left, the green tick mark is indicative of a single test. If there were multiple tests the line would be expandable with lines for the individual tests beneath. This (correct behaviour) can be seen lower down. On the right, the test name is incorrectly decorated with the text "(2 tests)" and the result text is bizarre - "Success: 2 tests failed".
Attachment(s):
R#10.1TestRunner.png
It looks like you're using SpecFlow. What's the test runner framework you're using? Is it xunit?
Hi Matt,
Yes it is SpecFlow.
SpecFlow produces tests that use NUnit (test attributes start with NUnit.Framework...)
The SpecFlow plugin is set to use the VisualStudio2012 test runner (Test Runner Tool property in the SpecFlow plugin's properties). I have been running using the ReSharper test runner setting as well. Both runner settings exhibit the issues within the R# test session window. I get slightly better test detection when using the VisualStudio2012 runner setting. (Note: both settings appear to cause the R# test runner to actually run the tests.)
(As far as test detection is concerned, I can make a change to a feature file (adding a new feature), add steps for the feature, save my changes, compile the solution, run the tests and the test runner fails to spot the test associated with the new feature. Very annoying. The only way I have found to get the test runner to spot the test associated with a newly added feature is to open the associated feature.cs file. As soon as the feature.cs file is opened, the test runner spots the new test and updates its UI.)
Regards
David Razzetti
I suspect the lack of child items is due to this issue (RSRP-451046), which means that dynamically created test rows (i.e. created at runtime, rather than in source) are broken. Please feel free to vote, track and add any details.
As for the SpecFlow tests not updating until you reopen the generated .cs file, I've added another issue for that: RSRP-451047
Here's another case where the test runner doesn't work properly.
The left hand gutter of the editor window spots that both methods are tests.
When the tests are executed the left gutter marks them all as having passed.
On the other hand, the test runner tree gets things wrong. It shows only six of the tests (the ones generated by the test case attribute). It marks them as having succeeded, but it marks the parent node as Aborted!
Workaround - don't use overloaded method names within tests!
[Test]
[TestCase("BSC0928_[PCHR]00Log20110928145923_20110928150245.log.zip", 2011, 9, 28, 14, 59, 23, false)]
[TestCase("BSC0928_[PCHR]00Log20110928145923(DST)_20110928150245(DST).log.zip", 2011, 9, 28, 14, 59, 23, true)]
[TestCase("RNC0927_(PCHR)02log20140313143407_20140313143439.log.zip.dat.gz", 2014, 3, 13, 14, 34, 7, false)]
[TestCase("RNC0927_(PCHR)02log20140313143407(DST)_20140313143439(DST)_01.log.zip.dat.gz", 2014, 3, 13, 14, 34, 7, true)]
[TestCase("NE256_02_20140511173030_20140511173240_1234.log.gz", 2014, 5, 11, 17, 30, 30, false)]
[TestCase("NE256_20140511173030_20140511173240_1234.mr.gz", 2014, 5, 11, 17, 30, 30, false)]
public void TestGetStartTimeFromFilename(string name, int years, int months, int days, int hours, int minutes, int seconds, bool isDst)
{
var result = FileHandling.GetStartTimeFromFilename(name);
// Do stuff to test result
}
[Test]
public void TestGetStartTimeFromFilename()
{
Assert.Throws<ArgumentException>(() => FileHandling.GetStartTimeFromFilename("I am a bad filename"));
}
Attachment(s):
R#TestRunnerWrong.png
FWIW, I've noticed that my test runner is running tests outside of the ones I've requested. Even if I clear out a session I still have the problem. You might be finding this causing some ofyour problems yourself..
Another issue that I have noticed with unit testing - the explorer does not show all of the discovered unit tests. You have to refresh the view. Then when you run the tests, some of the results are dropped from the session list. VS2015 with MSTest.
I'm seeing this too, and it's definitely a dealbreaker for me. I don't want to have to unfold every single tree level to know if the tests succeeded or failed.
Is this reported somewhere on the bug tracker?
Yes I've been observing the same thing - running extra tests. Really annoying when trying to debug just one test.
It is a known issue for NUnit 3.0 tests https://youtrack.jetbrains.com/issue/RSRP-450313. Please try installing R# 10.0.2 EAP https://confluence.jetbrains.com/display/ReSharper/ReSharper+10.0.2+EAP - it's been fixed there.