Code Coverage and Continuous Testing makes TDD impossible on larger projects
I'm working on a fairly large solution with 58 projects, and love the idea of continuous testing (This is a new feature for me, having just updated to the latest R# Ultimate), however it's actually slowing me down, and quite significantly, to the point where I'm looking at disabling the R# testing features and reverting back to the native VS testing tools.
Everytime I make a small code change, and build/save, the testing runs (it's configured with the setting "On Build, run dirty tests"), and the tests are performed in a heartbeat, however this immediately triggers the code coverage analysis, which takes around 5 minutes to run, and further runs of the unit tests are blocked until the code coverage analysis has completed.
When you're practising TDD, this makes continuous testing quite painful to work with.
So, is there any way to disable code coverage automatically running in a continuous testing session? Or can the code coverage analysis be streamlined so it only analyses the dirty modules, instead of the entire project (which I assume is what's happening at the moment, given the time it takes to run)?
Please sign in to leave a comment.
Hi Bryan,
Thank you for your feedback.
We have changed some time-consuming internal processes of code analysis. It should speed up continuous testing.
Could you please try the latest ReSharper 2017.2 EAP build? Here's the link: https://www.jetbrains.com/resharper/eap/
The first run after update could still be as long as you described, but next runs should take way less time.
Also, in this EAP build you could increase N for "Run up to N assemblies in parallel" option in ReSharper >> Options >> Tools >> Unit testing - it should also improve performance.
Hi Fedor,
Thanks for your feedback, I've updated to that build, and have performed some testing against the same solution with 58 projects, and have timed how long it takes to perform the code coverage analysis, unfortunately, I don't see any increase at all. The time taken is exactly the same as before (to the second!), even after multiple runs, I've also tested by increasing the amount of assemblies that run in parallel from 1 to the maximum of 8 as per your recommendation (though the location of this setting in the settings dialog suggest this applies to unit tests execution and not code coverage).
I'm not too concerned by the amount of time it takes for code coverage to complete though, I'd be happy if I could use continuous testing to run just the unit tests only, and manually initiate the code coverage analysis, at a time that suits me, i.e. at the end of a sprint. Is this something that could be considered?
Hi Bryan,
Thanks for trying the EAP build!
Actually dotCover doesn't start coverage analysis "after" running tests in continuous testing session. It executes coverage analysis while running tests. When tests are finished and "Coverage analysis in progress..." message is still on, dotCover executes post-processing of coverage data collected during the tests run.
Unfortunately we can't run tests in continuous testing without coverage - continuous testing needs coverage analysis in order to know which code affects which tests.
It's a bit surprising for me that the EAP build doesn't help to reduce post-processing time. I would very appreciate if you could make a video screencast of what you see (you may use free tool http://www.techsmith.com/jing.html).
Also could you please enable ReSharper logging, reproduce the issue and send us log files? In order to enable logging please start Visual Studio with the following command line arguments:
devenv.exe /ReSharper.LogLevel Verbose /ReSharper.LogFile
Log files will be saved in the '%Temp%\JetLogs' folder with auto-generated names, so empty this folder before you launch VS.
Please note that enabled logging can make things even slower.
Thanks in advance!
I've performed some further testing with this large project (I'm not allowed to provide logs or screens caps for this project I'm afraid).
In the "Unit Test Sessions" window, when I run all the unit tests and perform a full code coverage analysis, the unit tests are completed within 12 seconds, however it takes 5 mins and 30 seconds for the code coverage analysis to complete.
In the "Continuous Testing Session" window, if I run all the unit tests and a full code coverage analysis, the unit tests take 59 seconds to complete, and the code coverage analysis takes 5 minutes and 11 seconds to complete.
By making one of the projects dirty, and allowing continuous testing to naturally run, the dirty tests complete in 8 seconds, with the continuous testing taking exactly 1 minute to complete.
The problem is, during that 1 minute window, further tests are blocked from being run, which when you're on a bit of a coding surge, can really affect productivity. (1 minutes seems like an eternity when you're waiting for it).
Would it be possible to add an option to continuous testing, such that the any currently running analysis can be abandoned & restarted if the solution/project is built again whilst the analysis is still being performed? That'd solve the problem for me (and probably others too).
Thanks for your help.
Hi Bryan,
Continuous testing already behaves as you described: it tries to cancel current analysis and start next one in case of new "save/build" event. But there are now parts of analysis process that can't be cancelled. Otherwise we can get inconsistent data. We will think on improving this behavior.
But it seems to me that 5 min (and even 1 min) it's abnormally long post-processing time. Probably you could send me Unit Test Session window log? It's quite brief and you can wipe out any private info from it (like paths and assembly names)? In order to get it press "OK" ("Show execution log") button in the right upper corner of the Unit Test Session window right after coverage analysis has finished. Open context menu and select "Filter by Severity >> TRACE". Open context menu again and select "Copy to Clipboard".
I've attached a log from a full run through of all the unit tests and code coverage. The project consists of 58 projects (inc test projects), with 910 unit tests in total.
The obfuscated log can be found here
It's probably worth mentioning, this is from R# 2017.2 EAP, Visual Studio 2017, Windows 10 x64, 8GB, Intel i7-4790 3.6GHz, 256GB SSD.
Many thanks.
Thanks for the log! But it looks like it wasn't done in the latest 2017.2 EAP. Could you please copy and send me the info from ReSharper's About window (ReSharper >> Help >> About JetBrains ReSharper Ultimate... >> "Copy >>")?
Ooh, well spotted! Yes, I am still running 2017.1.3, despite installing 2017.2 EAP yesterday. The installation appeared to go okay, and reported success, though I may have installed it with my administrator account (different user profile). I'm reinstalling again, this time, without using my administrator account.
I'll repeat the tests and report back once I've confirmed that the correct version is installed and working.
Thanks.
First run of testing after updating, the unit tests and code coverage completed in 1 minute and 2 seconds.
After that, I made a project dirty, and allowed code coverage to run by itself, this time it completed all unit tests and code coverage analysis in just 23 seconds, so that's quite a significant improvement, compared to the previous time of 1 minute with 2017.1.3
Resharper did throw an exception though (The given key was not present in the dictionary), but it didn't seem to cause a problem, and I've submitted the details of that exception separately via the built in exception browser.
Thanks for your help, I'll see how I get on with the much improved performance.
Glad to hear this. Thanks for the assistance!