Test Run Duration is Obviously Wrong
I have been running a few hundred acceptance tests using the ReSharper test runner. I started noticing that the R# test runner reported duration seemed off. The durations I've observed and the means of measurement are below
Wall clock 30 - 40 minutes
Visual Studio TRX 30 minutes
ReSharper test runner 1 hour 24 minutes
The R# test runner reports individual test run times in what appears to be [minutes:seconds.milliseconds] or [0:00.00]. For example, I observed that one of the tests took 6 seconds (wall clock) and the R# test runner reported
Update_the_thing [0:06.03] Success
The total test run duration was shown in the same format [0:00.00]
Test Run [1:23.97]
So, R# thinks my whole test run took 1 minute and 23 seconds. If we assume the format is [hours:minutes.seconds], that makes even less sense. Individual tests are not taking 6 or so minutes. And 1 hour and 23 minutes is twice the observed wall clock duration for the entire suite.
Please sign in to leave a comment.
I've run some smaller test sets to look at the test run duration and do some calculations manually... but I was stopped short when I saw the following (reproduced from the report)
<test-project> (6 tests) [0:03.83] Success
<test-namespace> (6 tests) [0:03.83] Success
test-class (6 tests) [0:03.83] Success
test-1 [0:15.14] Success
test-2 [0:04.18] Success
test-3 [0:03.77] Success
test-4 [0:03.85] Success
test-5 [0:04.71] Success
test-6 [0:03.83] Success
Do you see what I see? The test run duration reported is the same as the last test in the test run. I performed several small test runs in the same session, each time, the total time was simply the sum of the last test in each suite.
Hello Anthony,
I've assigned a task to our QA engineer: http://youtrack.jetbrains.net/issue/RSRP-199114
and we will try reproducing this behavior on our side. Thank you!
Andrey Serebryansky
Senior Support Engineer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"