Test Run Duration is Obviously Wrong

I have been running a few hundred acceptance tests using the ReSharper test runner. I started noticing that the R# test runner reported duration seemed off. The durations I've observed and the means of measurement are below

    Wall clock             30 - 40 minutes
    Visual Studio TRX      30 minutes
    ReSharper test runner  1 hour 24 minutes

The R# test runner reports individual test run times in what appears to be [minutes:seconds.milliseconds] or [0:00.00]. For example, I observed that one of the tests took 6 seconds (wall clock) and the R# test runner reported

    Update_the_thing    [0:06.03] Success

The total test run duration was shown in the same format [0:00.00]

    Test Run    [1:23.97]

So, R# thinks my whole test run took 1 minute and 23 seconds. If we assume the format is [hours:minutes.seconds], that makes even less sense. Individual tests are not taking 6 or so minutes. And 1 hour and 23 minutes is twice the observed wall clock duration for the entire suite.

2 comments
Comment actions Permalink

I've run some smaller test sets to look at the test run duration and do some calculations manually... but I was stopped short when I saw the following (reproduced from the report)

    <test-project> (6 tests)        [0:03.83] Success
        <test-namespace> (6 tests)  [0:03.83] Success
            test-class (6 tests)    [0:03.83] Success
                test-1              [0:15.14] Success
                test-2              [0:04.18] Success
                test-3              [0:03.77] Success
                test-4              [0:03.85] Success
                test-5              [0:04.71] Success
                test-6              [0:03.83] Success

Do you see what I see? The test run duration reported is the same as the last test in the test run. I performed several small test runs in the same session, each time, the total time was simply the sum of the last test in each suite.

0
Comment actions Permalink

Hello Anthony,

I've assigned a task to our QA engineer: http://youtrack.jetbrains.net/issue/RSRP-199114
and we will try reproducing this behavior on our side. Thank you!

Andrey Serebryansky
Senior Support Engineer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

I have been running a few hundred http://specflow.org/ using the
ReSharper test runner. I started noticing that the R# test runner
reported duration seemed off. The durations I've observed and the
means of measurement are below

Wall clock             30 - 40 minutes
Visual Studio TRX      30 minutes
ReSharper test runner  1 hour 24 minutes
The R# test runner reports individual test run times in what appears
to be or . For example, I
observed that one of the tests took 6 seconds (wall clock) and the R#
test runner reported

Update_the_thing    Success

The total test run duration was shown in the same format

Test Run   

So, R# thinks my whole test run took 1 minute and 23 seconds. If we
assume the format is , that makes even less
sense. Individual tests are not taking 6 or so minutes. And 1 hour and
23 minutes is twice the observed wall clock duration for the entire
suite.

---
Original message URL:
http://devnet.jetbrains.net/message/5279200#5279200



0

Please sign in to leave a comment.