Any performance improvements planned for test runner?

I've been doing some primitive benchmarks on test runners lately, and I found
that the R# test runner is almost twice a slow as the NUnit-gui test runner.
The test case was 1000 trivial tests (Assert.IsTrue(true)). I know this test
case is very trivial, but most unit tests should run almost this quickly,
based on the act of creating lots of small, focused, fast tests.

For 1000 tests, NUnit-gui ran them in 15 seconds, nunit-console ran them
in 7 seconds, and R# ran them in 30 seconds.

R# was run in hierarchical mode, not following the currently running tests.
Tests were organized into 10 classes, 100 tests per class.

bab


6 comments
Comment actions Permalink

I think futher UnitTestRunner performace improvement is a tricky thing. 'Cos
we should always have the output in-sync with current selection.

Moreover, our UnitTestRunner is not intented for batch running. The primary
goal is debugging. So the usability of the debugging features are more
important than the performance ones.

--
Eugene Pasynkov
Developer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"
"Brian Button" <bbutton@agilestl.com> wrote in message
news:b9a1d86b8cc078c7960bfada4820@news.jetbrains.com...

I've been doing some primitive benchmarks on test runners lately, and I
found that the R# test runner is almost twice a slow as the NUnit-gui test
runner. The test case was 1000 trivial tests (Assert.IsTrue(true)). I know
this test case is very trivial, but most unit tests should run almost this
quickly, based on the act of creating lots of small, focused, fast tests.

>

For 1000 tests, NUnit-gui ran them in 15 seconds, nunit-console ran them
in 7 seconds, and R# ran them in 30 seconds.

>

R

  1. was run in hierarchical mode, not following the currently running

tests. Tests were organized into 10 classes, 100 tests per class.

>

bab

>



0
Comment actions Permalink

Hello Eugene,

Moreover, our UnitTestRunner is not intented for batch running. The
primary goal is debugging. So the usability of the debugging features are
more important than the performance ones.


First let me say that your NUnit integration is the best I've seen. I use it
every day!

In TDD you try to focus on running your tests very frequent. You program in
very small steps and in between these steps you run your tests. Because of
test performance sometimes I didn't run all tests and an error creeps in the
code, found later. So for me the performance of the unit test runner is
critical.

But maybe you can explain the technical problem you see to get the same
performance as NUnit-gui.


P.S. It would be nice to have a "Run all tests in Solution" function which
scans all projects for tests and runs them.


Sincerely,
Stefan Lieser


0
Comment actions Permalink

1) The main performance penalty is caused by output window (and the
necessity to sync the output with selection in the tree on every action)

2) There is the ability to run all tests in solution - just right-click on
solution node in the Solution Explorer, and select "Run Unit Tests"

--
Eugene Pasynkov
Developer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"
"Stefan Lieser" <slieser@t-online.de> wrote in message
news:dhtg7t$tk0$1@is.intellij.net...

Hello Eugene,

>
>> Moreover, our UnitTestRunner is not intented for batch running. The
>> primary goal is debugging. So the usability of the debugging features are
>> more important than the performance ones.
>

First let me say that your NUnit integration is the best I've seen. I use
it every day!

>

In TDD you try to focus on running your tests very frequent. You program
in very small steps and in between these steps you run your tests. Because
of test performance sometimes I didn't run all tests and an error creeps
in the code, found later. So for me the performance of the unit test
runner is critical.

>

But maybe you can explain the technical problem you see to get the same
performance as NUnit-gui.

>
>

P.S. It would be nice to have a "Run all tests in Solution" function which
scans all projects for tests and runs them.

>
>

Sincerely,
Stefan Lieser



0
Comment actions Permalink

Hello Eugene,

1) The main performance penalty is caused by output window (and the
necessity to sync the output with selection in the tree on every action)


I see.

>

2) There is the ability to run all tests in solution - just right-click on
solution node in the Solution Explorer, and select "Run Unit Tests"


Oops, I didn't try that. Thanks!!


Sincerely,
Stefan Lieser


0
Comment actions Permalink

Hello Eugene,

I think futher UnitTestRunner performace improvement is a tricky
thing. 'Cos we should always have the output in-sync with current
selection.

Moreover, our UnitTestRunner is not intented for batch running. The
primary goal is debugging. So the usability of the debugging features
are more important than the performance ones.


Are you saying that the test runner is only really supposed to be used to
allow people to pick and choose individual tests to run through the debugger,
and the use case of choosing all tests in a project and running them is secondary?
That would seem to be completely contradictory to the spirit of TDD. Have
you talked with your IntelliJ folks about this? I'm pretty sure that IntelliJ
supports, helps, promotes, encourages, and facilitates TDD every way that
it can.

I'm sorry, but your answer just doesn't ring true. Everything else that your
projects do seems very much in line with the agile way of thinking -- this
comment does not.

Can you please clarify?

Thanks,
bab


0
Comment actions Permalink

Hello Brian,

We'll try to optimize our runner further and I hope will achieve better speed.
Thanks for your feedback!

Valentin Kipiatkov
Chief Scientist, Vice President of Product Development
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Hello Eugene,

>> I think futher UnitTestRunner performace improvement is a tricky
>> thing. 'Cos we should always have the output in-sync with current
>> selection.
>>
>> Moreover, our UnitTestRunner is not intented for batch running. The
>> primary goal is debugging. So the usability of the debugging features
>> are more important than the performance ones.
>>

Are you saying that the test runner is only really supposed to be used
to allow people to pick and choose individual tests to run through the
debugger, and the use case of choosing all tests in a project and
running them is secondary? That would seem to be completely
contradictory to the spirit of TDD. Have you talked with your IntelliJ
folks about this? I'm pretty sure that IntelliJ supports, helps,
promotes, encourages, and facilitates TDD every way that it can.

I'm sorry, but your answer just doesn't ring true. Everything else
that your projects do seems very much in line with the agile way of
thinking -- this comment does not.

Can you please clarify?

Thanks,
bab



0

Please sign in to leave a comment.