Unit tests and performance

Hi,

We're experiencing performance problems running NUnit test with ReSharper. Running a simple test like this, just printing to the console, takes well over 10 seconds:

Using log4net.Info("") instead of Console.WriteLine("") increases the execution time to over 30 seconds. The same test takes about 15-30 milliseconds to run from a console application or the NUnit gui.

We're running the tests with ReSharper 2.0, build 206 in Visual Studio 2005 June CTP on Windows XP with SP2, lots of ram and processor etc.

Btw, when these kinds of problems are sorted out, we're definitly investing in a bunch of licenses because ReSharper rocks :)

//Spoiled IDEA and Eclipse user

18 comments
Comment actions Permalink

Ok, my first post on this topic obviously caught no ones attention, so I'll try to rephrase:

Printing to the console in a unit test run from ReSharper is slow.

Running the same code from elsewhere is fast.

Has anyone else noticed this behaviour? Can anyone else reproduce this?

This is not a non-issue to us, tests that print a lot to the console using log4net/Console.WriteLine almost slows to a grinding halt taking several minutes to complete instead of a few seconds.

Best regards

0
Comment actions Permalink

Erik,

The developer, who is responsible for file templates is currently on PDC,
they are going to return on 19th if I'm not mistaken. I hope, he'll clarify
the question.

--
Sergey V. Coox
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

I've also noticed how slow the unit test runner is, which drives me crazy
because it's so productive for me!
Derek

EB> Ok, my first post on this topic obviously caught no ones attention,
EB> so I'll try to rephrase:
EB>
EB> Printing to the console in a unit test run from ReSharper is
EB> slow.
EB>
EB> Running the same code from elsewhere is fast.
EB>
EB> Has anyone else noticed this behaviour? Can anyone else reproduce
EB> this?
EB>
EB> This is not a non-issue to us, tests that print a lot to the console
EB> using log4net/Console.WriteLine almost slows to a grinding halt
EB> taking several minutes to complete instead of a few seconds.
EB>
EB> Best regards
EB>


0
Comment actions Permalink

Hello Derek,

I've also noticed how slow the unit test runner is, which drives me
crazy
because it's so productive for me!


This is very interesting... I've had great results with the unit test runner.
It is fast, responsive, and lets me get into that TDD rhythm. It is much
faster than the VSTS test runner built into higher versions of Visual Studio.Net.

bab


0
Comment actions Permalink

Brian,

Do you print a lot to the console with your tests?
Our tests run smoothly and as fast as one would expect if logging to the console is turned off. Logging to file doesn't seem to affect the performance.

lets me get into that TDD rhythm

Agree, I've come to depend on tools like these when doing test driven development.

Peace

0
Comment actions Permalink

Hello Erik,

Brian,

Do you print a lot to the console with your tests?


No. We actually consciously avoid doing this. We do, however, have lots of
tests that expect exceptions, and the exception text seems to be being dumped
to the output window for us (by the runner???), so we do have a lot of text
spitting out. It doesn't seem to slow us down at all.

bab


0
Comment actions Permalink

Hello Brian,

I just ran 167 tests in one of my test assemblies in NUnit that took 8.5
seconds. The Test Runner is still running after 10 minutes and has only
completed 16 tests. This is certainly not responsive and fast! I want to
use the Test Runner so badly, but this is not acceptable! I wish I knew
what I could do to help figure out where the bottleneck is. The CPU is at
6% with Omea Reader taking up 2%, so I'm not seeing any CPU spikes.

Derek

BB> Hello Derek,
BB>
>> I've also noticed how slow the unit test runner is, which drives me
>> crazy
>> because it's so productive for me!
BB> This is very interesting... I've had great results with the unit
BB> test runner. It is fast, responsive, and lets me get into that TDD
BB> rhythm. It is much faster than the VSTS test runner built into
BB> higher versions of Visual Studio.Net.
BB>
BB> bab
BB>


0
Comment actions Permalink

Can anyone from JetBrains please tell me how to speed things up here or
at least how to send helpful debugging info?

Thanks,
Derek

DP> Hello Brian,
DP>
DP> I just ran 167 tests in one of my test assemblies in NUnit that took
DP> 8.5 seconds. The Test Runner is still running after 10 minutes and
DP> has only completed 16 tests. This is certainly not responsive and
DP> fast! I want to use the Test Runner so badly, but this is not
DP> acceptable! I wish I knew what I could do to help figure out where
DP> the bottleneck is. The CPU is at 6% with Omea Reader taking up 2%,
DP> so I'm not seeing any CPU spikes.
DP>
DP> Derek
DP>


0
Comment actions Permalink

Hello Derek,

it was mentioned in this thread that console output may relate to performance
issue.
Do you test write a lot to console?


Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Can anyone from JetBrains please tell me how to speed things up here
or at least how to send helpful debugging info?

Thanks,
Derek

DP>> Hello Brian,
DP>>
DP>> I just ran 167 tests in one of my test assemblies in NUnit that
DP>> took 8.5 seconds. The Test Runner is still running after 10
DP>> minutes and has only completed 16 tests. This is certainly not
DP>> responsive and fast! I want to use the Test Runner so badly, but
DP>> this is not acceptable! I wish I knew what I could do to help
DP>> figure out where the bottleneck is. The CPU is at 6% with Omea
DP>> Reader taking up 2%, so I'm not seeing any CPU spikes.
DP>>
DP>> Derek
DP>>


0
Comment actions Permalink

Hello Dmitry,

Let me review some of my test assemblies and I'll get back to you.

Thanks,
Derek

DS> Hello Derek,
DS>
DS> it was mentioned in this thread that console output may relate to
DS> performance
DS> issue.
DS> Do you test write a lot to console?
DS> Regards,
DS> Dmitry Shaporenkov
DS> JetBrains, Inc
DS> http://www.jetbrains.com
DS> "Develop with pleasure!"
>> Can anyone from JetBrains please tell me how to speed things up
>> here or at least how to send helpful debugging info?
>>
>> Thanks,
>> Derek


0
Comment actions Permalink

I'll fix performance issues with console output in UnitTestRunner shortly

--
Eugene Pasynkov
Developer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"
"Derek Price" <derek.price@gmail.com> wrote in message
news:71a7df64f140e8c78b11e2f65686@news.intellij.net...

Can anyone from JetBrains please tell me how to speed things up here or
at least how to send helpful debugging info?
Thanks,
Derek

>

DP> Hello Brian,
DP> DP> I just ran 167 tests in one of my test assemblies in NUnit that
took
DP> 8.5 seconds. The Test Runner is still running after 10 minutes and
DP> has only completed 16 tests. This is certainly not responsive and
DP> fast! I want to use the Test Runner so badly, but this is not
DP> acceptable! I wish I knew what I could do to help figure out where
DP> the bottleneck is. The CPU is at 6% with Omea Reader taking up 2%,
DP> so I'm not seeing any CPU spikes.
DP> DP> Derek
DP>



0
Comment actions Permalink

Is there any chance we'll be able to configure a 'run directory' for the
unit test runner soon?

Using the NunitGUI application to run our unit tests, we can specify a
command line, so it works well. But I can't find a place to do this in
Resharper's test runner, which makes it useless to us.

Our solution consists of a lot of projects that build DLLs, and an
application that loads those dlls on demand at runtime. Therefore we have
post-build steps that copy all the dlls into one common directory, and since
everything is linked at runtime, nothing works unless ALL the dlls are
available in the same directory.

Without the ability to speicfy to the test runner that it shoudln't run the
dll output into the project/bin/debug directory, but to run the one in the
global /bin directory, the unit test runner is essentially useless for us.

I'd LOVE to use it, if it were possible to configure it to be useable for
us. Pretty-please?

"Eugene Pasynkov (JetBrains)" <Eugene.Pasynkov@jetbrains.com> wrote in
message news:dgofof$3rt$1@is.intellij.net...

I'll fix performance issues with console output in UnitTestRunner shortly

>

--
Eugene Pasynkov
Developer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"
"Derek Price" <derek.price@gmail.com> wrote in message
news:71a7df64f140e8c78b11e2f65686@news.intellij.net...

>> Can anyone from JetBrains please tell me how to speed things up here or
>> at least how to send helpful debugging info?
>> Thanks,
>> Derek
>>
>> DP> Hello Brian,
>> DP> DP> I just ran 167 tests in one of my test assemblies in NUnit that
>> took
>> DP> 8.5 seconds. The Test Runner is still running after 10 minutes and
>> DP> has only completed 16 tests. This is certainly not responsive and
>> DP> fast! I want to use the Test Runner so badly, but this is not
>> DP> acceptable! I wish I knew what I could do to help figure out where
>> DP> the bottleneck is. The CPU is at 6% with Omea Reader taking up 2%,
>> DP> so I'm not seeing any CPU spikes.
>> DP> DP> Derek
>> DP>
>>
>



0
Comment actions Permalink

Why can't you build all your projects to the single directory, thus omitting
the post-build step?

--
Eugene Pasynkov
Developer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"
"Paul Bradshaw" <pbradshaw@advsol.com> wrote in message
news:dgpb3q$l52$1@is.intellij.net...

Is there any chance we'll be able to configure a 'run directory' for the
unit test runner soon?

>

Using the NunitGUI application to run our unit tests, we can specify a
command line, so it works well. But I can't find a place to do this in
Resharper's test runner, which makes it useless to us.

>

Our solution consists of a lot of projects that build DLLs, and an
application that loads those dlls on demand at runtime. Therefore we have
post-build steps that copy all the dlls into one common directory, and
since everything is linked at runtime, nothing works unless ALL the dlls
are available in the same directory.

>

Without the ability to speicfy to the test runner that it shoudln't run
the dll output into the project/bin/debug directory, but to run the one in
the global /bin directory, the unit test runner is essentially useless for
us.

>

I'd LOVE to use it, if it were possible to configure it to be useable for
us. Pretty-please?

>

"Eugene Pasynkov (JetBrains)" <Eugene.Pasynkov@jetbrains.com> wrote in
message news:dgofof$3rt$1@is.intellij.net...

>> I'll fix performance issues with console output in UnitTestRunner shortly
>>
>> --
>> Eugene Pasynkov
>> Developer
>> JetBrains, Inc
>> http://www.jetbrains.com
>> "Develop with pleasure!"
>> "Derek Price" <derek.price@gmail.com> wrote in message
>> news:71a7df64f140e8c78b11e2f65686@news.intellij.net...
>>> Can anyone from JetBrains please tell me how to speed things up here
>>> or at least how to send helpful debugging info?
>>> Thanks,
>>> Derek
>>>
>>> DP> Hello Brian,
>>> DP> DP> I just ran 167 tests in one of my test assemblies in NUnit that
>>> took
>>> DP> 8.5 seconds. The Test Runner is still running after 10 minutes and
>>> DP> has only completed 16 tests. This is certainly not responsive and
>>> DP> fast! I want to use the Test Runner so badly, but this is not
>>> DP> acceptable! I wish I knew what I could do to help figure out where
>>> DP> the bottleneck is. The CPU is at 6% with Omea Reader taking up 2%,
>>> DP> so I'm not seeing any CPU spikes.
>>> DP> DP> Derek
>>> DP>
>>>
>>
>>
>



0
Comment actions Permalink

This is just the way our huge solution is configured. There are many
reasons, too numerous to go into, but fact is that there is no way we can
change all our developer's configs of almost a hundred different projects
(so many that nobody EVER loads them all into a single solution) just so we
can use Resharper's test runner, when other test runners work just fine.

I'd LOVE to use the convenient built-in test runner, but it's simply
impossible right now, since virtually every DLL requires a .config file, and
all those config files are in the common bin directory... and since every
DLL requires run-time access to the other DLLs, and attempts to load them
will fail unless run from the common bin directory, where all the DLLs are
copied.

I can't believe that we're the only ones in the universe that have projects,
solutions, and applications configured this way (given that all other test
runners seem to handle this situation). Just because the Test Runner is
integrated with the IDE rather than as a separate external application
doesn't mean we should lose necessary configuration flexibility.

As far as I can tell, I really only need one configuration value ... if a
directory is set for a solution, run all unit tests from that solution
instead of the target directory. If not, do the default behavior. I don't
need (or really want) anything that is "per project", just the ability to
specify a directory to override the target directory at the solution level.
Most people would never use it (or need to see it), but having it would make
the test runner usable in my company, and it would be a big win for pushing
them to upgrade all our seats to 2.0 :)

"Eugene Pasynkov (JetBrains)" <Eugene.Pasynkov@jetbrains.com> wrote in
message news:dgr63t$qc8$1@is.intellij.net...

Why can't you build all your projects to the single directory, thus
omitting the post-build step?

>

--
Eugene Pasynkov
Developer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"
"Paul Bradshaw" <pbradshaw@advsol.com> wrote in message
news:dgpb3q$l52$1@is.intellij.net...

>> Is there any chance we'll be able to configure a 'run directory' for the
>> unit test runner soon?
>>
>> Using the NunitGUI application to run our unit tests, we can specify a
>> command line, so it works well. But I can't find a place to do this in
>> Resharper's test runner, which makes it useless to us.
>>
>> Our solution consists of a lot of projects that build DLLs, and an
>> application that loads those dlls on demand at runtime. Therefore we
>> have post-build steps that copy all the dlls into one common directory,
>> and since everything is linked at runtime, nothing works unless ALL the
>> dlls are available in the same directory.
>>
>> Without the ability to speicfy to the test runner that it shoudln't run
>> the dll output into the project/bin/debug directory, but to run the one
>> in the global /bin directory, the unit test runner is essentially useless
>> for us.
>>
>> I'd LOVE to use it, if it were possible to configure it to be useable for
>> us. Pretty-please?
>>
>> "Eugene Pasynkov (JetBrains)" <Eugene.Pasynkov@jetbrains.com> wrote in
>> message news:dgofof$3rt$1@is.intellij.net...
>>> I'll fix performance issues with console output in UnitTestRunner
>>> shortly
>>>
>>> --
>>> Eugene Pasynkov
>>> Developer
>>> JetBrains, Inc
>>> http://www.jetbrains.com
>>> "Develop with pleasure!"
>>> "Derek Price" <derek.price@gmail.com> wrote in message
>>> news:71a7df64f140e8c78b11e2f65686@news.intellij.net...
>>>> Can anyone from JetBrains please tell me how to speed things up here
>>>> or at least how to send helpful debugging info?
>>>> Thanks,
>>>> Derek
>>>>
>>>> DP> Hello Brian,
>>>> DP> DP> I just ran 167 tests in one of my test assemblies in NUnit that
>>>> took
>>>> DP> 8.5 seconds. The Test Runner is still running after 10 minutes and
>>>> DP> has only completed 16 tests. This is certainly not responsive and
>>>> DP> fast! I want to use the Test Runner so badly, but this is not
>>>> DP> acceptable! I wish I knew what I could do to help figure out where
>>>> DP> the bottleneck is. The CPU is at 6% with Omea Reader taking up 2%,
>>>> DP> so I'm not seeing any CPU spikes.
>>>> DP> DP> Derek
>>>> DP>
>>>>
>>>
>>>
>>
>>
>



0
Comment actions Permalink

Hi Eugene

I'm doing the same thing Paul is (copying all assemblies to a single directory and running the tests from there). Probably for other reasons though. In my case it's because I need files other than the base assemblies (there's a bunch of library assemblies and also some config files etc.)

Futhermore it's done to get past the assembly locking bug in VS (where VS locks an assembly if it goes over 68 kB because of intellisense performance).

So in all it would be really cool if we could point the testrunner to a specific directory to look in.

Regards Patrick

0
Comment actions Permalink

I see.
We'll consider to implement this option

--
Eugene Pasynkov
Developer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"
"Paul Bradshaw" <pbradshaw@advsol.com> wrote in message
news:dh94rp$c0l$1@is.intellij.net...

This is just the way our huge solution is configured. There are many
reasons, too numerous to go into, but fact is that there is no way we can
change all our developer's configs of almost a hundred different projects
(so many that nobody EVER loads them all into a single solution) just so
we can use Resharper's test runner, when other test runners work just
fine.

>

I'd LOVE to use the convenient built-in test runner, but it's simply
impossible right now, since virtually every DLL requires a .config file,
and all those config files are in the common bin directory... and since
every DLL requires run-time access to the other DLLs, and attempts to load
them will fail unless run from the common bin directory, where all the
DLLs are copied.

>

I can't believe that we're the only ones in the universe that have
projects, solutions, and applications configured this way (given that all
other test runners seem to handle this situation). Just because the Test
Runner is integrated with the IDE rather than as a separate external
application doesn't mean we should lose necessary configuration
flexibility.

>

As far as I can tell, I really only need one configuration value ... if a
directory is set for a solution, run all unit tests from that solution
instead of the target directory. If not, do the default behavior. I
don't need (or really want) anything that is "per project", just the
ability to specify a directory to override the target directory at the
solution level. Most people would never use it (or need to see it), but
having it would make the test runner usable in my company, and it would be
a big win for pushing them to upgrade all our seats to 2.0 :)

>

"Eugene Pasynkov (JetBrains)" <Eugene.Pasynkov@jetbrains.com> wrote in
message news:dgr63t$qc8$1@is.intellij.net...

>> Why can't you build all your projects to the single directory, thus
>> omitting the post-build step?
>>
>> --
>> Eugene Pasynkov
>> Developer
>> JetBrains, Inc
>> http://www.jetbrains.com
>> "Develop with pleasure!"
>> "Paul Bradshaw" <pbradshaw@advsol.com> wrote in message
>> news:dgpb3q$l52$1@is.intellij.net...
>>> Is there any chance we'll be able to configure a 'run directory' for the
>>> unit test runner soon?
>>>
>>> Using the NunitGUI application to run our unit tests, we can specify a
>>> command line, so it works well. But I can't find a place to do this in
>>> Resharper's test runner, which makes it useless to us.
>>>
>>> Our solution consists of a lot of projects that build DLLs, and an
>>> application that loads those dlls on demand at runtime. Therefore we
>>> have post-build steps that copy all the dlls into one common directory,
>>> and since everything is linked at runtime, nothing works unless ALL the
>>> dlls are available in the same directory.
>>>
>>> Without the ability to speicfy to the test runner that it shoudln't run
>>> the dll output into the project/bin/debug directory, but to run the one
>>> in the global /bin directory, the unit test runner is essentially
>>> useless for us.
>>>
>>> I'd LOVE to use it, if it were possible to configure it to be useable
>>> for us. Pretty-please?
>>>
>>> "Eugene Pasynkov (JetBrains)" <Eugene.Pasynkov@jetbrains.com> wrote in
>>> message news:dgofof$3rt$1@is.intellij.net...
>>>> I'll fix performance issues with console output in UnitTestRunner
>>>> shortly
>>>>
>>>> --
>>>> Eugene Pasynkov
>>>> Developer
>>>> JetBrains, Inc
>>>> http://www.jetbrains.com
>>>> "Develop with pleasure!"
>>>> "Derek Price" <derek.price@gmail.com> wrote in message
>>>> news:71a7df64f140e8c78b11e2f65686@news.intellij.net...
>>>>> Can anyone from JetBrains please tell me how to speed things up here
>>>>> or at least how to send helpful debugging info?
>>>>> Thanks,
>>>>> Derek
>>>>>
>>>>> DP> Hello Brian,
>>>>> DP> DP> I just ran 167 tests in one of my test assemblies in NUnit
>>>>> that took
>>>>> DP> 8.5 seconds. The Test Runner is still running after 10 minutes
>>>>> and
>>>>> DP> has only completed 16 tests. This is certainly not responsive and
>>>>> DP> fast! I want to use the Test Runner so badly, but this is not
>>>>> DP> acceptable! I wish I knew what I could do to help figure out
>>>>> where
>>>>> DP> the bottleneck is. The CPU is at 6% with Omea Reader taking up
>>>>> 2%,
>>>>> DP> so I'm not seeing any CPU spikes.
>>>>> DP> DP> Derek
>>>>> DP>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>



0
Comment actions Permalink

Just an update on the test runner in v207. I ran another test assembly that
had 603 tests and it finished in about 5 seconds. This is a great improvement!

Thanks,
Derek

Hello Brian,

I just ran 167 tests in one of my test assemblies in NUnit that took
8.5 seconds. The Test Runner is still running after 10 minutes and
has only completed 16 tests. This is certainly not responsive and
fast! I want to use the Test Runner so badly, but this is not
acceptable! I wish I knew what I could do to help figure out where
the bottleneck is. The CPU is at 6% with Omea Reader taking up 2%, so
I'm not seeing any CPU spikes.

Derek



0
Comment actions Permalink

Thank you.

--
Eugene Pasynkov
Developer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"
"Derek Price" <derek.price@delete.gmail.com> wrote in message
news:71a7df6464188c7917989b9ddf9@news.jetbrains.com...

Just an update on the test runner in v207. I ran another test assembly
that had 603 tests and it finished in about 5 seconds. This is a great
improvement!

>

Thanks,
Derek

>
>> Hello Brian,
>>
>> I just ran 167 tests in one of my test assemblies in NUnit that took
>> 8.5 seconds. The Test Runner is still running after 10 minutes and
>> has only completed 16 tests. This is certainly not responsive and
>> fast! I want to use the Test Runner so badly, but this is not
>> acceptable! I wish I knew what I could do to help figure out where
>> the bottleneck is. The CPU is at 6% with Omea Reader taking up 2%, so
>> I'm not seeing any CPU spikes.
>>
>> Derek
>



0

Please sign in to leave a comment.