Unit test runner question: build or not before running tests

Hi all,

I want to ask your advice in making a decision with one issue related to
ReSharper's unit test runner. The question is whether we should build projects
before running/debugging unit tests. Currently we do not build the project(s)
automatically when invoking "Run" or "Debug" from the Unit Test Runner window
(but build when invoking from the source code or Solution Explorer). But
this solution does not seem to be a good one (see below).

Below there are four solutions (including the currently implemented one)
that I can imagine:

1. Always build the project(s) before running/debugging unit tests.
Positive: always correct results, not confusing for users.
Negative: building projects may consume significant time (tens of seconds
in our case) even in case there were no changes at all.

2. (Currently implemented.) Do not build the project(s) automatically when
invoking "Run" or "Debug" from the Unit Test Runner window. There is a button
"Build Projects" right in the Unit Test Runner window that builds the corresponding
project(s).
Positive: no time spent for unnecessary build.
Negative: in case when the build is required, running unit tests from the
Unit Test Runner window requires 2 steps (both time consuming!): press "Build
Projects" (and wait), press "Run"/"Debug" (and wait again). It would be better
to have only one time cosuming step so that user can invoke it and switch
to some other activity for some period of time.

3. Have a toogle button "Build Project(s) before Run" in the Unit Test Runner
window.
Positive: the same as in #2 and it's negative side is avoided.
Negative: in case when the solution is big and I want to avoid unnecessary
building I need to check the button state before invoking Run or Debug.

4. Have a toogle button "Build Project(s) before Next Run" in the Unit Test
Runner window. Pressing this button affects the next Run/Debug from the window
and then button automatically untoggles.
Positive: the same as in #3 and I should not check the button state, I
should press it if I want to build the changes and should not do anything
otherwise.
Negative: a but complicated and non-intuitive behavior with the button
which untoggles automatically.

I, personnally, find #4 to be the most convenient solution for me. What do
you think?

Valentin Kipiatkov
Chief Scientist, Vice President of Product Development
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"


13 comments
Comment actions Permalink

Why not a mixture of both 3 and 4... basically a configuration option that
allows the user to choose from:

  • Do not build projects before running unit tests

  • Build projects before the next run only

  • Always build projects before running unit tests


This could be a toggle button or a combo box in the Unit Test Runner window,
or a configuration item in the "Unit Testing" configuration window.

My 2c
/gerrod

Hi all,

I want to ask your advice in making a decision with one issue related
to ReSharper's unit test runner. The question is whether we should
build projects before running/debugging unit tests. Currently we do
not build the project(s) automatically when invoking "Run" or "Debug"
from the Unit Test Runner window (but build when invoking from the
source code or Solution Explorer). But this solution does not seem to
be a good one (see below).

Below there are four solutions (including the currently implemented
one) that I can imagine:

1. Always build the project(s) before running/debugging unit tests.
Positive: always correct results, not confusing for users.
Negative: building projects may consume significant time (tens of
seconds
in our case) even in case there were no changes at all.
2. (Currently implemented.) Do not build the project(s) automatically
when
invoking "Run" or "Debug" from the Unit Test Runner window. There is a
button
"Build Projects" right in the Unit Test Runner window that builds the
corresponding
project(s).
Positive: no time spent for unnecessary build.
Negative: in case when the build is required, running unit tests
from the
Unit Test Runner window requires 2 steps (both time consuming!): press
"Build
Projects" (and wait), press "Run"/"Debug" (and wait again). It would
be better to have only one time cosuming step so that user can invoke
it and switch to some other activity for some period of time.

3. Have a toogle button "Build Project(s) before Run" in the Unit Test
Runner
window.
Positive: the same as in #2 and it's negative side is avoided.
Negative: in case when the solution is big and I want to avoid
unnecessary
building I need to check the button state before invoking Run or
Debug.
4. Have a toogle button "Build Project(s) before Next Run" in the Unit
Test
Runner window. Pressing this button affects the next Run/Debug from
the window
and then button automatically untoggles.
Positive: the same as in #3 and I should not check the button state,
I
should press it if I want to build the changes and should not do
anything
otherwise.
Negative: a but complicated and non-intuitive behavior with the
button
which untoggles automatically.
I, personnally, find #4 to be the most convenient solution for me.
What do you think?

Valentin Kipiatkov
Chief Scientist, Vice President of Product Development
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"



0
Comment actions Permalink

I'd go with option #3, because of the non-intuitiveness of #4.

It allows the user to select his preferred default behaviour (always or
never build the projects) and change it if needed.

In case you want to always build before running the tests (probably the best
choice for small projects), #3 and #1 are the only ones really usable. And
#1 is only usable in this case :)


Anyway, isn't there a way to detect when something (anything, to keep it
simple) has been changed in a project and only build it in this case? It
would make for a better option #1 :)


Regards,

Lionel


0
Comment actions Permalink

Hello Valentin,

I vote for 3.

Regrads,
Maxim


0
Comment actions Permalink

I would go with 3, giving an option to toggle would work for me, too many
times i make the change and run the tests only to find that i forgot to do
the build on the library...

Krishna

"Valentin Kipiatkov (JetBrains)" <valentin@jetbrains.com> wrote in message
news:3fdb29a6b08928c81c0987922ac1@news.intellij.net...

Hi all,

>

I want to ask your advice in making a decision with one issue related to
ReSharper's unit test runner. The question is whether we should build
projects before running/debugging unit tests. Currently we do not build
the project(s) automatically when invoking "Run" or "Debug" from the Unit
Test Runner window (but build when invoking from the source code or
Solution Explorer). But this solution does not seem to be a good one (see
below).
Below there are four solutions (including the currently implemented one)
that I can imagine:

>

1. Always build the project(s) before running/debugging unit tests.
Positive: always correct results, not confusing for users.
Negative: building projects may consume significant time (tens of seconds
in our case) even in case there were no changes at all.

>

2. (Currently implemented.) Do not build the project(s) automatically when
invoking "Run" or "Debug" from the Unit Test Runner window. There is a
button "Build Projects" right in the Unit Test Runner window that builds
the corresponding project(s).
Positive: no time spent for unnecessary build.
Negative: in case when the build is required, running unit tests from the
Unit Test Runner window requires 2 steps (both time consuming!): press
"Build Projects" (and wait), press "Run"/"Debug" (and wait again). It
would be better to have only one time cosuming step so that user can
invoke it and switch to some other activity for some period of time.

>

3. Have a toogle button "Build Project(s) before Run" in the Unit Test
Runner window.
Positive: the same as in #2 and it's negative side is avoided.
Negative: in case when the solution is big and I want to avoid
unnecessary building I need to check the button state before invoking Run
or Debug.

>

4. Have a toogle button "Build Project(s) before Next Run" in the Unit
Test Runner window. Pressing this button affects the next Run/Debug from
the window and then button automatically untoggles.
Positive: the same as in #3 and I should not check the button state, I
should press it if I want to build the changes and should not do anything
otherwise.
Negative: a but complicated and non-intuitive behavior with the button
which untoggles automatically.

>

I, personnally, find #4 to be the most convenient solution for me. What do
you think?

>

Valentin Kipiatkov
Chief Scientist, Vice President of Product Development
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

>



0
Comment actions Permalink

#3 seems the best option to me.

- Nate


Valentin Kipiatkov (JetBrains) wrote:

Hi all,

I want to ask your advice in making a decision with one issue related to
ReSharper's unit test runner. The question is whether we should build
projects before running/debugging unit tests. Currently we do not build
the project(s) automatically when invoking "Run" or "Debug" from the
Unit Test Runner window (but build when invoking from the source code or
Solution Explorer). But this solution does not seem to be a good one
(see below).
Below there are four solutions (including the currently implemented one)
that I can imagine:

1. Always build the project(s) before running/debugging unit tests.
Positive: always correct results, not confusing for users.
Negative: building projects may consume significant time (tens of
seconds in our case) even in case there were no changes at all.

2. (Currently implemented.) Do not build the project(s) automatically
when invoking "Run" or "Debug" from the Unit Test Runner window. There
is a button "Build Projects" right in the Unit Test Runner window that
builds the corresponding project(s).
Positive: no time spent for unnecessary build.
Negative: in case when the build is required, running unit tests from
the Unit Test Runner window requires 2 steps (both time consuming!):
press "Build Projects" (and wait), press "Run"/"Debug" (and wait again).
It would be better to have only one time cosuming step so that user can
invoke it and switch to some other activity for some period of time.

3. Have a toogle button "Build Project(s) before Run" in the Unit Test
Runner window.
Positive: the same as in #2 and it's negative side is avoided.
Negative: in case when the solution is big and I want to avoid
unnecessary building I need to check the button state before invoking
Run or Debug.

4. Have a toogle button "Build Project(s) before Next Run" in the Unit
Test Runner window. Pressing this button affects the next Run/Debug from
the window and then button automatically untoggles.
Positive: the same as in #3 and I should not check the button state, I
should press it if I want to build the changes and should not do
anything otherwise.
Negative: a but complicated and non-intuitive behavior with the button
which untoggles automatically.

I, personnally, find #4 to be the most convenient solution for me. What
do you think?

Valentin Kipiatkov
Chief Scientist, Vice President of Product Development
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0
Comment actions Permalink

Actually, I'd also like some way to disable the build-before-run when
invoking unit tests from the source code... especially in the case where
nothing has changed. With my solution, the build process can take literally
5 minutes (thanks to the stupid website project) and it really kills the
usefulness of that feature for me. I love the idea of being able to just
click on the little icon to run a specific test, but it takes so long, it's
really not worth it at this point.

(maybe a shift or ctrl click would avoid a rebuild, while regular click
would do the build? Not intuitive, but certainly useful)


"Valentin Kipiatkov (JetBrains)" <valentin@jetbrains.com> wrote in message
news:3fdb29a6b08928c81c0987922ac1@news.intellij.net...

Hi all,

>

I want to ask your advice in making a decision with one issue related to
ReSharper's unit test runner. The question is whether we should build
projects before running/debugging unit tests. Currently we do not build
the project(s) automatically when invoking "Run" or "Debug" from the Unit
Test Runner window (but build when invoking from the source code or
Solution Explorer). But this solution does not seem to be a good one (see
below).
Below there are four solutions (including the currently implemented one)
that I can imagine:

>

1. Always build the project(s) before running/debugging unit tests.
Positive: always correct results, not confusing for users.
Negative: building projects may consume significant time (tens of seconds
in our case) even in case there were no changes at all.

>

2. (Currently implemented.) Do not build the project(s) automatically when
invoking "Run" or "Debug" from the Unit Test Runner window. There is a
button "Build Projects" right in the Unit Test Runner window that builds
the corresponding project(s).
Positive: no time spent for unnecessary build.
Negative: in case when the build is required, running unit tests from the
Unit Test Runner window requires 2 steps (both time consuming!): press
"Build Projects" (and wait), press "Run"/"Debug" (and wait again). It
would be better to have only one time cosuming step so that user can
invoke it and switch to some other activity for some period of time.

>

3. Have a toogle button "Build Project(s) before Run" in the Unit Test
Runner window.
Positive: the same as in #2 and it's negative side is avoided.
Negative: in case when the solution is big and I want to avoid
unnecessary building I need to check the button state before invoking Run
or Debug.

>

4. Have a toogle button "Build Project(s) before Next Run" in the Unit
Test Runner window. Pressing this button affects the next Run/Debug from
the window and then button automatically untoggles.
Positive: the same as in #3 and I should not check the button state, I
should press it if I want to build the changes and should not do anything
otherwise.
Negative: a but complicated and non-intuitive behavior with the button
which untoggles automatically.

>

I, personnally, find #4 to be the most convenient solution for me. What do
you think?

>

Valentin Kipiatkov
Chief Scientist, Vice President of Product Development
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

>



0
Comment actions Permalink

I just wonder why it's impossible to build only those projects that were modified and needed for the particular unit test module? I have a solution containing ~70 projects and it tries to compile all 70 projects when I debug BaseUnitTest that depends on only one project.

0
Comment actions Permalink

Valentin Kipiatkov (JetBrains) <valentin@jetbrains.com> writes:

I, personnally, find #4 to be the most convenient solution for me. What do
you think?


For me, the current solution (#2) works fine. I'd opt for #3, if I still
have the build-button from #2.

Regards,
Martin

0
Comment actions Permalink

Hello Andy,

I suspect that the current behavior is affected by the VS's option 'Build
only startup project and dependencies
on run', since ReSharper essentially modifies the debug configuration of
the startup project of your solution in order to run unit tests. Try to enable
this option and check if it makes any difference.

Regards,
Dmitry Shaporenkov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

I just wonder why it's impossible to build only those projects that
were modified and needed for the particular unit test module? I have
a solution containing ~70 projects and it tries to compile all 70
projects when I debug BaseUnitTest that depends on only one project.



0
Comment actions Permalink

I vote for 3 too.

Antonio.

0
Comment actions Permalink

(maybe a shift or ctrl click would avoid a rebuild, while regular
click would do the build? Not intuitive, but certainly useful)


The negative side is that such feature would be never discovered by the majority
of users. Maybe a global option ("Build when running tests from source")?

Valentin Kipiatkov
Chief Scientist, Vice President of Product Development
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Actually, I'd also like some way to disable the build-before-run when
invoking unit tests from the source code... especially in the case
where nothing has changed. With my solution, the build process can
take literally 5 minutes (thanks to the stupid website project) and it
really kills the usefulness of that feature for me. I love the idea
of being able to just click on the little icon to run a specific test,
but it takes so long, it's really not worth it at this point.

(maybe a shift or ctrl click would avoid a rebuild, while regular
click would do the build? Not intuitive, but certainly useful)

"Valentin Kipiatkov (JetBrains)" <valentin@jetbrains.com> wrote in
message news:3fdb29a6b08928c81c0987922ac1@news.intellij.net...

>> Hi all,
>>
>> I want to ask your advice in making a decision with one issue related
>> to
>> ReSharper's unit test runner. The question is whether we should build
>> projects before running/debugging unit tests. Currently we do not
>> build
>> the project(s) automatically when invoking "Run" or "Debug" from the
>> Unit
>> Test Runner window (but build when invoking from the source code or
>> Solution Explorer). But this solution does not seem to be a good one
>> (see
>> below).
>> Below there are four solutions (including the currently implemented
>> one)
>> that I can imagine:
>> 1. Always build the project(s) before running/debugging unit tests.
>> Positive: always correct results, not confusing for users.
>> Negative: building projects may consume significant time (tens of
>> seconds
>> in our case) even in case there were no changes at all.
>> 2. (Currently implemented.) Do not build the project(s) automatically
>> when
>> invoking "Run" or "Debug" from the Unit Test Runner window. There is
>> a
>> button "Build Projects" right in the Unit Test Runner window that
>> builds
>> the corresponding project(s).
>> Positive: no time spent for unnecessary build.
>> Negative: in case when the build is required, running unit tests from
>> the
>> Unit Test Runner window requires 2 steps (both time consuming!):
>> press
>> "Build Projects" (and wait), press "Run"/"Debug" (and wait again). It
>> would be better to have only one time cosuming step so that user can
>> invoke it and switch to some other activity for some period of time.
>> 3. Have a toogle button "Build Project(s) before Run" in the Unit
>> Test
>> Runner window.
>> Positive: the same as in #2 and it's negative side is avoided.
>> Negative: in case when the solution is big and I want to avoid
>> unnecessary building I need to check the button state before invoking
>> Run
>> or Debug.
>> 4. Have a toogle button "Build Project(s) before Next Run" in the
>> Unit
>> Test Runner window. Pressing this button affects the next Run/Debug
>> from
>> the window and then button automatically untoggles.
>> Positive: the same as in #3 and I should not check the button state,
>> I
>> should press it if I want to build the changes and should not do
>> anything
>> otherwise.
>> Negative: a but complicated and non-intuitive behavior with the
>> button
>> which untoggles automatically.
>> I, personnally, find #4 to be the most convenient solution for me.
>> What do you think?
>>
>> Valentin Kipiatkov
>> Chief Scientist, Vice President of Product Development
>> JetBrains, Inc
>> http://www.jetbrains.com
>> "Develop with pleasure!"


0
Comment actions Permalink

Well, that'd be an option I guess (one can always MANUALLY rebuild if they
need to, and then click the icon to run the test). In fact, if you do
nothing else, I hope you implement that, because I'd set it to never rebuild
when clicking the icon from the source, and just handle rebuilds manually if
needed. That's just the way I work anyway.

But hey, there are dozens of features in Resharper that many people would
never know about unless they read the documentation or look at the "cheat
sheet" keymap you guys always ship... what's one more? :)


"Valentin Kipiatkov (JetBrains)" <valentin@jetbrains.com> wrote in message
news:3fdb29a6b0bd88c81c93d11179d1@news.intellij.net...
>> (maybe a shift or ctrl click would avoid a rebuild, while regular
>> click would do the build? Not intuitive, but certainly useful)
>

The negative side is that such feature would be never discovered by the
majority of users. Maybe a global option ("Build when running tests from
source")?

>

Valentin Kipiatkov
Chief Scientist, Vice President of Product Development
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

>
>> Actually, I'd also like some way to disable the build-before-run when
>> invoking unit tests from the source code... especially in the case
>> where nothing has changed. With my solution, the build process can
>> take literally 5 minutes (thanks to the stupid website project) and it
>> really kills the usefulness of that feature for me. I love the idea
>> of being able to just click on the little icon to run a specific test,
>> but it takes so long, it's really not worth it at this point.
>>
>> (maybe a shift or ctrl click would avoid a rebuild, while regular
>> click would do the build? Not intuitive, but certainly useful)
>>
>> "Valentin Kipiatkov (JetBrains)" <valentin@jetbrains.com> wrote in
>> message news:3fdb29a6b08928c81c0987922ac1@news.intellij.net...
>>
>>> Hi all,
>>>
>>> I want to ask your advice in making a decision with one issue related
>>> to
>>> ReSharper's unit test runner. The question is whether we should build
>>> projects before running/debugging unit tests. Currently we do not
>>> build
>>> the project(s) automatically when invoking "Run" or "Debug" from the
>>> Unit
>>> Test Runner window (but build when invoking from the source code or
>>> Solution Explorer). But this solution does not seem to be a good one
>>> (see
>>> below).
>>> Below there are four solutions (including the currently implemented
>>> one)
>>> that I can imagine:
>>> 1. Always build the project(s) before running/debugging unit tests.
>>> Positive: always correct results, not confusing for users.
>>> Negative: building projects may consume significant time (tens of
>>> seconds
>>> in our case) even in case there were no changes at all.
>>> 2. (Currently implemented.) Do not build the project(s) automatically
>>> when
>>> invoking "Run" or "Debug" from the Unit Test Runner window. There is
>>> a
>>> button "Build Projects" right in the Unit Test Runner window that
>>> builds
>>> the corresponding project(s).
>>> Positive: no time spent for unnecessary build.
>>> Negative: in case when the build is required, running unit tests from
>>> the
>>> Unit Test Runner window requires 2 steps (both time consuming!):
>>> press
>>> "Build Projects" (and wait), press "Run"/"Debug" (and wait again). It
>>> would be better to have only one time cosuming step so that user can
>>> invoke it and switch to some other activity for some period of time.
>>> 3. Have a toogle button "Build Project(s) before Run" in the Unit
>>> Test
>>> Runner window.
>>> Positive: the same as in #2 and it's negative side is avoided.
>>> Negative: in case when the solution is big and I want to avoid
>>> unnecessary building I need to check the button state before invoking
>>> Run
>>> or Debug.
>>> 4. Have a toogle button "Build Project(s) before Next Run" in the
>>> Unit
>>> Test Runner window. Pressing this button affects the next Run/Debug
>>> from
>>> the window and then button automatically untoggles.
>>> Positive: the same as in #3 and I should not check the button state,
>>> I
>>> should press it if I want to build the changes and should not do
>>> anything
>>> otherwise.
>>> Negative: a but complicated and non-intuitive behavior with the
>>> button
>>> which untoggles automatically.
>>> I, personnally, find #4 to be the most convenient solution for me.
>>> What do you think?
>>>
>>> Valentin Kipiatkov
>>> Chief Scientist, Vice President of Product Development
>>> JetBrains, Inc
>>> http://www.jetbrains.com
>>> "Develop with pleasure!"
>



0
Comment actions Permalink

Why I need to rebuild unit test project after change other project included in unit test project?

Resharper 2019.1.1

0

Please sign in to leave a comment.