Resharper 3.0 beta (build 448), wrong order of tests

Resharper 3.0 beta (build 448), wrong order of tests
Build 423 (and Resharper 2.x.x) executed NUnit tests in the order they appear in the source code (just as NUnit itself does). But build 448 runs tests in alphabetical order.

For unit tests, the order of tests should be irrelevant, but for integration tests, in complex scenarios and when you do not want to build the whole fixture for every test, tests may be dependant on each other.

7 comments
Comment actions Permalink

Hello,

We appreciate your feedback.

The corresponding JIRA request has been created, and you are welcome to monitor
its status at http://www.jetbrains.net/jira/browse/RSRP-42385.

Best regards,
- Development Team.


0
Comment actions Permalink

Thanks for quick fix! I already switched back to build 423, but after this and one another quick fix of a showstopper I'll certtainly try new builds and continue with feedback :)

0
Comment actions Permalink

Mileta, even if NUnit has a habit of running tests in declared order, you are asking for a world of trouble by depending on it. Good unit testing practice includes having no interdependencies between tests, especially order of execution.

You haven't so much exposed a bug in R#; R# has exposed a bug in your test implementation. :)

0
Comment actions Permalink

Hello Jeremy,

There are other problems with alphabetic sorting, like

test1
test10
test11
test12
test2
test21

:)

Sincerely,
Ilya Ryzhenkov

JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"


JG> Mileta, even if NUnit has a habit of running tests in declared
JG> order, you are asking for a world of trouble by depending on it.
JG> Good unit testing practice includes having no interdependencies
JG> between tests, especially order of execution.
JG>
JG> You haven't so much exposed a bug in R#; R# has exposed a bug in
JG> your test implementation. :)
JG>


0
Comment actions Permalink

I said for unit tests order should be irrelevant.

But for integration tests (for example the ones that include database), the amount of time/cost needed to setup a fixture for every test that you would like could be so high to make automated testing not feasible. Then you try to minimize creating fixtures by making your previous test a fixture for your next test.

For example you need to test a CRUD API for a complex entity containing of a lot of smaller detail entities. You have Create, Retrieve, Update and Delete tests. You issue a Create test, then Retrieve for the same entity instance, because you do not want to create new entity as a fixture for Retrieve test.

I know this could be troublesome, but this works for us and save us a lot of time that could go to preparing proper fixtures in the database for every test.

0
Comment actions Permalink

Also, fixtures have to be simple, otherwise who is going to test them? You will need 'meta-tests' to automate testing fixtures!

0
Comment actions Permalink

At the end of the day, if you can't run each and every -marked method in any combination and in any order, you've got a problem with your tests.

Either cook up your own custom runner that has fully-controllable text execution order (though having said that, I have a project to maintain where this has happened, and I'll be pushing to see it removed in future) or make sure that your methods marked can run in any combination and in any order. If you need to compose smaller tests into larger ones, create new -marked methods that call down into the others in your required sequence. Still, even this requires careful consideration, as it would imply that the called -marked methods have side-effects which, since they need to be independent in and of themselves they should not, may indicate a problem with those tests.

0

Please sign in to leave a comment.