Feature request: dynamically add unit test tasks

Hi folks. I'd like to put in a feature request for the next version of ReSharper.

I've been writing the unit test plugin for the xunit.net testing framework (xunitcontrib.codeplex.com) and I've hit a bit of a brick wall trying to support xunit's Theories. A Theory is like mstest's TestRow and nunit's TestCase, providing data rows for multiple calls into a single test.

For some scenarios, I can implement this such that each test element node in the hierarchy has multiple child nodes, one for each test row (theory) - this is how the mstest support works. The problem is that this is very hard to get right before the tests run. xunit's support for Theories is very extensible - data can come from attributes (which I can statically enumerate and provide a good experience for) or from properties, files, databases or via a custom attribute. It's these that are hard to do. I don't know without running the tests what those test rows are, how many there are, etc. And even if I do read the database or file or enumerate the property before running the tests, there is no guarantee that this doesn't change when the tests are run (for example, I could write a theory attribute to return a random number of rows, each containing random data. Or I could modify the database as I enumerate, perhaps to maintain a count of the number of times this test row passes).

So, I'd like to request that the remote unit test runner be able to add a new unit test task at runtime, rather than have all unit test tasks declared before hand.

(I think one way to implement it would be that the remote unit test runner passes a new unit test task to IRemoteTaskServer.TaskStarted, etc. This task wouldn't be known and so couldn't be mapped to an existing test element. But the task has the runner ID, so the correct plugin could be queried via a new interface method to return a UnitTestElement that could be used to update the UI.)

Sound like a good idea?

Thanks

Matt

Please sign in to leave a comment.