Documentation for writing UnitTestProvider

I compiled the unit test runner from the trunk at MbUnit, and execute R# in debugger using the "recipe" in Readme-debugging.txt.

Any project compiled against the MbUnit framework should suffice, I am attaching the simple project I use for testing. Not sure if you need the Gallio test framework installed - the Gallio and MbUnit dll's are in the project zip file, otherwise install latest release from www.gallio.org

OK - I have debugged some more, and now have a firm method to provoke the problem. It does not seem to be related to the ProcessCancelledException.

How to get all tests discovered (Present in Unit Test Explorer and runnable from Unit test explorer)

1. Make sure a file without a test class is open (Program.cs in the attached solution).
2. Make a change to the file (Enter a whitespace).
3. Compile. This triggers the ExploreAssembly method - all tests are detected and added.
4. Now select a test file (For example Test2.cs in the attached solution) - this triggers the ExploreFile method.
5. All tests (T2_0, T2_1 and T2_3) are correctly discovered by the Gallio indexer, and are passed over to the R# UnitTestElementConsumer (I see this in the debugger)
6. The Unit Test Explorer now looks as if Test2 is still present, but forcing a refresh removes the test from the explorer THIS IS NOT THE EXPECTED BEHAVIOR
7. Now make a change to a test in Test2 - ExploreFile is triggered, and the tests reappear in the test explorer
8. Try to run the test - it runs, but is removed from the explorer after complete.

From what I discovered during debugging, it looks like the problem occurs on a final ExploreFile call.

The important steps seem to be:

1. ExploreFile is called because the test file is modified -> Expected behavior
2. ExploreAssembly is called when test execution is requested because the file was dirty, triggering a build, triggering an ExploreAssembly
3. Test is executed - and sometimes I could see the GUI showing the test as pending
4. After test execution ExploreFile is called one more time (Do not understand why) and this seems to remove the test.

Any feedback to what might be happening would be greatly appreciated.

Hmm. It looks like there are a couple of things fundamentally wrong here.

Firstly, a new unit test element is created every time the file or assembly is parsed. ReSharper expects your IUnitTestElement types to implement IEquatable<IUnitTestElement>, which makes it look like the unit test element is a value type, i.e. two instances, with the same property values are treated the same. This isn't the case. ReSharper uses IEquatable for finding and caching, not for actual equality. You can see this because the interface has several mutable properties, namely Parent, Children and most especially, State. During editing, a test method might be deleted, or created, which causes a change to the Children collection in a test class element. Having multiple instances of the test class clearly causes problems. The same is true of the State value. Before parsing a file or assembly, ReSharper sets the State value of all affected elements to Pending. At the end, it removes (makes Invalid) any that are still at Pending (this saves the provider from having to track elements and tell ReSharper which elements have been removed).

In other words, you need to maintain one element per test class/method/etc. You can use the UnitTestElementManager.GetElementById to try and retrieve an existing element. If it returns a value, check it's the right one, update anything that's changed (e.g. the assembly location might change if you've switched between Debug and Release) and *VERY IMPORTANT* set the State to Valid. This tells ReSharper the element is valid and in use, and it doesn't get removed. If GetElementById returns null, ReSharper doesn't know about the element, so create a new one. This fixes a large portion of the issues with disappearing elements.

Secondly, the implementation of GetDeclaredElement is flawed. I hit an issue where I could rename a class, and all test methods in that class would be correctly marked as tests. Editing one of the tests would then cause all of the other tests to lose their test method status. I believe this is due to GallioTestElement.GetDeclaredElement. This method defers to IDeclaredElementResolver, which has an implementation for the PSI (i.e. the abstract syntax tree of the file) and one for the assembly metadata. This no longer works when maintaning a single instance of the element. It should always talk to the PSI (since the metadata is out of date as soon as the file is edited), but the current PSI implementation caches an instance of IDeclaredElement, which isn't guaranteed to still be valid - and indeed this is what's happening - and so ReSharper thinks the element isn't valid and removes it. As it happens, the metadata implementation of the interface is much closer to what's required all the time - it's taking the metadata information and looking it up in the PSI.

Finally, and perhaps a little less importantly, GallioTestElement.Parent needs to have some logic to ensure it's handled correctly. When an element is marked as invalid (such as renaming or deleting a test method), it is removed from the Parent test class, by setting the Parent property to null. At this point, the element should remove itself from its parent's Children collection (arguably, ReSharper could do this, too, but, well, it doesn't). This shouldn't affect any of the workings of the test runner of provider, but frequent edits can cause unnecessary memory usage.

I think this should fix the problems. Let me know how you get on, or if you hit any more issues.
Thanks
Matt

Thank you for the reply - I will try to find time to address these issues - the task seems to be quite a bit larger than I expected.

Is there any documentation/guidelines on how to correctly implement a unit test provider anywhere? I allready read the section in the SDK doc (http://confluence.jetbrains.net/display/ReSharper/4.09+Test+Framework+Support+%28R7%29) - it is quite shallow...
The source code for the NUnit test provider would probably be the best documentation - is that available.

Also - you added an excellent feature to R#6 for datadriven tests (http://blogs.jetbrains.com/dotnet/2011/08/new-features-in-resharper-6-unit-test-runner/) - how can we implement support for this in the Gallio runner?

Regards,

Espen

I have made a simple patch that improves the situation a lot, see attached.
I still have not addressed the issue with GetDeclaredElement, as I was uncertain as to how to proceed.

I did not get any response on my question regarding documentation - please provide us with some improved documentation, otherwise getting Gallio integration running will be hard.
The best is probably to get access to the NUnit test provider source code - can you provide this?

Hi. We don't ship the source to the nunit test provider, however, you can view the decompiled source to all of ReSharper in dotPeek. You can also download the source to the xUnit.net plugin, which can be a little easier to follow, since the xUnit API is simpler than nunit's. This also shows how to implement GetDeclaredElement by querying ReSharper, rather than caching. You can get this at http://xunitcontrib.codeplex.com

Implementing data driven tests is very similar to implementing normal test methods - you just create a new test element for each row. If you know how many elements you need ahead of time (due to the use of TestCase, for example), you can create all the elements you need when parsing the files or assembly metadata. Alternatively, you can just use a new element in the runner, and the test framework will try to create a dynamic test element by querying your provider for IDynamicUnitTestProvider and calling CreateDynamicElement (I'm away from my computer right now, so I might have got some of the names wrong). Again, the xunitcontrib source demonstrates how to do this.

And I agree with your comments on documentation. It doesn't provide enough depth to be able to implement a test provider without more information. It is on my list to update. In the meantime, please let me know any questions you have and I'll do my best to help.

Thanks
Matt