ReSharper test runner tries to load the wrong configuration Follow

I have a strange issue with ReSharper's unit test runner.

As long as I build my Release configuration, everything is fine. However, when I clean all the build outputs and set the solution to Debug configuration, when I try to run the tests ReSharper still tries to load assemblies from the bin\Release directory, and there is nothing there so it fails.

I've configured the test runner to automatically rebuild; it builds the Debug configuration, and then tries to load assemblies from the Release configuration, which fails. When the runner tries to load the test assembly, I get a message:

Unit Test Runner failed to run tests

no type: Could not load assembly from [path elided]\bin\Release\Assembly.Specifications.dll: File does not exist

Huh?

How can I make the test runner follow my solution's configuration?

(P.S. I'm using Machine.Specifications as my testing framework).