Slow Tests = No Tests

Despite the conclusion of my previous post my conscience got the better of me (it usually does). Therefore I decided to implement automatic tests for our calculation engine. I could write a wrapper class to take care of most of the setup and if these calculations are wrong the whole system is a bust.

However almost immediately I ran into a problem. We use the Data Access Application Block from the Enterprise Library. One of the first things that our calculation engine does is execute a query to retrieve a pile of XML from the database. The first time that this code is executed from NUnit everything looks great (sub-second execution time). The second time however the test takes over40 seconds to complete. A fundamental TDD principle is that the faster your tests are the more frequently you will run them and the more frequently you will find problems. I need a lot of different tests here and if they will take hours to execute I'm not going to run them!

I created a simple test case and started to experiment. I tried implementing the XmlReader properly (at least as defined in the documentation). I tried using a DataSet instead of an XmlReader. I tried using SQL Server as the database in case this was an MSDE issue but it is not. I tried using a simple query that didn't even include the FOR XML AUTO clause. I tried kicking the wall and swearing. None of these made any difference.

I used a profiling tool to look at what's happening in the database. The query execution time is approximatley the same each time. However on the 2nd and subsequent executions the Audit Logout event for my SPID (and a companion shadow process) is over 40 seconds.

The problem does not occur in the web application itself. And if I run the NUnit tests using the 'Debugger' option of TestDriven.NET add-in (which of course is slower than standard execution) I do not encounter the issue. If I restart Visual Studio (or the TD.NET add-in from the system tray) the first execution is good and then the problem resurfaces. I experience the problem if running the tests from nunit-gui.exe but not from nunit-console.exe.

Currently I am stumped. I suspect that a resource is not being cleaned up properly but how to identify which one is beyond me. Experience has taught me to take a break and come back to this later. Hopefully I'll have a revelation!

Designing for testability

I've investigated a bewildering number of TDD tools and integrated some into an existing application that I inherited. It's been a great learning experience. One of the tools that I evaluated at was FitNesse. It looks fantastic and I wrote a bunch of test tests (great English there!) but ultimately I did not integrate it into the application. I must admit that I was conscious of making things excessively difficulty for the poor schmuck who will have to support the application when I've moved onto something new and shiny. But the deciding issue was that I was not able to find a piece of the application where I could get significant value from using FitNesse as opposed to standard unit tests.

That changed today when I started to look at some issues with our calculation engine. It is crying out for FitNesse tests. Our user community are much better placed to know what results to expect compared to the developers. But the difficulty of writing tests (of any type) in the calculation engine is phenomenal due to large number of setup steps and multiple layers that must be traversed in order to perform a calculation.

One of the most powerful lessons that I have learned is that TDD retrofitting is horribly painful if the application was not designed with TDD in mind. A lack of interfaces is the obvious crime and because no form of Dependency Injection is used (everything is hardwired together) it is impossible to test discrete pieces of functionality. Without a comprehensive suite of tests I don't want to risk a major refactoring exercise and that is going to leave FitNesse on the sidelines for this project which is a crying shame.