Slow Tests = No Tests
Despite the conclusion of my previous post my conscience got the better of me (it usually does). Therefore I decided to implement automatic tests for our calculation engine. I could write a wrapper class to take care of most of the setup and if these calculations are wrong the whole system is a bust.
However almost immediately I ran into a problem. We use the Data Access Application Block from the Enterprise Library. One of the first things that our calculation engine does is execute a query to retrieve a pile of XML from the database. The first time that this code is executed from NUnit everything looks great (sub-second execution time). The second time however the test takes over40 seconds to complete. A fundamental TDD principle is that the faster your tests are the more frequently you will run them and the more frequently you will find problems. I need a lot of different tests here and if they will take hours to execute I'm not going to run them!
I created a simple test case and started to experiment. I tried implementing the XmlReader properly (at least as defined in the documentation). I tried using a DataSet instead of an XmlReader. I tried using SQL Server as the database in case this was an MSDE issue but it is not. I tried using a simple query that didn't even include the FOR XML AUTO clause. I tried kicking the wall and swearing. None of these made any difference.
I used a profiling tool to look at what's happening in the database. The query execution time is approximatley the same each time. However on the 2nd and subsequent executions the Audit Logout event for my SPID (and a companion shadow process) is over 40 seconds.
The problem does not occur in the web application itself. And if I run the NUnit tests using the 'Debugger' option of TestDriven.NET add-in (which of course is slower than standard execution) I do not encounter the issue. If I restart Visual Studio (or the TD.NET add-in from the system tray) the first execution is good and then the problem resurfaces. I experience the problem if running the tests from nunit-gui.exe but not from nunit-console.exe.
Currently I am stumped. I suspect that a resource is not being cleaned up properly but how to identify which one is beyond me. Experience has taught me to take a break and come back to this later. Hopefully I'll have a revelation!
However almost immediately I ran into a problem. We use the Data Access Application Block from the Enterprise Library. One of the first things that our calculation engine does is execute a query to retrieve a pile of XML from the database. The first time that this code is executed from NUnit everything looks great (sub-second execution time). The second time however the test takes over40 seconds to complete. A fundamental TDD principle is that the faster your tests are the more frequently you will run them and the more frequently you will find problems. I need a lot of different tests here and if they will take hours to execute I'm not going to run them!
I created a simple test case and started to experiment. I tried implementing the XmlReader properly (at least as defined in the documentation). I tried using a DataSet instead of an XmlReader. I tried using SQL Server as the database in case this was an MSDE issue but it is not. I tried using a simple query that didn't even include the FOR XML AUTO clause. I tried kicking the wall and swearing. None of these made any difference.
I used a profiling tool to look at what's happening in the database. The query execution time is approximatley the same each time. However on the 2nd and subsequent executions the Audit Logout event for my SPID (and a companion shadow process) is over 40 seconds.
The problem does not occur in the web application itself. And if I run the NUnit tests using the 'Debugger' option of TestDriven.NET add-in (which of course is slower than standard execution) I do not encounter the issue. If I restart Visual Studio (or the TD.NET add-in from the system tray) the first execution is good and then the problem resurfaces. I experience the problem if running the tests from nunit-gui.exe but not from nunit-console.exe.
Currently I am stumped. I suspect that a resource is not being cleaned up properly but how to identify which one is beyond me. Experience has taught me to take a break and come back to this later. Hopefully I'll have a revelation!
posted by JAGID at 9:56 AM
0 Comments:
Post a Comment
<< Home