Graeme Hill on .NET development

A couple months ago I wrote this article explaining why I think it is reasonable for unit tests to hit a real database. Subsequently, I wrote a follow up article describing some techniques for rolling back your database to its original state after each test. In that article I found that just using simple transactions did not solve the problem because you need access to all database connections being used, and they all have to be rolled back. I have since found a way around this problem using distributed transactions.

With the Microsoft Distributed Transaction Coordinator (MSDTC) the activity over multiple connections can be lumped into a single transaction using the TransactionScope class. MSDTC needs to be running for this to work, but since this is just for unit tests it doesn’t need to be enabled on your production environment.

In order to use the TransactionScope class your project will need a reference to System.Transactions. Here’s a sample unit test using MSTest and Entity Framework where the database is altered with multiple connections within a transaction and then the changes are rolled back:

Imports System.Transactions
Imports System
Imports System.Text
Imports System.Collections.Generic
Imports Microsoft.VisualStudio.TestTools.UnitTesting
 
<TestClass()> _
Public Class UnitTestSample
 
    <TestMethod()> _
    Public Sub ProofOfConceptTest()
        Using New TransactionScope
            Dim conn1 As New DataTestEntities
            Dim conn2 As New DataTestEntities
 
            Dim row1 As New Users With {.userName = "user1", .password = "pass"}
            Dim row2 As New Users With {.userName = "user2", .password = "pass"}
 
            conn1.AddToUsers(row1)
            conn2.AddToUsers(row2)
 
            conn1.SaveChanges()
            conn2.SaveChanges()
 
            Dim conn3 As New DataTestEntities
            Assert.AreEqual(conn3.Users.Count, 6)
        End Using
    End Sub
 
End Class

Alternatively, if you want every test method inside a test class to be within its own TransactionScope without adding a Using block to every single test, you can use the initialization and cleanup methods like this:

Imports System.Transactions
Imports System
Imports System.Text
Imports System.Collections.Generic
Imports Microsoft.VisualStudio.TestTools.UnitTesting
 
<TestClass()> _
Public Class UnitTestSample
 
    Private _transaction As TransactionScope
 
    <TestInitialize()> _
    Public Sub Setup()
        _transaction = New TransactionScope
    End Sub
 
    <TestCleanup()> _
    Public Sub TearDown()
        _transaction.Dispose()
    End Sub
 
    <TestMethod()> _
    Public Sub ProofOfConceptTest()
        Dim conn1 As New DataTestEntities
        Dim conn2 As New DataTestEntities
 
        Dim row1 As New Users With {.userName = "user1", .password = "pass"}
        Dim row2 As New Users With {.userName = "user2", .password = "pass"}
 
        conn1.AddToUsers(row1)
        conn2.AddToUsers(row2)
 
        conn1.SaveChanges()
        conn2.SaveChanges()
 
        Dim conn3 As New DataTestEntities
        Assert.AreEqual(conn3.Users.Count, 6)
    End Sub
 
End Class

As long as the use of MSDTC is an option, I have found this method to be far better than any of those described in the last article. It guarantees that the state or your database is maintained and is extremely fast (at least on small amounts of data).

Update (Jan 23, 2010): I have made a new post explaining a method that I think is better than any of those described here.

In part one I talked about how there is no true way to unit test your data access code under the standard definition of a unit test. However, I think it is useful to consider your database and your data access layer as a single unit when it comes to automated testing (read part one if you’re wondering why). Everything is a trade off though so there are two important drawbacks to hitting a real database in your tests:

• If a test fails you don’t necessarily know right off the bat whether it was your .NET code that failed or something to do with the database.

  This can be a pain if you really like your unit tests to point out the exact issue when they fail, but I would say it is a very minor problem. Also, don’t you want to know when there’s a bug in your database anyways?

• It can be very slow to roll back the database to its original state after every test.

  This one is the real kicker. It is important that each test executes as fast as possible, but in order to prevent cross contamination between your tests, the database needs to be restored to its original state after each test. Even though it is very fast to access a local SQL Server instance, the process of rolling back can be slow.

I will go over three methods for rolling back that database, but I only recommend the last one:

• Put each test in a transaction

  Rolling back a SQL Server transaction is an appealing option. It is by far the fastest method and seems easy to implement. The problem is that a transaction belongs to a single session, so you can get yourself into deadlock if you have multiple connections. If you begin a transaction on connection A and alter a table with that connection, then try to read from that table with connection B, then you have deadlock. Using the default SQL Server isolation level (read committed) connection B will wait for connection A to end its transaction, but the transaction doesn’t end until the test is finished, which requires connection B to read the data.

  Basically, using this method puts a major restriction on the actual implementation making it pretty much useless. There are other problems too as your test needs access to the connection so that it can begin and end the transaction. In many cases you want the connection to be internal, so this leads to bad design.

• Rebuild the database after each test

  This certainly works, but it is really slow and nobody likes slow unit tests.

• Use SQL Server snapshots

  In SQL Server 2005 and later you can make a snapshot of a database, and then restore the database to that snapshot at any time. Restoring the snapshot is much faster than rebuilding the entire database because the snapshot file doesn’t store the entire database, it just stores data that has been changed so that it can quickly be reverted. There are a few caveats to this approach. One is that snapshots are not available in SQL Server Express, you will need one of the non-free versions. The other is that you cannot restore a snapshot while the database has active connections, so you need to make sure you kill them before attempting to restore the snapshot after each test.

  Using SQL, you can create a snapshot like this:

  CREATE DATABASE SnapshotName
  ON (NAME=DatabaseFileLogicalName, FILENAME='PATH_TO_NEW_FILE')
  AS SNAPSHOT OF DatabaseName

  And then you can later restore the database to that snapshot:

  RESTORE DATABASE DatabaseName
  FROM DATABASE_SNAPSHOT = 'SnapshotName'

  Restoring a snapshot usually still has a delay of half a second or so, but it’s better than the alternatives.

• Update (Jan 23, 2010): Distributed Transactions
  Assuming that the use of MSDTC is an option, this is likely the best choice. In is described in this article.

As almost anyone who tries to unit test a database application will quickly discover, databases present a huge problem for unit testing. Strictly speaking, if you are testing your C# or VB code and you actually hit a real database, then it isn’t really a unit test. It is actually an integration test. However, I have found that it doesn’t really matter what you call it, the end result is that your tests are much more useful if they actually hit a real database. You don’t have to worry about whether the test failed because you screwed up your mock object or if the actual application is buggy and you get better code coverage because even broken SQL will lead to a failed test.

There are several methods that can be used to prevent your unit tests from actually using a real SQL Server database, but they all have their problems:

• Using an in-memory provider like SQLite

  There is an Entity Framework provider for SQLite that allows you to interact with a database without using a network or even going to your file system. This could certainly increase the execution speed of your unit tests and makes it easy to prevent cross contamination in your tests, but they are still integration tests. The only difference is that you are now testing whether your code works on SQLite, rather than the DBMS that you will actually use in production. The problem is that all database systems have different behaviors and feature sets, so your tests are no longer valid if you use a different DBMS for testing. There is also currently no system in place to automatically generate the SQLite schema from your entity data model, so you will need to find your own way of doing that, or you have to manually maintain a separate SQLite schema. Gross. If you are going to use another provider, it needs to be specially designed to behave exactly like your production database (ie: a mock SQL Server provider) but to my knowledge no such providers exists (if I’m wrong, please let me know!).

• Mock the Entity Framework ObjectContext

  If all you want to do is read data, then this works well and is easy to implement. Unfortunately, in the vast majority of cases, we also need to write data and that’s where this method gets tricky. Your mock ObjectContext needs to be able to track changes and save them to an in-memory repository. And again, you have to make sure that it behaves exactly like your production database. Because this method often involves either a huge wrapper or major alterations to auto-generated code (which means you also need to make your own generator or you’ll lose maintainability) the mock object itself is extremely complicated, leaving a high likelihood that it will have errors. Since the mock is so complicated one could argue that you are again doing integration tests, not unit tests. But this time instead of testing your code and the database, you are testing your code and the mock ObjectContext. Just like the SQLite example, this is much worse because you are testing whether your code integrates with something you will not use in production. If you are going to do integration tests anyways, then you might as well integrate with the real thing. This method could lead to faster executing tests, but don’t forget that a local SQL Server instance is actually extremely fast and might be just as good.

• Encapsulate your data access layer and then mock it

  I see this response on message boards all the time. Whenever someone asks how they unit test their data access code someone will respond “You’re doing it wrong, put all of your data access code into a separate module that you can mock”. There are a couple problems with this. First of all, you still need to test the code in the data access layer. If you have a function in your DAL that executes a complicated LINQ to Entites query, then you want to test that query. Without using one of the techniques mentioned above, this requires hitting the database. Secondly, making your client code completely unaware of the data access layer’s implementation leads to some issues. Let’s pretend that my data access layer looks like this:

  Public Interface IUsersModel
   
      Function GetUsers() As IEnumerable(Of Users)
      Sub Save()
   
  End Interface
   
  Public Class UsersModel
      Implements IUsersModel
   
      Private _context As New DataTestEntities
   
      Public Function GetUsers() As IEnumerable(Of Users) Implements IUsersModel.GetUsers
          Return _context.Users
      End Function
   
      Public Sub Save() Implements IUsersModel.Save
          _context.SaveChanges()
      End Sub
   
  End Class

  It’s pretty simple, the code just allows you to get a collection of users and save any changes you make. UsersModel correctly implements the interface using the Entity Framework. Then we also have a controller that accesses the DAL. It looks like this:

  Public Class UsersController
   
      Private _usersModel As IUsersModel
   
      Public Sub New(ByVal usersModel As IUsersModel)
          _usersModel = usersModel
      End Sub
   
      Public Sub ChangeFirstUserNameToFoobar()
          _usersModel.GetUsers().First.userName = "foobar"
          _usersModel.Save()
      End Sub
   
  End Class

  UsersController has a dependency on IUsersModel, so when unit testing the ChangeFirstUserNameToFoobar method, we pass in a mock implementation of IUsersModel, but we cannot simply verify that Save() was called, we also need to know what is going to happen when Save is called. Specifically, we need some way of checking that the first user’s username was changed to “foobar”. This means that a mocking framework like RhinoMocks or Moq will not be sufficient. There must be a fake implementation of IUsersModel that keeps track of the changes that have been made. Now we are getting back into “mock the ObjectContext” territory because that’s basically what we will have done.

There is a definite trend here: each of the above methods is complicated enough that you lose the benefits of isolating your tests from the database. They are all integration tests. In every case you are testing your client code, plus the repository. Since you have to test a repository, it might as well be the real one. Of course, this presents its own challenges. You will want to use a local instance of SQL Server (or whatever DBMS you use) to keep the tests fast (and isolated from other developers) and you will need to roll back changes after each test. In subsequent articles I will look at how to deal with these issues.

Update: I have posted the second article: Unit testing an Enitity Framework DAL part 2: Rolling back the test database