07.01.09

Unit Test vs Integration Tests

Posted in Chefbook at 12:31 am by davy803

I’m finding that I prefer having a mix of integration tests to just having strictly unit tests.  Unit tests are great for helper method or things that’re used all over the place, but often times there are one-off methods that don’t perform directly affect the end result, they’re just one way of achieving a specific functionality, a means to an end and not the end itself.
One of the things that’s nice about integration tests is that they test the what, not the how.  You can write integration tests based off a functional spec.  If a user moves a file to another folder we just want to know that it’s no longer in the originating folder and that it is in the destination folder.  The integration test doesn’t care how this is achieved.  If you later decide to refactor the method, the integration should still be valid.
Unit tests do have the benefit of being able to track down a problem more easily to a particular method, but I feel often times that’s not really necessary, more of a convinience.  With good debugging tools you should be able to step through and find the problem.
One issue I found initially with using Visual C# Express instead of one of the paid versions of Visual Studio is the lack of plug-in support like TestDriven.NET that lets you run tests under debug.  A good workaround to that I discovered is just create a test console app project (I just put it in the solution, and when I’m trying to debug some broken tests, just set that as the startup project) and run the Setup method followed by the specific test method you want to debug.  It’s a little ghetto but I found i works really well, much better than doing old school debugging (i.e. sprinkling gratuitous amounts of Console.WriteLine’s everywhere. (Then forgetting to remove them later)
I still do a mix of unit tests and integration tests, but I don’t really make as much of a distinction between them as some people.  I break one of the rules of strictly unit testing, which is 1 assertion per test.  I prefer to just extend a test rather than create a whole new one that does mostly the same thing.  For instance I have a long string of undo and redo tests all in the same test.  I do some stuff, undo it, redo it, do some more, and undo it all and put in appropriate asserts in between them to make sure that it behaves correctly each step of the way.
I’ve also strayed a little from strictly test-driven development.  I sometimes write the method first and sometimes the test first, but always write tests and methods pretty close to each other.  If you write all the code first it becomes really hard to get the motivation to write 50 tests for all that code, and often you get sloppy about what gets tested.
A pattern I’ve fallen in to which seems to work pretty well for me is that if I’m writing new functionality I write the method first.  If I’m altering existing functionality I write the test first.  It’s more for convinience than anything else. I don’t think it results in better code than purely test first, but I don’t think it makes it too much worse either.  I do lose out on the ability to verify that my test is actually testing the right thing (i.e. it fails before the code is written to make it pass)  A green test could mean that the code under test works.  Or it could mean that it’s not actually testing the code you think it is.
One important thing I do is if I think of a tricky scenario where a a bug could be introduced (such as a strange input that I might not account for) I always write the test case for that first.
The tests (whether you choose to call them unit tests or not) have been invaluable though.  They caught a lot of bugs when I was adding some functionality (such as undo/redo support) as well as when refactoring to make sure I didn’t subtly change the behavior.

I’m finding that I prefer to have a mix of integration tests with my unit tests rather than just strictly unit test.  A lot of people try to differentiate the two, and maybe for really large projects it does matter, but for smaller projects I find the distinction to be rather moot.   Here’s some other discussion about the difference between unit tests and integration tests.

One of the things that’s nice about integration tests is that they test the what, not the how.  You can write integration tests based off a functional spec.  If a user moves a file to another folder we just want to know that it’s no longer in the originating folder and that it is in the destination folder.  The integration test doesn’t care how this is achieved.  If you later decide to refactor the method, the integration should still be valid.

Unit tests do have the benefit of being able to track down a problem more easily to a particular method, but I feel often times that’s not really necessary, more of a convinience.  With good debugging tools you should be able to step through and find the problem.

One issue I found initially with using Visual C# Express instead of one of the paid versions of Visual Studio is the lack of plug-in support like TestDriven.NET that lets you run tests under debug.  A good workaround to that I discovered is just create a test console app project (I just put it in the solution, and when I’m trying to debug some broken tests, just set that as the startup project) and run the Setup method followed by the specific test method you want to debug.  It’s a little ghetto but I found i works really well, much better than doing old school debugging (i.e. sprinkling gratuitous amounts of Console.WriteLine’s everywhere. (Then forgetting to remove them later)

I still do a mix of unit tests and integration tests, but I don’t really make as much of a distinction between them as some people.  I break one of the rules of strictly unit testing, which is 1 assertion per test.  I prefer to just extend a test rather than create a whole new one that does mostly the same thing.  For example this is how my undo/redo test looks like:

[Test]
public void ChangesToAnIngredientShouldBePartOfRecipeUndoStack()
{
    var ing1 = new Ingredient();
    recipe.AddIngredient(ing1);
    recipeVM = new RecipeViewModel();
    var ingVM2 = new IngredientViewModel();
    recipeVM.RecipeModel = recipe;
    recipeVM.AddIngredient(ingVM2);
    ingVM2.Name = "BoB";
    ing1.Name = "Phil";
    Assert.True(recipeVM.Undo()); //Undo BoB namechange
    Assert.AreEqual("Phil", ing1.Name); //Phil shouldn't be undone because it wasn't performed on VM
    Assert.AreEqual(string.Empty, ingVM2.Name);
    Assert.True(recipeVM.Undo()); //Undo Add Ingredient
    Assert.False(recipeVM.Ingredients.Contains(ingVM2));
    Assert.False(recipeVM.Undo()); //Undo stack should be empty
    Assert.True(recipeVM.Redo()); //Redo Add Ingredient
    Assert.Contains(ingVM2, recipeVM.Ingredients);

    Assert.True(recipeVM.Redo()); //Redo BoB namechange
    var ingVM3 = new IngredientViewModel();
    recipeVM.AddIngredient(ingVM3);
    ingVM3.Name = "Jean";
    ingVM3.Amount = 3;

    Assert.True(recipeVM.Undo()); //Undo amount Change 3
    Assert.AreEqual(0, ingVM3.Amount);
    Assert.AreEqual("Jean", ingVM3.Name);
    Assert.AreEqual("BoB", ingVM2.Name);
    Assert.Contains(ingVM3, recipeVM.Ingredients);

    Assert.True(recipeVM.Undo()); //Undo name Change 3
    Assert.AreEqual(string.Empty, ingVM3.Name);
    Assert.AreEqual("BoB", ingVM2.Name);
    Assert.Contains(ingVM3, recipeVM.Ingredients);

    Assert.True(recipeVM.Undo()); //Undo add ingredient
    Assert.AreEqual("BoB", ingVM2.Name);
    Assert.False(recipeVM.Ingredients.Contains(ingVM3));

    Assert.True(recipeVM.Undo()); //Undo name Change 2
    Assert.AreEqual(string.Empty, ingVM2.Name);
    Assert.Contains(ingVM2, recipeVM.Ingredients);

    Assert.True(recipeVM.Undo()); //Undo add ing2 2
    Assert.False(recipeVM.Ingredients.Contains(ingVM2));

    Assert.False(recipeVM.Undo()); //Undo stack should be empty
}

Okay, I’ll be the first to admit it looks pretty ugly but the test has a clear purpose which is to test a large series of undo and redo operations and verify that the data is correct each step of the way, and not only verifies that an undo or redo operation worked, but also that a string of such operations works.

I’ve also strayed a little from strictly test-driven development.  I sometimes write the method first and sometimes the test first, but always write tests and methods pretty close to each other.  If you write all the code first it becomes really hard to get the motivation to write 50 tests for all that code, and often you get sloppy about what gets tested.

A pattern I’ve fallen in to which seems to work pretty well for me is that if I’m writing new functionality I write the method first.  If I’m altering existing functionality I write the test first.  It’s more for convenience than anything else. I don’t think it results in better code than purely test first, but I don’t think it makes it too much worse either.  I do lose out on the ability to verify that my test is actually testing the right thing (i.e. it fails before the code is written to make it pass)  A green test could mean that the code under test works.  Or it could mean that it’s not actually testing the code you think it is.

One important thing I do is if I think of a tricky scenario where a a bug could be introduced (such as a strange input that I might not account for) I always write the test case for that first.

The tests (whether you choose to call them unit tests or not) have been invaluable though.  They caught a lot of bugs when I was adding some functionality (such as undo/redo support) as well as when refactoring to make sure I didn’t subtly change the behavior.

Leave a Comment