I was going to do some quick coding this morning on a project.  Haha...  2 hours later I still haven't started to write any code.  Why?  Visual Studio 2008 updates which I needed to be able to open the solution and be productive.

1. Installed the TFS client, which took a while.
2. Installed VS 2008 SP1.
3. I had to reinstall VS 2008 checking the "Unit Testing Tools" checkbox (see image).
4. Upgraded ReSharper 4.0 to 4.1
5. Installed the VS 2008 SP1 hotfix (KB958502) to add JScript intellisense support from "-vsdoc.js" files.

Number 3 was a bit strange.  I opened the solution only to find VS complaining about an unsupported project type.  I guess when I initially installed VS 2008 I figured I would never be using MSTest.  If only that were still true.

I installed KB958502 to get intellisense specifically for JQuery, since there is now a -vsdoc.js file available on the JQuery site.

You know, it seems like VS 2008 isn't all that far behind VS 2005 in regards to the number of installation steps/packages required to get a usable environment (at least for this project).

The other day I asked a coworker whether one our proprietary GUI based tools had a command line interface, to which he replied no.  I said that's cool, we don't really need one to run the tool headless from the command line anyway.  Command lines are antiquated.

Before the days of Windows command shells were the only way to run applications.  Command lines allow us to pipe input and output, batch programs, and otherwise automate repetitive tasks.  In *nix-land they are especially sharp tools.  However they are not without their own problems, foremost in my mind is argument processing.  Using a command line version of an application can be a pain, especially for any non-trivial app. 

If you look at any serious application that allows automation, they have their own automation framework.  Office allows you to write VBA, .NET, and also exposes a COM automation interface.  Maya has its own scripting language (Mel script, guess what Mel stand for) embedded in it along with the ability to run C++ plugins.  Now I bet you're thinking, yes big applications use automation frameworks and scripting languages, but I don't have that kind of time to invest in my little app.  Good, me neither.

In recent years Microsoft has provided less heavy handed approaches to consuming .NET assemblies. There are a couple different approaches that I can think of to use our .NET objects from the command line without writing a console application: Windows PowerShell and the DLR. 

PowerShell is more geared towads sysadmins, as such I like to use the DLR, more specifically IronPython.  Python is a very easy language to pickup, widespread (at least in the *nix world), and powerful.  I like to think of IronPython as C# without all the typing (OK, I stole that from Boo, but remember Boo syntax is based on Python).  Both have one important ability which we will need, the ability to use .NET assemblies and types from the command line (or script).

As an example I have a .NET library assembly that will run JSUnit tests, however I now need a way to run these unattended from the command line as part of a BVT.  Instead of writing a command line app for it I decide to write a quick little IronPython script which behaves very much like a console application.

import clr
import sys

clr.AddReferenceToFile("Sneal.JsUnitUtils.dll")
from Sneal.JsUnitUtils import *
from Sneal.JsUnitUtils.Browsers import *
from System.Collections.Generic import List
from System.IO import Path

# usage
if sys.argv.Count < 3:
	print "Usage: ipy IPJsUnit.py c:\source\mywebroot jsunittest1.html jsunittest2.html"
	sys.exit(False)

# command line params
webRootDirectory = sys.argv[1]

testFiles = List[str]()
for arg in sys.argv[2:]:
	testFiles.Add(Path.Combine(webRootDirectory, arg))

# create an execute test runner
mgr = JsUnitTestRunnerFactory()
runner = mgr.CreateRunner(testFiles, webRootDirectory, With.InternetExplorer)
success = runner.RunAllTests()

# print all results
if not success:
	print "Test run failed."
	for error in runner.Errors:
		print error.TestPage
		print error.FunctionName
		print error.Timing
		print error.Message
else:
	print "Yay!  All tests passed."

You'll notice that the application in fact takes two+ command line parameters, the webroot and the test files.  The advantage of this script over a console application from my point of view is:

1. Less code to write.
2. Documentation by example.
3. Starting script for an end user to modify to their needs.

Number 3 I think is particularly important.  Within a minute or two an end user can modify this script to their needs and desires.  They could hardcode the arguments if they wanted, or pass additional arguments without the headache of launching Visual Studio and recompiling a console application.  Almost anyone with a text editor could modify this and quickly get on with their life.

Here instead of a well meaning (but over-complicated) console application, its been replaced with just enough glue to bootstrap us into business.  Gluing components together is what scripting languages are good at, so lets take them into consideration instead of always reaching for another console app.

Generating test data for your application is not an uncommon task, but there are definitely many different ways to seed your application.  Probably one of the most common ways is to use SQL scripts to directly import data into the database, although while fast this lacks the control and validation that your middle tier code provides.

Loading data directly using SQL works best when your business logic resides in stored procedures near the database, but in a modern application where business logic is generally found in the middle tier application layer written in C#, this doesn't work quite as well.  Loading data directly could potentially cause data corruption since its perfectly plausible your data may very well be invalid.  Besides of the possible side effects of invalid data you're likely skipping some of the automatic management of data your middle tier affords you.  In other words, loading data using your middle tier API is probably more robust and easer to script.

To script the middle tier API to load data we could create data using C# or some other static language, but unfortunately that requires a compile and is much more rigid than a malleable SQL script.  Making the C# "script" data driven does help, but it lacks the flexibility that a native scripting language provides.

With a scripting a language the only part to manage is the script itself which gets checked into source control (or even a My Documents folder).  The other nice benefit is that we can have a nice GUI tool (test suite manager?) that dynamically picks up user scripts from one or more predefined script folders making them a available to run at the click of a button.  No recompile, or reloading, just edit-save-and-go.

So, given that we have a C# middle tier layer that's perfectly capable of persisting data into a database, how do we go about writing a script against it?  There are a few ways, but one of the easiest ways is to use IronPython.  If you're not familiar with IronPython:

IronPython is a new implementation of the Python programming language running on .NET. It supports an interactive console with fully dynamic compilation. It is well integrated with the rest of the .NET Framework and makes all .NET libraries easily available to Python programmers, while maintaining full compatibility with the Python language.

That's exactly what we need!  Something light weight and easy to mange like SQL, but interacts with our middle tier.  We could even embed the interpreter in our GUI runner if we so desired which gives us the full power of a real scripting language and .NET from our own tools.

I created a small spike that tests out how IronPython could be used to load data directly through the middle tier API.  First of all I created a Customer domain object, an Address component, and a CustomerRepository in C#.  From here we directly write Python against this.  Here's what my little spike project looked like:

For my purposes I've created a Scripts folder to hold my Python scripts (in a real environment these would probably be located elsewhere).  As you might have inferred, the 10Customers.py script creates 10 customers in my customer repository.  I could potentially add other scripts in this folder, and even chain them together to do other more substantial things, but for now this will do.  Now for the interesting part, the python script which loads ten semi-random customers.

import sys
import clr

sys.path.append(r"E:\Source\spikes\IronPythonSpike\IronPythonSpike\bin\debug")
clr.AddReferenceToFile("IronPythonSpike.dll")
from IronPythonSpike import *
import System.Random as Rand

rand = Rand()
names = ["Joe", "Bob", "John", "Smith", "Hank", "Aaron", "Neal", "Pat", "Tim", "Jones", "Bill"]
streets = ["128th St. W", "Main St.", "1st Ave.", "A.B.D. Rd.", "Lonely Lane", "Pacific Ave.", "6th Ave.", "Foobar Ct."]
states = ["WA", "OR", "CA", "NY", "AK", "NV", "AL", "TX", "FL"]

repository = InMemoryCustomerRepository()
	
for i in xrange(10):
	c = Customer()
	c.Id = i;
	c.FirstName = names[rand.Next(names.Count)]
	c.LastName = names[rand.Next(names.Count)]
	c.Address = Address()
	c.Address.Id = i
	c.Address.HouseNumber = rand.Next(32767);
	c.Address.Street = streets[rand.Next(streets.Count)]
	c.Address.State = states[rand.Next(states.Count)]
	c.Address.PostalCode = rand.Next(10000, 99999).ToString()
	
	repository.SaveOrUpdate(c)

The first thing we do is import sys which we use to add the path to our C# DLL to the available path's for the interpreter to search for our DLL in. 

sys.path.append(r"E:\Source\spikes\IronPythonSpike\IronPythonSpike\bin\debug") 

From here we can use the clr object to add a reference to our C# middle tier assembly.  This assembly of course provides our domain objects.  From there its just a matter of randomly generating the customer and address attributes from a predefined list of possibilities: names, streets, states.

Certainly you could create a separate generator script which would provide for much more varied data, you could even pull these bits from another database or even a web service.  You could even write the script so that the data was not random but always the same.  For my purposes though, this works just fine.  A more realistic domain would have more validation rules and would probably require a more intelligent script.

To verify that everything was added to the repository I can just make a quick call at the end of the script

for c in repository.GetAll():
	print c.ToString()
	print ""

Which on this run prints out:

Tim Tim
959 Lonely Lane
AL, 96094

John Pat
19577 Pacific Ave.
FL, 62699

Pat John
2194 Lonely Lane
OR, 58176

Smith Neal
6430 Pacific Ave.
AL, 88910

Jones John
11059 Pacific Ave.
TX, 67800

Smith Smith
24162 6th Ave.
CA, 81493

Neal Hank
4054 Pacific Ave.
WA, 33392

Smith Jones
32012 Pacific Ave.
FL, 44461

Tim Hank
28812 Foobar Ct.
AL, 98069

Neal Joe
1751 Main St.
NV, 52048

Not bad for a few lines of script!

I think the important part to remember is that we're using the middle tier to load data complete with validation and any other business logic that normally executes on save, all the while keeping things flexible and lightweight.

I had a situation where I needed to temporarily modify some read-only files on disk that were under source control (TFS).  Basically what I wanted to do was:

1. If the file is read-only, make it writable.
2. Modify the file.
3. Set it back to read-only if it was originally read-only.

You could just put an if then for steps 1 and 3, but that's ugly IMO.  Here's what the originally looked like before I refactored it:

bool isReadOnly = IsFileReadonly(webConfigPath);
FileAttributes originalFileAttributes = File.GetAttributes(webConfigPath);

if (isReadOnly)
{
	File.SetAttributes(file, FileAttributes.Normal);
}

File.Copy(backupWebConfigPath, webConfigPath, true);

if (isReadOnly)
{
	File.SetAttributes(webConfigPath, originalFileAttributes);
}

So here's my refactored solution using a DisposableAction which wraps a delegate that is invoked on dispose.  Inside the using statement the file is ensured to be writable, but at the end of the using statement its original attributes are set back on the file.  All the if statements in the code above get replaced with the following concise using statement:

using (FileAsWritable(webConfigPath))
{
	File.Copy(backupWebConfigPath, webConfigPath, true);
}

I think that's much cleaner, what do you think?  The rest of the supporting code if you're interested:

private static DisposableAction FileAsWritable(string file)
{
	if (!IsFileReadOnly(file))
	{
		return new DisposableAction(/* no-op */);
	}

	FileAttributes originalFileAttributes = File.GetAttributes(file);
	File.SetAttributes(file, FileAttributes.Normal);
	return new DisposableAction(() => File.SetAttributes(file, originalFileAttributes));
}

namespace Sneal.JsUnitUtils.Utils
{
    /// <summary>
    /// Wraps a delegate all that is called on Dispose, used for wrapping
    /// an action from a method call.
    /// </summary>
    public class DisposableAction : IDisposable
    {
        private Action action;

        public DisposableAction() {}

        public DisposableAction(Action action)
        {
            this.action = action;
        }

        public void Dispose()
        {
            if (action != null)
            {
                action();
            }
        }
    }

    public delegate void Action();
}

Before I left on vacation a couple of weeks ago I was working with another developer on a story that made use of some 3rd party services on a VM.  Before leaving I gave him my VM to use so we could avoid the unnecessary PITA of configuring  a new environment. 

Unfortunately now that I've come back I would like to run my VM again, but I can't since he is still running it and we can't share instances (isolated development environment). 

What I ended up doing was taking the VM image as I had it before I left and essentially cloned it.  By renaming the PC and changing the SIDs on the box I essentially have a cloned VM that will peacefully coexist on the same Windows network.

Here are the steps to take an existing VM joined to the domain and essentially clone it as a different machine.

1.       Unjoin the machine from the domain.  Reboot.

2.       Download NewSid utility from Microsoft (sysinternals).

3.       Run NewSid from the command line: newsid /a mynewpcname

4.       After reboot, login and rejoin the PC to the domain.

5.       Done

From then on the two VMs can coexist with one another on the same network and domain.

Test code is sometimes more difficult to maintain then production code for various reasons, but chief among these reasons is that developers just don't know how to organize their test code.  Poorly organized tests lead to poor performing tests, which leads to tests not being run as frequently as they should, which leads to more frequent and longer broken builds, which leads to fewer checkins, which leads to a loss of productivity, which leads to helpless little kittens dying in Africa*...

Test code should be organized along test types: unit, integration, functional, acceptance, and perhaps some others that tend to be more QA specific.  I often use unit and integration as the base primitives for my code bases.  Its not uncommon that I will have 2 test assemblies (unit and integration) for each assembly under test.  Essentially all my fast unit tests that I constantly run are in the unit test project, while all the slower tests (that hit the DB) go into the integration project.

Why the fine grained separation?  Why not just use one monolithic test project per solution? 

The reason we split test code up is the same reason we split up a regular application, for maintainability.  Well tested applications have an equal or greater number of lines of test code when compared to the actual application code.  If we had just one test project for an entire solution it just wouldn't be manageable for anything but the smallest of solutions.

By splitting the unit and integration (aka slow) tests into their own project it becomes much easier to run just the fast in memory unit tests all the time.  Categories are nice, and indeed useful for test segregation, however it still requires some filtering in the testrunner to only run the unit tests.  Splitting unit and integration tests into separate projects just makes things faster and reinforces the intent of each particular test area.

As the code base grows we will find that build times increase.  By having smaller test projects we may find we're not building as much code which helps us keep the TDD rhythm.  It also gives us the flexibility to move our core library to a separate solution if we wish, which may be beneficial if there is a lot of dependencies on that library project.  If we had one solution wide test project we wouldn't be able to do this.

Speed is King

Your unit tests need to be blazingly fast, anything else will lead to less effective tests.  Tests need to be run often to be relevant.  What you will find over time is that tests that hit external dependencies like databases or services are at least several times slower than a real unit test that has no external dependencies.  Several hundred tests that hit a database will absolutely kill your test speed.

The important distinction between integration and unit tests is speed, test speed should guide you where to put a particular test.  Slower tests go into the integration project, while the fast tests go into the unit test project.

The distinction between unit and integration tests isn't whether the SUT is completely isolated from its collaborators. Its OK to have unit tests that are using concrete collaborators, no where is there a rule requiring us to mock all collaborators.  Does the system really need a proliferation of interfaces that are all internal to the assembly itself?  I would argue in the general case no, you're probably wasting your time writing and maintaining too many interfaces.  Interfaces should be used as general extensibility points not only as mocking extension points.

The reason we call slow tests integration tests is that they usually have one thing in common, access to an external resource (like a database).  External calls outside the process are a magnitude slower than in process calls.  When choosing what type of test to write, its almost always preferable to choose to write a unit test.  Your unit tests are your first line of defense against defects.

When to run

With our tests organized and segregated between fast and slow, we can make further optimizations to our build pipeline.

Unit tests, specifically the fixture(s) that are relevant to what a developer is working on will get run over and over again while development takes place.  Before checkin, ideally several times a day, a developer will run all the unit tests.  This keeps the feedback on their progress immediate, which is only possible because we have fast unit tests.

Upon checkin our multiphase CI process takes the code, builds it, and runs all the unit tests.  This ensures a minimal amount of functionality about the software that should be good enough to prevent any blocking issues for other developers.  This build fails/passes the build and should always be kept green and fast.

The second phase of our build process then goes through and runs all of the slow integration tests as a secondary build.  Depending on the amount of hardware and number of tests this may happen on every build or at scheduled intervals; perhaps a nightly build process runs all the slow build targets, things like: integration tests, creating the installer, and API documentation.

Failing integration tests should ideally be rare, except for perhaps legacy systems that lack unit test coverage.  Broken integration tests almost always means that you are missing unit test coverage somewhere.  If possible your first step to fix a broken integration test is to write one or more failing unit tests and then make them green.  Once you have those unit tests in place, go back and run the integration tests before checking in.

Summary

With some organization and thought about where to place and how to write your tests you will be in a much better position a year down the road when your product's code base has really grown.  By keeping the unit tests fast we keep the build fast, which enables faster turnaround on checkins and features.  Do it for the kittens!

*I don't have any numbers to back up my claim about poor innocent kittens being harmed by poor performing unit tests, but it makes me take pause.  Can I save a kitten with this test?

Somehow I managed to find myself reading a very long rant on RoR and the RoR community by Zed, the creator of Mongrel.  He picks on lots of people, groups of people, technologies, and companies, but here's some of my favorite parts from his rant.

This is exactly what makes Rails a ghetto. A bunch of half-trained former PHP morons who never bother to sit down and really learn the computer science they were too good to study in college.

You hear that? The #1 money maker for 2008 years will be Rails cleanup. I’m not shitting you, it’s true and so just get over it and make the money.

If anyone had known Rails was that unstable they would have laughed in his face. Think about it further, this means that the creator of Rails in his flagship products could not keep them running for longer than 4 minutes on average. Repeat that to yourself. “He couldn’t keep his own servers running for longer than 4 minutes on average.” Assuming his statements are true (which we may never know) he basically duped us all.

What pisses me off is that I know they’re responsible (ThoughtWorks) for turning Ruby on Rails into the next Visual Basic.

After ThoughtWorks left the most recent project we revamped the team. We got rid of pair programming, cut down the number of tests, started cleaning more and more code out, got rid of their shitty tools, and we all started leaving at 6pm. What happens? We doubled our productivity with fewer people.

First it started on the fringe in start-ups and a few lonely places where it’s having mixed success (mostly due to the poor performance of the Ruby platform). Now it’s getting adopted internally at companies where of course it’ll get fucked up again and die off. After that it’ll move to the government sector where it will languish along with it’s new found buddy COBOL.

One of the topics brought up at the agile beer night was test fragility and how mocking frameworks can be a contributor to test smell.  Actually, I'm the one who brought up the point about mocking frameworks and fragility, but further examination revealed some disparity on the subject, specifically related to replay semantics.

I know that when I first started using a mocking framework I assumed it was best practice to use strict mocks.  I mean, why wouldn't you want to verify everything possible?  Apparently I'm not the only one to think this at one point or another.  I found this blog post by Derik Whittaker stating how he prefers strict mocks, it gets really interesting once you read through the comments.  Scott Bellware made the following comment:

Strict mocks is often a design smell.  It causes tests to express knowledge of an operation's internals, which is a violation of encapsulation of the system under test.

Any large code base that depends on strict mocking by default will suffer from unnecessary productivity loss due to breaking tests that have to be fixed where those tests suffer from inappropriate intimacy.

An API's internals are supposed to be allowed to change independently of its tests.  This is an essential quality of an agile codebase.

Strict mocking by default is a rather extreme stance.  Agile succeeds by finding those places where strictness can be slackened without causing any observable and categorical loss in integrity.  Opting for the strictest possible default runs contrary to this heuristic.

Through years of experience with mocking frameworks have I come to the realization that loose mocks should be your default.  Apparently the creator of Rhino Mocks, Ayende, agrees (emphasis is mine).

As an aside, I am deprecating CreateMock in favor of StrictMock. Using strict mocks by default was a bad design decision on my part.

One of the primary reasons mocks introduce fragility is that they require knowledge of the SUT that goes deeper than the public API.  You often need to know what methods on a collaborator will get called and setup some sensible return values that force your SUT through a particular branch of code. To me that's the very definition of tightly coupled.

So, how do we take advantage of a mocking framework, without making our tests fragile and resistive to change?  Unfortunately there's no way to completely avoid the mock object tax, but we can certainly minimize it.

One of the very first things we can do is setup our mocks in a shared fixture setup method. Sharing mock setup between tests is a great way to reduce the verbosity of each test.  A shared setup method will also setup the SUT with the appropriate mocks.

[SetUp]
public void SetUp()
{
    mocks = new MockRepository();
    userRepository = mocks.DynamicMock<IUserRepository>();
    purchaseService = mocks.DynamicMock<IPurchaseService>();
    sut = new Controller(userRepository, purchaseService);
}

You'll notice that the shared fixture setup I've created is using Rhino Mocks dynamic mocks.  You could instead use an auto mocking container to create the mocks for you to save some keystrokes and repetition.

With loose replay semantics (dynamic mocks in Rhino terminology) any method call during the replay state is accepted and if there is no special handling setup for this method a null or zero is returned. All the expected methods must be called if the object is to pass verification.

Too many times I've been burned by an over specified test using strict mocks.  With strict replay semantics only the methods that were explicitly recorded are accepted as valid. This means that any call that is not expected would cause an exception and fail the test. All the expected methods must be called if the object is to pass verification.  Depending on the mock this could be rather verbose leading to many expectations that have nothing to do with the actual test.

Tests should test only one thing.  You may have multiple assertions or expectations per test, but all of those assertions should have the same goal or theme.  I only want to setup the minimal amount of expectations on a mock that will verify a particular behavior in my SUT, anything more is over specified.  More to the point, keep your tests short and concise.

By using dynamic mocks rather than strict mocks, we are free to ignore the parts of the interaction between the SUT and the mock that we don't care about for our particular test.  We can also use setup result instead of expect if we can verify the interaction using a classicist approach.

[Test]
public void Should_show_cart_empty_message_if_cart_is_empty()
{
    SetupResult.For(purchaseService.GetCart()).Return(new ShoppingCart());
    mocks.ReplayAll();

    sut.ShowCart();

    Assert.AreEqual("You're shopping cart is empty", sut.Flash["warning"]);
}

Here I'm verifying behavior without explicit help of the mock framework, I'm only using the mocking framework to stub in my external dependencies and then verifying the expected behavior using a traditional assertion. 

If it weren't for the external service dependency I probably wouldn't be using a mocking framework here.  Mocking frameworks are most valuable at the edges of your components where things come together with the SUT, like databases, web services, and files.

The normal flow through the controller actually uses the purchase service for more than just grabbing the cart, it also uses it to grab the customer's billing information, but for this particular test I don't care whether the billing details get populated or not.  Here's the simple controller method we're testing:

public void ShowCart()
{
    PropertyBag["customer"] = userRepository.GetCustomerDetails(userID);
    PropertyBag["billing"] = purchaseService.GetBillingInfo(userID);

    ShoppingCart cart = purchaseService.GetCart();
    if (cart.IsEmpty)
    {
        Flash["warning"] = "You're shopping cart is empty";
    }
}

Lets say I now want to verify that the billing information gets populated when ShowCart is called.  Here the tables are turned, I don't really care about the call to GetCart() on the purchaseService.  I really only care that the property bag gets properly populated with billing information.

[Test]
public void Should_show_billing_info()
{
    SetupResult.For(purchaseService.GetBillingInfo(0))
        .IgnoreArguments()
        .Return(new BillingInfo());

    mocks.ReplayAll();

    sut.ShowCart();

    Assert.IsNotNull(sut.PropertyBag["billing"]);
    Assert.IsInstanceOfType(typeof(BillingInfo), sut.PropertyBag["billing"]);
}

Perhaps now we're concerned with populating the correct billing information, because as you might imagine, showing someone else's billing info would be a horrible security flaw.  So lets change our test to verify that using a mock.

[Test]
public void Should_show_billing_info()
{
    sut.CurrentUserID = 5;
    Expect.Call(purchaseService.GetBillingInfo(5))
        .Return(new BillingInfo());

    mocks.ReplayAll();

    sut.ShowCart();

    Assert.IsNotNull(sut.PropertyBag["billing"]);
    Assert.IsInstanceOfType(typeof(BillingInfo), sut.PropertyBag["billing"]);

    mocks.VerifyAll();
}

Just by changing from SetupResult.For to Expect.Call we are now explicitly checking that controller gets its billing information using the correct user ID.  We also added in a VerifyAll call.  Now if the purchaseService.GetBillingInfo method is called with anything other than 5, or not called, the test will fail.

By using dynamic mocks we've verified just enough to see if our SUT works without adding a bunch of noise or unneeded fragility to our tests.