Pols Consulting

I enjoyed reading Keith Braithwaite’s post on tests and gauges.  I like the way Keith uses the the metaphor of gauges in metalwork to describe Fit style tests:

 You don’t work anything out from a gauge.  What you do is apply it to see if the workpiece is within tolerance or not.  And then you trim off, or build a bit up, or bend it a bit more, or whatever, a re-apply the gauge.  And repeat.

I agree with Keith.  The fit tests I write are business facing and for the benefit of the business customer.

On a recent project our customer got so enthusiastic about our fit tests he got extremely upset when I implemented a story without a fit test.  He refused to let the system go live until we had the fit test in place.

The story in question was very technical and involved sending a particular xml message to an external system. We just could not work out what a fit test would look like for this type of requirement.  Placing the expected xml message, with all it’s gory detail,  in the fit test would not have been helpful as this is a technical artefact and of no interest to the business.  We could not work out what to do.  The customer was not around to discuss this so I just went ahead and implemented the story (very naughty!).

What the Customer wanted was to be sure that we were sending the correct product information in the xml message.  To resolve the issue, I suggested that we have a Fit test that shows how the product attributes get mapped onto the xml message using xpath; although I still thought this was too technical for a business user.

We gave the Customer a couple of links to explain what xpath was so he could explore if this was a good solution for him.   To my amazement, he was delighted with xpath (I now know who to turn to when I have an problem with xpath) and filled in the fit test.

The interesting bit for me (and why I’m posting it on the blog) is that as soon as he knew what the message looked like and how it was structured, he realised that it did not really support the business - we were sending information that was outside our scope of our work and should have been supplied by another system.  He was also skeptical about the speed the external team could add new products due to the complex nature of the xml.

Most agile people tell this story to think we have the “perfect customer”!

I though it was worth sharing some experience we are having speeding up our very slow build from 1 hour plus to around 10 minutes.

In the past we have spent a lot of time exploring why our tests are slow and have never really come up with a good solution. We’re continually extending the test suite, so any savings we make don’t last that long.

The unit tests are quick (~40 seconds), but it’s the acceptance tests that start up and run the application that are painfully slow.

We decided to group the acceptance tests into small logically grouped test suites that can run in 10 minutes or less. Then we added build agents (each running on a separate machine), so we can run as many of these small test suites in parallel as we can.

Every time the build gets too slow we break them down into smaller chunks and add another build agent. Hardware is cheap so it’s easy to sale this solution.

So far this is working really well.

I have just discovered an old post by Jim Shore (http://www.jamesshore.com/Blog/How-I-Use-Fit.html) about how he started out using FIT. He started out by switching from a unit test driven approach to a more story driven development approach.

I encountered a big problem with this approach, though: I stopped doing good TDD. “Storytest-driven development” blurred the line between Fit and unit tests. The “STDD” cycle largely replaced the TDD cycle and, as a result, I had far fewer unit tests than normal.

That was a problem because Fit examples are not unit tests. Fit examples are just that: examples of a business rule in action, from a business perspective. In TDD, I write tests for every line of code that can possibly break. In my Fit examples, I try to talk about every possible business case. That’s a bigger net than TDD, and without TDD, my code wasn’t well tested.

I had the exact same experience. I also found that the design suffered due to the lack of tests (it did not force me to refactor as often as I should) and it had a detrimental impact on my productivity. Story tests are larger in scope and take longer to run. Unit tests provide a much faster feedback loop when you break something.

I wish I had read this before I started using FIT.

Nat Pryce, Romilly Cocking and I have just run a session on Test Driven Development using the new (so new, it has only just been released) JMock 2.0 library at SPA2007.

We got a lot of positive feedback, so I thought I would share the slides and some example code that we developed for the session.

The best part of XPDay is talking to people about what they do. I managed to catch Joe Walnes and Dan North’s talk on Awesome Acceptance Testing.

For as long as I can remember, Chris Matts and Dan North have been saying A user story’s behaviour is simply its acceptance criteria. If the system fulfils all the acceptance criteria, it’s behaving correctly; otherwise, it isn’t.

What I had missed is their really nice acceptance criteria template.

Given some initial context,
When an event occurs,
then ensure some outcomes.

For example

Given Joe has a current account with a balance of £1 and an overdraft limit of £100,
When Joe withdraws £50.00,
then the balance will be £49.00.

This is so expressive and so wonderfully simple that anyone in the business can write them. Thanks to Joe for showing me this.

Organise the Fit Tests around your iteration’s stories

I like to take small baby steps when I write software. It makes life easy. I don’t have to merge very much code, I get regular closure on what I’m doing, I don’t have to carry too much around in my head, and I can discard mistakes without loosing too much work.

Automated acceptance tests, such as Fit, can throw a throw a spanner in the works. I like to explore the problem by writing the Fit test with a domain expert before I start writing any code. I treat the acceptance test(s) as a definition of success. Before I start work on a new feature I need to understand how I’ll know when the Customer/Domain Expert is happy with what I have done. I flesh out the answer in the fit test (I often have several Fit tests exploring different aspects of the story).

Great, but Fit tests are course grained in nature. I typically make at least 20 baby steps (source code commits) before a story is complete. I don’t want to commit the Fit test before I start writing code as the Fit test will fail and break the build. Not good. I don’t want to wait until the fit test is complete as that will loose all the advantages of developing in baby steps.

Some teams use different directories to distinguish between the Fit tests that have been completed and should pass, and those that are currently being worked on. The tests get moved around when they have been completed. I guess that solves the baby steps problem, but I’ve never liked moving files around to indicate completion and it’s still hard to see how the Fit tests relate to the originating Customer Stories.

William Jones has a much nicer solution with his Agilifier project.

There is not much documentation at the moment (well, it is opensource!), so I’ll give you a quick overview.

• You group the Fit tests around the stories they are associated with. All the tests associated with a story are contained in a directory. You place the original story in the story.txt file. You simply add your Fit test files to the directory.
• You use test suites to make the distinction between the previously completed tests (i.e. we have broken something if they don’t pass - and should fail the build!), the tests associated with the current iteration (i.e. these provide a progress bar of the current iteration) and those tests an individual (or pair) are currently working on. The suites are simple text files containing the tests to be included in the suite.
• Agilifier runs the test suites and glues the story and Fit tests together in a nice html report.

Download it and give it a try. You should be able to figure it out from it’s own acceptance tests!

Write the test so it reads like a requirement specification and not a test

I was trying to explain a FIT test to a new colleage. It soon became clear that it had been written as a test. It didn’t convey the intent of what was required. Nor did it communicate why it did what it did. There was a lot of tacit knowlege in my explanation of what was going on. It obvioulsy failed to communicate the requirements. It did, however, test that a particular feature of the system worked! So, what is the purpose of a FIT test?

Brian Marick makes the distinction between Business Facing tests and Technology Facing tests.

A business-facing test is one you could describe to a business expert in terms that would (or should) interest her. If you were talking on the phone and wanted to describe what questions the test answers, you would use words drawn from the business domain: “If you withdraw more money than you have in your account, does the system automatically extend you a loan for the difference?”

A technology-facing test is one you describe with words drawn from the domain of the programmers: “Different browsers implement Javascript differently, so we test whether our product works with the most important ones.” Or: “PersistentUser#delete should not complain if the user record doesn’t exist.”

The FIT tests should be business facing, they are a communication tool with the added benefit being executable.  FIT tests should explain the requirements so that people know what the system does.  The collection of FIT tests forms an executable requirement specification. They should not really be called tests at all!

Our FIT test was clearly written from a developer’s point of view. It was technology facing. Fit is probably the wrong tool for technology facing tests. I would much rather write these in a more developer friendly tool such as Java or ruby.

Alan Francis has posted a wonderful posting about Test First Development.

It’s very much in vogue at the moment to talk about TDD, and to mention, knowingly, and in passing that “of course, TDD is not about testing, it’s about X”. X is behaviour, design, specification and any number of other things.

I hate this trend too. He also manages to get the word bollocks in a blog posting!

I couldn’t agree more.

I’ve been running several two week “challenge” assignments recently and really enjoyed them.

This week I’m trying to prove to a company that they can automate lots of their 3rd party vendor acceptance tests. They’re spending vast sums of money on manual testing - any reduction in this would be a godsend.

The first application is a java web app. That should be easy. It turns out that it’s using lots of really strange javascript.

For example, the login form looks like this:

Error: Could not open strangeForm.html

The post action “security_check” simply returns an error saying you can’t post! Why would anyone subvert the standard web form process like that? They have some javascript called from an image link that submits the form. If you don’t do things in a standard way you can’t test your application using standard tools.

Httpunit could not parse the javascript. If I clicked the link it simply returned to the same page (due to the href=”#”). JWebUnit uses Httpunit under the hood, so was not really an option (although I did give it a spin). Selenium did not work as I could not change the deployed webapp (the selenium javascript needs to be served up from the same server).

Then I discovered the lovely JExplorer library from MIIK. It basically wraps the IE components so you can embed Internet Explorer in a Swing application.

It worked a treat! I can now automate all my acceptance tests using JExplorer.

I have always been frustrated at how slow ant is at running my tests.  If I run my tests in Intellij it takes around 4 minutes to run 1300 tests.  The same tests in ant take over 12 minutes.

This is not such a big deal when I’m developing as I run them all in Intellij.  The problem is that the Continuous Integration server runs ant.  If the automatic build takes too long following a commit, I will have started working on something else.

I was telling people this as a recent Agile Summer School that Duncan Pierce and I ran. Justin Forder did some digging around and found the magic forkmode attribute…

You normally fork a new JVM to run the tests since it isolates your tests from Ant’s environment (which places a lot of libraries on the classloader).  The reason it’s slow is that the default behavior is to fork a new JVM for each test case class.  That’s a lot of forking.

It turns out that Ant 1.6.2 introduced a new junit task attribute called “forkmode“. If you set it to “once“, Ant will fork a single Java VM for all your tests classes.

Fantastic, the build server is now running all the tests in 6 minutes (it’s not as fast as my development machine).

Thanks to Justin Forder for pointing this out to me.   See Stefan Bodewig’s Weblog for more info