Agile Application Development: Romilly Cocking's Review

TiddlyWiki to MindMap converter

2008-04-28T09:50:00.012+01:00

Knowledge is my stock in trade, and I use TiddlyWiki and Mind Mapping a lot.

Like all Wikis, TiddlyWiki is a great tool for capturing ideas and facts in text and linking them to other relevant topics. It's not so good at showing you the big picture.

Mind Maps (tm) are a great way of presenting the big picture, but programs like FreeMind and MindManager (tm) only show the notes for a single branch at a time.

What if you think of each tool as giving a different view on the same data? Then you could get the best of both worlds.

There's already a hypergraph extension for TiddlyWiki which allows you to see a visual representation of wiki contents but I prefer a more conventional Mind Map that I can edit.

I'm currently experimenting with a simple Python script which converts a TiddlyWiki into a FreeMind Mind Map. The converter works well for small wikis and I'm currently researching how best to structure larger maps.

The converter isn't ready for prime time yet, and documentation is minimal, but it's evolving fast. If you're interested please drop me a note ( at romilly dot cocking at gmail dot com) and I'll send you a copy.

History meme

2008-04-25T16:26:00.003+01:00

Mom, Tim made me do it...

romilly@samba:~$ history | awk
'{a[$2]++}END{for(i in a){print a[i] " " i}}'
| sort -rn | head
106 ls
96 sudo
52 cd
22 nano
16 man
16 find
16 exit
13 chmod
12 cat
10 scp

and, on the Virtual server that hosts samba,

romilly@pc031:~$  history | awk
'{a[$2]++}END{for(i in a){print a[i] " " i}}'
 | sort -rn | head
111 ls
78 cd
76 sudo
32 exit
23 top
21 ping
18 df
13 du
11 scp
11 man

Layers considered harmful: design to interfaces

2008-04-15T11:52:00.003+01:00

Mark Dalgarno recently published an excellent summary of a session presented by Steve Love at ACCU.

The talk offers a critique of the traditional doctrine that architectures should be layered. Instead, Love proposes a style based on designing to interfaces. That's music to my ears: it's exactly the approach we take in our new TDD with jMock2 course.

Love's presentation is available as a pdf. The first half consists of visuals alone; the second half combines the same visuals with notes.

Testing Hibernate-based Persistent objects without Spring

2008-04-06T11:30:00.005+01:00

Nat Pryce just pointed me at an excellent article on Testing Persistent Objects Without Spring by James Richardson.

It shows both the integration test and the code under test.

Lots of good ideas/good practice.

Testing RIAs with WebDriver

2008-03-28T14:19:00.006Z

If you need to test Rich Internet Applications from Java, take a look at Webdriver.

Things I like about it:

It uses real browsers to run the tests, quirks and all
It's got a literate testing interface based on Hamcrest and inspired by LiFT.

Here's a very simple example.It's doing simple tests on a static html site:

public class HomePageTest extends HamcrestWebDriverTestCase {
   
   @Override
   protected WebDriver createDriver() {
      return new HtmlUnitDriver();
   }
   
   public void testHomePageHasTitleAndHeading()
         throws Exception {
      goTo("http://test.intranet");
      assertPresenceOf(title()
        .with(text(startsWith("Agile"))));
      assertPresenceOf(title()
        .with(text(containsString("Cocking and Co."))));
      assertPresenceOf(heading(1)
        .with(text(equalTo("Agile Application Development"))));
   }
}

This won't quite work out of the box, as there is no heading finder, but adding one is easy:

public class HeadingFinder extends HtmlTagFinder {
  private final int level;

  public HeadingFinder(int level) {
     this.level = level;
  }

  @Override
  protected String tagDescription() {
     return "heading level "+level;
  }

  @Override
  protected String tagName() {
     return "h"+level;
  }

  public static HeadingFinder heading(int level) {
     return new HeadingFinder(level);
  }
}

It's fairly new, but there are now pre-built binaries available.

Category Theory/Haskell learning group in London

2008-03-25T10:00:00.004Z

Peter Marks organised a BOF (Birds of a feather) session on Category Theory at Spa2008. To my surprise about twelve of us turned up. We discovered a shared intuition (bizarre though it seems) that Category Theory will somehow shape the next big thing in application development. Less surprisingly, we also discovered a common enthusiasm for Haskell.

We want to build on this. Most of us are based in London, or can reach it easily. We plan to meet regularly (about once a month) to explore these ideas and to help each other to learn.

I'll post more about this as soon as we have a date and location for our first meeting.

Spa2008 retrospectve

2008-03-25T09:56:00.005Z

I'm back from Spa2008, and my head is buzzing.

I felt this was the best Spa conference in recent years. As chairman I'm biased, but I'm not responsible for the quality. That's down to the programme chairs (Eoin Woods and Ivan Moore) and to Andy Moorley who organised things perfectly.

I went to (and really enjoyed):

Mirror, Mirror on the wall - why me? Chris Cooper-Bland and Portia Tung
Thou Shalt Integrate by the Sweat of Thy Brow... Robert James and Eoin Woods
The Need for Speed Paul Dyson and John Nolan
Awesome Acceptance Testing Dan North and Joe Walnes
Spanish Guitar Recital David Harvey
Is Software Practice Advancing? John Daniels and friends
A few things I learned in 50 years of programming L Peter Deutsch
Big Ball of Money, Big Ball of Mud: Economics and Legacy Code Michael Feathers
Closing the Knowing Doing Gap Allan Kelly and Lise Hvatum

Now it's time to start thinking about Spa2009!

Fewer, better people

2008-02-26T11:36:00.004Z

I've just read Martin Fowler's post about the Cheaper Talent Hypothesis. It's an issue that most of us have encountered.

If you're in IT, feel underpaid, and think that you're above average (as > 50% of us do), you'll have wondered why you aren't rewarded in keeping with your worth.

I suspect the main reason that good developers are undervalued is that most people think that software is easy to write. Since many IT projects fail, the implication is that we can't very good at what we do.

The conclusion is false because the premise is false. Writing good software is one of the hardest things that people do. I'd love to find some way to allow non-developers to find that out for themselves.

IBM used to run introductory programming courses which included a session called keep the alien alive. The instructor played a dumb alien who obeyed spoken instructions literally; the class had to tell him/her how to drink a glass of water. It was surprisingly hard to do, and the instructor usually got soaked. But it really brought home the fact that programming is hard.

How could we update this for the 21st century?

Jython lives!

2007-10-16T12:23:00.000+01:00

It's good to see that Jython is under active development again.

I use cPython a lot. I'd pretty much stopped using Jython because it does not currently support language features that I find essential - decimals, generators and list comprehensions, for example.

The project seemed to be dormant, but Jython 2.2 has just been released, and it looks like Jython 2.5 is on the way, with all the critical features that I need. Excellent news!

Download all pdfs in a web page

2007-09-24T18:32:00.002+01:00

A while ago I found an out-of-print book with a web-page that listed a series of downloadable pdfs.

Since there were 17 chapters and several other sections, I didn't want to right-click my way through the list so I wrote a quick Python script to do it for me.

This must be a Python idiom. I've had to re-invent it so often, I thought I'd blog it so that I -and you - will never have to create it from scratch again.

Here it is.

import urllib, re, sys
# download each pdf linked to in a web-page.
# assumes that the urls are all relative,
# which is usually the case
pdflink = re.compile(r'<a href="(.*\.pdf)">')
baseURL = sys.argv[1]
page = urllib.urlopen(baseURL+"contents.html")

for line in page.readlines():
  match = pdflink.search(line)
  if match:
      filename = match.group(1)
      print "downloading %s" % filename
      urllib.urlretrieve(baseURL+filename, filename)

Google soft-launches social bookmarking tool

2007-09-24T15:00:00.000+01:00

Is Google shared stuff the latest hot thing? It's a new social bookmarking tool.

It's available now, although you won't yet find it in the list of Google services and tools, nor on the Google labs page.

As well as sharing with your friends, the new bookmarklet allows you to share your links via several third-party sites, including Facebook, digg, reddit and del.icio.us.

In itself, it wouldn't be worldshaking; but

It's from Google
There are rumors of a soon-to-be-released Google api for social networking.

Count Blog subscriptions with this Python script

2007-09-22T07:36:00.000+01:00

If you blog, you're probably keen to know the size of your readership. Many of your subscribers will use an RSS aggregation service, and these days there are lots of feed aggregators. How do you count the subscribers?

This short Python script does just that.

If you can access the httpd logs of the server that hosts your blog, you'll find that many aggregators send you a subscriber count when they check for new blog entries.

A log entry for a typical aggregator request looks like this:

65.214.44.29 - - [21/Sep/2007:04:12:34 +0100]
   "GET /atom.xml HTTP/1.1" 200 0 "-"
   "Bloglines/3.1 (http://www.bloglines.com; 10 subscribers)"

The Python script below filters out lines that contain the string "subscribers", extracts the Ip address and subscriber count, and calculates the total number of subscriptions.

The script is very simple, but there are a couple of wrinkles.

A singe log file can include multiple requests from a given aggregator. The script puts counts into a dictionary keyed by the aggregator's ip address, so the total count includes the latest value for each aggregator.

In theory, you could get a request that included the string 'subscriber' from some other source. (For example, someone might visit a URL for newsletter subscribers.)
The script has a fudge to treat such requests as if they came from a dummy aggregator with no subscribers.

Here's the script:

# count blog subscribers from httpd log file 
import re

ip =r"^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"
anything = ".*"
count = r"\s(\d+)\s"
subscriber = "subscriber"
subscription_entry = re.compile(ip+anything+count+subscriber)

def ipc(line):
    match = subscription_entry.search(line)
    if match:
        return match.group(1,2)
    else:
        # just in case the line contains 'subscriber'
        # but does not match the regular expression
        null_entry = ("0.0.0.0", 0)
        return null_entry

def total_counts(ips_and_counts):
    # there may be multiple log entries for a given aggregator
    # so we use a dictionary and just keep the last value 
    #  which is the latest one for each aggregator
    dict = {}
    for (ip, count) in ips_and_counts:
        dict[ip] = int(count)
    return sum(dict.values())

def countSubscribersInLogFile(filename):
    file = open(filename)
    counts = [ipc(line) for line in file if subscriber in line]
    total = total_counts(counts)
    file.close()
    return total

if __name__ == "__main__":
    import os, sys
    filename = sys.argv[1]
    details = (countSubscribersInLogFile(filename), filename)
    print "%d subscriptions in %s" % details

Builders

2007-09-10T11:21:00.000+01:00

Nat Price has posted an excellent overview of Test Data Builders. (Don't be put off by the image!)

Sadly, he doesn't (yet) say much about the problem of unit testing builders, though he does point out that Builder classes may well find their way into production code. Nat, that's a hint...

XPDay7 is shaping up; now on to SPA2008!

2007-09-10T09:39:00.001+01:00

The programme committee for XPDay7 met last week to review session proposals. We'd received an outstanding collection of submissions, so there was no problem in filling the program with quality sessions. The hard part, as usual, was to decide which ones we would not include.

The meeting went very smoothly. Angela Martin and Ivan Moore are joint program chairs this year. They put a lot of hard work into preparation, and it really paid off.

If you've submitted a proposal for XPDay7, expect to hear from Angela or Ivan soon.

I'm now bending my mind to SPA2008. Ivan Moore and Eoin Woods are joint program chairs. As conference chair, I'm as keen as they are to see another outstanding programme.

If you submitted a proposal for XPDay7, consider submitting a session for SPA. We like a mix of new and proven sessions, so acceptance for XPay7 shouldn't stop you from submitting a similar proposal for SPA. Not should rejection - there were excellent submissions for which we just had no room, and they might well fit in the (longer) SPA programme.

The SPA deadline is Monday 17th September, but you don't need to submit finished material - just a proposal.

Backing up vital data with Amazon S3

2007-09-09T15:16:00.000+01:00

I finally got around to trying Amazon S3 for data backup.

Just in case you've missed out on S3, it's a commercial network-based data storage service offered by Amazon. They claim to use the same technology for S3 that lies behind the Amazon stores.

S3 comes with no service level guarantees and there have been some reports of occasional unavailability and/or slow transfer. I'm not too worried; the things that are critical to me are data security and cost.

I'm using the JetS3t java(tm) tool kit to manage the interaction with the S3 service. The cockpit application gives a GUI view of what's in your S3 buckets; the synchronize application gives a simple command line interface that allows you to keep data on S3 in step with data on your machine(s), and retrieve that data when necessary.

You need to be a little careful, as synchronization can delete data as well as add or update it. To mitigate this, there is a --noaction option which allows you to see what would happen to your data without actually changing anything.

Minor snags

I've hit a couple of minor snags so far; neither are caused by the S3 service itself, but you need to be aware of them.

The first arose when I tried to back up a large amount of data. I'm keen to keep a copy of my cvs repository off-site; it's well backed-up, but all the recent backups are in my home office. If we had a fire or a burglary I'd be in real trouble.

I realised soon after the upload started that it would take a while. The repository is about 2.7Gb and although I have fast broadband, my upstream speed is only 256 kb/s. The upload took 25 hours!

Smart synchronization

Synchronize is smart enough not to upload files if they are unchanged. I figured that the next upload would take a few minutes at most. The second snag arose when I tried to check that.

Part way through the synchronize application bombed with an out of memory error. I've now modified the script to give it 512M of heap space, and it runs just fine. And now a burglary would be inconvenient but not disastrous.

Measuring cultural differences

2007-09-07T13:20:00.000+01:00

James Manktelow has just published an interesting article about measuring and understanding cultural differences. It's based on Geert Hofstede's work on cultural dimensions.

Hofstede has identified five key dimensions which can be used to classify and understand local cultures. His analysis is based on a data base collected by IBM in the 60's and 70's surveying the values of employees in more than 70 countries.

Hofstede's Cultural Dimensions

Hofstede analyses culture in these dimensions:

Power Distance
Individualism
Masculinity
Uncertainty Avoidance and
Long-Term Orientation

It looks as if these measures would form a useful framework for analysing and understanding the culture of the organisations we work for.

Introducing Agile - Gerry Weinberg's approach

2007-09-07T10:57:00.000+01:00

David Petersen just sent me a link to a great interview: "5Qs on Agile with Gerald M. Weinberg".

The interview includes a neat way of getting people to want to adopt Agile techniques, rather than trying to force them.

Weinbeg is one of my favourite authors, but I wasn't subscribed to his blogs. I am now.

His writing blog has another interesting post about self-publishing.

Virtual Appliances: only run them if you trust them!

2007-08-31T14:21:00.000+01:00

I forgot to add an important warning to yesterday's post about Virtual Applicances!

When you start up a virtual appliance it will probably try to get an IP address on your private network via DHCP. If the Virtual Appliance contains malicious code, it could then do all kinds of nasty things.

The bottom line: as with any software that you install behind your firewall, don't run a virtual appliance unless you trust the source you got it from.

The Virtual Appliance: a simple way to install complex software

2007-08-30T16:17:00.000+01:00

This morning I saw an email asking for suggestions for a cheap or low-cost web content management system. There was one other important requirement - the software would be installed, managed and used by "semi-technical people".

A couple of people suggested drupal. I've heard good things about drupal, so I took a quick look at the website. Here's a sample of what hits you on the installation page:


Drupal requires a web server, PHP4 (4.3.3 or greater) or PHP5 (http://www.php.net/) and either MySQL (http://www.mysql.com/) or PostgreSQL (http://www.postgresql.org/).

That's hardly ideal for semi-technical people.

Once drupal is installed, however, it seems easy to configure and use. What my friend's users need is a simple way to install the application in the first place.

The best approach I can think of is to install a Virtual Appliance.

A Virtual Appliance is a virtual machine image which you can run using VMware, Xen or Parallels. It contains a complete application, pre-configured and ready to run.

Using a Virtual Appliance with the free VMware player takes just 5 steps:

Download VMware Player.
Install it. (on Windows, that's a one-click operation).
Download and unzip a suitable Virtual Appliance.
Start the Player and
Start the Appliance.

Now your chosen application is up and ready for use.

I tried out the drupal appliance from JumpBox. It took me about 5 minutes to install and start using it. (I had a bit of a head-start because I already had VMware player installed).

If you know someone who wants user-friendly software but who lacks the technical expertise to install it, look and see if you can find a virtual appliance to suit.

More and more software (open source and commercial) is being offered this way, and it's also a great way to distribute software that you've written or configured.

Automated end-to-end testing made easier with VMWare Server

2007-08-27T17:52:00.000+01:00

Automated testing is one of the most important practices in Agile Development. I like to have a hierarchy of automated tests that cover the whole system as deployed and all of its component parts, right down to unit tests that tell me whether each class is fulfilling its contract.

Starting with end-to-end tests

There's a lot published about unit testing and acceptance tests, but much less about end-to-end testing. That's disappointing, given that end-to-end tests are arguably the first thing you should work on when developing a new application.

The need for a clean deployment environment

End-to-end testing gives you a chance to check out your deployment process and verify that the deployed application works in the target environment. Sometimes, though, a badly planned testing process can lull you into a false sense of security; most of us have encountered applications that work well in their test environment but fail when deployed to a clean machine.

It usually takes too much time and effort to create a fresh testing environment from scratch every time you want to do an end-to-end test. There is a simple fast alternative using VMWare Server.

VMware server to the rescue

VMware server is a free (as in beer) version of VMware's virtualisation software. I run it on an Ubuntu server, and use it to create a clean environment for applications that I'm developing. It has a snapshot capability that allows me to capture a clean "before deployment" image, and restore that in a few seconds prior to running a test. Best of all, there's an external scripting interface which allows you to automate the process.

You can download VMware Server for free (after registration). I also use their commercial VMware workstation product; the two play well together, as you'd expect, but you can do everything you need for end-to-end testing with the free product.

JavaScript - are you serious?

2007-07-19T07:15:00.001+01:00

One of the sessions at Monday's miniSPA conference was Serious JavaScript led by Peter Marks and David Harvey.

I missed the original session at SPA2007, and I was very glad to get a chance to see it second time round. I found it fascinating and alarming.

It was fascinating because the presenters brought out some of JavaScript's real strengths as a development language. One example: JavaScript supports functions as first class objects, which allows smart developers to do all kinds of clever and useful things.

The session was alarming because it showed how seriously JavaScript is marred by arbitrary and counter-intuitive semantics.

Time after time the session leaders asked the audience to predict the results of simple JavaScript expressions. Time after time we predicted incorrectly (and sometimes the presenters did too).

The language seems to have been designed around the principle of most surprise.

People are suggesting that JavaScript is going to be the next big language, and it looks like a fair few people at Google are in that camp, including Steve Yegge. If so, heaven help us all. Power and unpredictability make a very dangerous combination.

miniSPA and jMock2

2007-07-18T20:40:00.000+01:00

I spent yesterday at miniSPA. It's a selection of popular presentations from last year's SPA conference; we run it to attract new participants and new session leaders.

This year's miniSPA was brilliantly organised by Ivan Moore and Andy Moorley. Ivan is programme chair for SPA2008. Andy runs the conference, organises miniSPA, and generally keeps the whole world turning as it should.

I was at miniSPA in two capacities; as next year's conference chair, and as a presenter from last year. Nat Pryce and I ran a tutorial on Test Driven Development with jMock2, which went really well. We've offered to run it again in November at XPDay7.

I'm about to analyse the feedback and first impressions are very positive. SPA is a remarkable conference. If you haven't been, take a look at the website and think about going next year. If you want to go next year, think about submitting a session proposal.

Unit testing builders

2007-07-11T07:05:00.000+01:00

Should one unit-test test Builder classes? If so, how?

A well-designed application is a composed network of simple components. Many developers use one or more builder classes to construct applications.

Builders can get quite complex.

In many cases we want a single component to be shared by several client components. It's easy to get this wrong. If we do so, we may build a network on which all dependencies appear to be satisfied, but we have actually created multiple copies of components when a single instance should be shared.

Builders often use lazy initialization, in which case they have state. This introduces additional opportunities for error.

Since it's easy to get builders wrong, we need to test them. End-to-end tests will usually tell us if we've build out application without all the necessary parts, but they don't pinpoint the problem, and they may not tell us if we've mistakenly introduced multiple objects instead of sharing a single one.

This suggests that we need to write unit tests for builders, but that's often difficult. The classes that are built will hide their implementation (as they should) so unit tests cannot easily verify that the right internal components are there inside them.

I'm coming to the conclusion that builder tests need to break encapsulation. That's normally a major code smell, but the nature of builders seems to leave no alternative.

Is measurement evil?

2006-12-02T09:22:00.000Z

A while back Dave Snowden of the Cognitive Edge (formerly Cynefin) posted a well-aimed comment about the UK Government's folly in trying to fix their outcome-based measures.

I first met Dave Snowden when he presented a keynote at XPDay5. I was bowled over by his talk, as were many of the audience.

Not so Tom Gilb, who was sitting next to me in the talk. Perhaps I shouldn't have been surprised: most of Tom's professional life has been focused on helping organisations to improve thorough quantification and measurement.

Both Dave and Tom are brilliant thinkers and I have been pondering ever since how two such clever people could hold such diametrically opposed views.

I had a delightful talk about this with Clarke Ching at XPDay6. which helped to clarify my ideas. I've tried to summarize my understanding below.

If I am trying to manage others, it's useless for me merely to set them measures. My team will game them. I'll get what I measure. It will not be what I want. (I think that's at the core of Dave Snowden's position. It's a sufficiently common view that Duncan Pierce and Jason Gorman ran a session about it and XPDay5.)
If I am a customer trying to define my requirements, it is sensible to quantify the qualities that I require of my supplier's product or service. (I think that's at the core of Tom Gilb's position).

In order for 2. to work, I must be able to:

quantify the qualities that I require
agree scales of measure for those qualities
agree realistic and achievable targets based on those scales of measure
monitor progress towards those targets (weekly if possible)
and negotiate changes to both the measures and the targets as appropriate.

This process relies on certain norms of behaviour by both parties. If either party seeks to cheat the other the process will break down. I think that this leads to the final precondition for Gilb's approach to work:

Either party must be able to terminate the relationship at any time without direct financial penalty.

Dave Snowden can produce plenty of examples where these conditions were not met and outcome-based measures were disastrous.

Tom Gilb can produce plenty of examples where these conditions were met and the results were excellent.

Conclusions:

both Dave Snowden and Tom Gilb have something valuable to tell us!
when we use measures to manage a supplier relationship, we must make sure that either party must be able to terminate the relationship at any time without direct financial penalty.

XPDay6 - better than ever...

2006-11-28T16:17:00.000Z

XPDay6 is nearly over, and it's been great.

One of my favorite sessions was Clarke Ching's Embracing Change: An Introduction to Agile Software Development. Clarke has given similar presentations at XPDay for several years. They've all been good and they keep getting better.

Another was a goldfish bowl session Facilitated by Nat Pryce and Jonathan Clarke asking Why Is Simple So Difficult?

It's hard to define simplicity; it's much easier to define or detect complexity (and then avoid it).

Lots of good ideas in the session; I liked Joe Walnes' comment that you shouldn't do the simplest thing - you should have the simplest thing (but it may be very difficult creating that simple thing).

I introduced Roy Sykes' 3am rule. This defines what I call simple code. It's the sort of code you'd hope to find if you have been called out of bed at 3am to fix a critical defect in production code.

My favorites session today was on LiFT - a framework for literate testing. It can be used as an alternative to FIT, but it can also be used to implement readable FIT fixtures. I'm hoping to give it a spin on my current project.

I really enjoyed the pub session afterwards. (Thanks, Google). Quote of the day - to Angela Martin, as she was taking photos of us all:

"I see you're taking a picture of me while I'm sober. I hope you have a fast film."

Ah well - off to the Old Bank Of England in Fleet Street for the post-conference Extreme Tuesday with more free drinks - thanks, Kizoom!