From Java to Java EE

Half-dead project saved! Thank you - Direct Communication

2009-06-20T22:12:00.001+02:00

Picture (c) Przemyslaw Bielicki

In my previous post I shortly described the half-dead project in an over-waterfall company me and my team had to save three months before production. In this part I will show you how direct (instead of discrete) communication with customer or good Product Owner in general can help saving almost dead projects.

As I described earlier me and my team had a lot of problems with the systems we had to integrate - technical issues were present but they were not dominant, even taking into account the fact that the team was not experienced in .NET and T-SQL stuff.

Customer saved our project
After some time struggling with technical issues we finally were contacted by the person who knew how the product should behave. This person (let me call her Product Owner - PO) had all acceptance tests in her mind and knew exactly how to help us understand the product. PO was in fact our client - to be completely clear, the product we were delivering was either win-win or lose-lose so we had to cooperate and the success was our mutual goal.

PO was remotely available (although in the same time zone) but that was not a problem for us at all - we exchanged tons of emails and some phone calls and that was perfectly sufficient.

We were working like this for over two months and we delivered every single customer's request to the product. After the development phase our software had to pass the certification phase which, in fact, was couple-hour testing process in production. Our PO was present there and was helping other clients with testing (yes, there were other clients eventually using our product).

I could risk a claim that it was a huge success because we just found couple of minor issues that were not even concerning the software but insufficient or incomplete data in our production databases. Everybody is sure that what saved the project and what really caused it to be such success was direct contact and constant help from our PO. As I wrote in one of my previous posts, namely Customer Team Member - a way to winning together having customer as your team member is usually really good and efficient idea.

Fool's gold?
There are also (of course) problems with such attitude - the most "popular" one is the problem with actually having a customer. But even if you have a customer, how would you encourage him/her to be involved in the project (they usually aren't, especially at the beginning of the project)? It may depend on the actual contract i.e. how you define success or whether customer gain something by helping you - to know which contract best suits your current situation please refer to Peter Stevens' article: 10 Contracts for your next Agile Software Project.

It saved us - it can save You
To summarize, from my experience every time I worked with customer that was involved in the project I produced software with the highest business value (at that time). Every time I worked directly with involved customer I delivered the product that everybody treated as theirs, everybody was the owner - that's very cool feeling. This has also real business effect - satisfied customer is more likely to come back to your company. You gain credibility working together with your customers e.g. by showing them that you really work hard to meet their needs and to bring the best business value reducing costs. That usually pays off.

So, my final advice here for you would be to take some effort and time and look for a GOOD Product Owner. When you find her then start bombing her with emails, phone calls, whatever that can help your team deliver the product with highest possible business value.

What are your experiences with working directly with customers? Are they positive or negative? How would you improve thing in your project - could customer's involvement help?

Originally published on AgileSoftwareDevelopment.com

The Vendor Client Relationship (In Real World Situations)

2009-06-10T09:19:00.002+02:00

ROTFL

Copied from {codesqueeze}.

Sysinternals (Process Explorer) saved my a**

2009-05-20T08:00:00.001+02:00

After deploying our .NET system in production functionally speaking everything was OK. However, after couple of days we saw our processes consuming more than 80% of the CPU of some pretty powerful servers - and we were not computing any nuclear or medical killer stuff. Something was wrong - but what?

At this moment I recalled using free Sysinternals tool from Microsoft to see what's deep inside my operating system (in runtime, of course).

What really happened?
I started Process Explorer belonging to the Sysinternals suite, found my process and looked at it's runtime properties:

What I saw was the increasing number of TCP/IP connections (because used connections were not closed/returned to the pool properly). After some time spent on monitoring the process I noticed that this pattern is stable and each request caused the number of connections to database grow - until the process reached max and OS couldn't handle this process any more. Of course I fixed this issue wit one or two lines of code because it was a "little" problem (as usual) that appeared to have a big impact on the operating software.

I think wouldn't be able to discover this issue so quickly without using Sysinternals. Why? Because my colleagues were trying to investigate and fix this issue for some time before me. They didn't know Process Explorer tool :)

Direct communication as a half-dead project saver: When you start in an over-waterfall company

2009-05-18T09:00:00.000+02:00

Picture courtesy of clagnut@flickr

At the end of year 2008 our team was given a project that was critical from the $$ point of view. We had to deliver the project by the end of March 2009 in order to avoid huge fees from our customer. The problem was that the project we got was being developed by different team in different location and the quality of the existing solution was at least questionable. It turned out that the whole old team left the company and our small commando squad had to save the day. I just have to mention that the company we all worked for was a pure-waterfall monster.

You know, agile books are fun to read, but how do you start it if you happen to be a developer in a pure waterfall company on an impossible project? I went through it and going to show what worked well for us. I will tell how we were able to get an impossible project apply the agile principles and survive.

A bit of a history
The company I used to work for sometime didn't know/use Agile methods. Everything there was pure waterfall i.e. there was a marketing that defined what it wanted (previously negotiated the contract i.e. feature list with the customer), then it passed the specification docs to product definition team that was in charge of designing the solution, finding ALL possible problems and answering ALL possible questions. Then after couple of months the technical specifications were passed to the development team that... started everything all over again. I mean that the development team usually found documents received from product definition team useless - it's not my opinion, rather my conclusion after talks with many friend-developers. Some docs like protocol specifications or conversion specs were REALLY useful but they were vast minority.

Anyway, when developers started working on the project it usually took just few classes to find some blocking issues that had not been predicted/designed in the docs. "How it's possible?" "They were supposed to find ALL possible problems and answer ALL possible questions". Well, they were supposed to but it was IMPOSSIBLE to do so. Even perfectly prepared and thoroughly checked documentation will not replace direct communication channels.

The project I'm going to describe was quite different - there was no documentation, no specification but more so there was not a single person understanding the problem. This was a real shocker for the company used to work in an "ordered" and "predictable" way.

The Project
In short we had to develop an unknown communication protocol to connect the two unknown existing systems. And the time we were given was pretty much estimated as we already knew all the guts and gears in this system. Ah - I forgot about some pretty "unimportant" detail: the test database for the test system was located around 1000 km from our desks and the VPN connection was not yet established. So, for couple of weeks we were just having a pretty "black box" environment - it was not so bad though because we had some time to learn how to compile given sources (still, this was the time when we were supposed to be developing software).

What did we have?
We had the documentation for the protocol we had to implement with examples (requests and responses). The documentation was not very useful, though because it was designed for people who knew this protocol before. This documentation was not explaining many concepts - it was assuming the reader's knowledge on quite high level - unfortunately we were GREEN.
We also had some stored procedures (in our unreachable database) that were supposed to contain business logic and they were supposed to work already - we were told that they require just small changes. Basically we had to develop a part that had to receive the requests, parse and process it and invoke already existing stored procedure with transformed parameters – we knew that we would also modify or create completely new stored procedures from scratch.

When we eventually got to know what to do we knew that we had to implement hotel availability (3 different request/responses) and booking (couple of quite complex transaction flows) messages. The availability part was supposed to be 100% ready (this way we could focus only on booking) - we just had to test it and confirm (probably do some small fixes).

To summarize: we had some germ of the C# code, fully functional stored procedures in T-SQL (MS SQL server) that required some significant rework and proprietary ASCII format specification. I also have to mention that neither of the team members was C# or T-SQL expert (I put it as delicate as possible).

Finally @Work
After some time of investigation we knew how the frontend and backend work but we still couldn't test the system because we had no database. When it was ready we established a test environment and we finally was able to integrate the whole system. We were able to start this machinery and inject some requests. We were even receiving some responses - but at this moment we had completely no idea whether these responses were correct or not.

Here is the moment we started communicating directly with our customer who started testing delivered system in test environment and giving us feedback. If you ask why we didn’t communicate with them earlier I would answer that we had nothing to show them as the test environment was unreachable for developers. If we couldn’t show them anything we weren’t able to get any feedback. That sucked…

I will leave the details of our communication process with customer for the next post.

Lessons learned so far
We took over the project in a rush and were just given a source codes from the previous team – which wasn’t the first class anyway (I mean the code). There was no unit tests, no continuous integration environment, no single product backlog but we started creating those artefacts. And you know what – this was the best decision we could make - we took as much as we could from the agile world in this project.

The main problem that was probably the root cause of all other issues was lack of communication. Before starting contacting directly with customer (who became our product owner, in fact) this was the worst part of the project because we not only didn’t know what to do but also we had no one who could verify our assumptions. Lack of communication was very visible and the deadlines were getting closer. We were really thinking that we will be late with such pace and information flow (or rather lack of it). We knew that with such level of uncertainty agile practices - especially open and effective communication - could help us saving our jobs.

In the next part I will explain how we solved our communication problems, how agile practices helped keep the project on track and keep the number of bugs close to zero as well as how the whole project finished. Stay tuned!

Feedback
Do you recall such situation in your own project(s)? What was the solution that worked for your team? I'm really curious your opinions and thoughts.

Originally published on AgileSoftwareDevelopment.com

Niezły kawałek muzyki

2009-05-15T20:36:00.004+02:00

This post will be in Polish but don't worry it won't be about Java or technology at all. Anyway, if you want to listen to some really good music (one of the artist - Piotr Czerski - is my friend from university and some really wasting parties ;) visit this website: http://towary.art.pl.

Mam nadzieję, że Groszek się nie obrazi jeśli skopiuję post z jego bloga - nic lepszego i tak nie wymyślę na temat nowiuteńkiej płyty grupy "Towary Zastępcze" pt. "Dolne Miasto" bo o tym będzie ten wpis. Przesłuchałem tę płytę już dzisiaj dwa razy i jestem bardzo kontent.

A oto co do powiedzenia ma na temat płyty Groszek (ze wszystkim się w stu procentach zgadzam):

Na płytę składa się dwanaście zaśpiewanych wierszy, opowiadających strzępy tragicznej historii dziewczyny z Grand Hotelu. Szczególną uwagę zwraca strona internetowa towary.art.pl promująca album, na której możemy wcielić się w rolę detektywa prowadzącego śledztwo. Oglądając kolejne fragmenty video i podążając właściwym tropem, z pewnością uda nam się zdobyć odpowiedzi na trzy pytania, co pozwoli ściągnąć album w formacie mp3 za darmo z sieci. Wytrwali mogą również odnaleźć prawdziwego mordercę i dotrzeć do kolejnych bonusów.

Album dostępny jest również w bardziej namacalnej postaci. W atrakcyjnej cenie można nabyć akta sprawy, w których jednym z dowodów jest płyta. Doskonały pomysł na nietuzinkowy prezent i wsparcie twórczości artystów. Gorąco polecam!

Ja też polecam - kawał dobrej roboty i muzyki!

Java Concurrency Bugs/Programmer's Errors

2009-05-15T15:13:00.004+02:00

Alex Miller has been posting very interesting series on Java concurrency bugs (or pitfalls programmers could fall into). Especially interesting is the last post: Inconsistent synchronization. This post is most probably inspired by the problems Alex discovered in Hibernate statistics and I am pretty proud I was able to find possible concurrency issues reading the code. Anyway, I was not able to fully and correctly solve those issues.

The conclusion for me is that I thought about "magic" solution instead of the correct one i.e. I suggested that declaring a integer field as volatile would solve the issue. In fact it would but the "correct" solution to this problem is using AtomicInteger from java.util.concurrent.atomic package.

Another conclusion is to remember to keep bombarding your Java code with FindBugs - it can find many many Java issues in your code (not only related to concurrency).

The rest of the Alex's series (so far) is here:
Java Concurrency Bugs #4: ConcurrentModificationException
Java Concurrency Bugs #3 - atomic + atomic != atomic
Java Concurrency Bugs #2 - what to synchronize on
Java Concurrency Bugs #1 - mutable statics

Enjoy you Java Concurrency Issues ;)

Fun soft: regex checker

2009-05-04T11:00:00.002+02:00

I'm not a regular expressions expert but I use them often - as probably most of software developers. Before using them I want to be sure my regexes are written correctly and they will serve their purposes. I wrote very small command-line app for doing the check for me :) As a first input user provides the regex to check, then the user can type all texts to check against previously given regex until she types 'q' or 'exit' (for which program exits). Enjoy!


package com.bielu;

import java.io.Console;

public class RegexTester {
  public static void main(String[] args) {
    Console in = System.console();
    if (in == null) {
      System.err.println("Cannot obtain system console - exiting application");
      return;
    }

    System.out.print("Enter regex to test: ");
    String regex = in.readLine();
 
    while (true) {
      System.out.printf("Type value to match with given regex '%s': ", regex);
      String string = in.readLine();
  
      if (string.equalsIgnoreCase("q") || string.equalsIgnoreCase("exit")) {
        break;
      }
  
      if (string.matches(regex)) {
        System.out.printf("'%s' matches '%s' regex\n", string, regex);
      } else {
        System.out.printf("'%s' does NOT match '%s' regex\n", string, regex);
      }
    }
  }
}

Certified Scrum Master Training in Gdańsk

2009-04-27T21:09:00.008+02:00

I hope you already know that a cool company that is definitely one generation ahead than their competitors (at least as far as the way they work is concerned) called Spartez organizes the first CSM Training in northern Poland, namely in Gdańsk. The training will be led by famous Henrik Kniberg from whose book my adventure with real Scrum began. It is really a PITTY I will not be there :(

But anyway - if you live somewhere in the central, eastern or northern Europe this training is definitely worth taking. I took CSM course in Zurich last week led by Angela Druckman and am very satisfied. This training is really worth it's price (or even more because I think the training is quite cheap).

Don't waste your time and sign up for this CSM session.

The power of the final keyword - updated

2009-04-17T16:00:00.016+02:00

Picture courtesy of Okko Pyykkö@flickr

These two articles of Alex Miller:

A final bit of advice

JMM and final field freeze

clearly explain why using final fields in concurrent programs is extremely powerful and safe (thread-safe).

Take a look a this simple example:


public class SomeClass implements Runnable {  
    private final LinkedHashMap map;
      
    public SomeClass(LinkedHashMap copiedMap) {
        map = copiedMap;
        preProcessMap();
    }  
     
    private void preProcessMap() {
        //some non-trivial and not-so-short computations
    }

    // A read method  
    public Object get(Object key) {  
        return map.get(key);  
    }  
     
    public void run() {
        //assume that in this method only 
        //unsynchronized read operations on map may occur
        //any unsynchronized write operation is not thread-safe
    }
    // etc...  
  }

public class Main {
  public static void main(String[] args) {
      LinkedHashMap map = // map creation
      // map initialization
      new Thread(new SomeClass(map)).start();
      // etc.
  }
}

UPDATE:
I preserved the original version of the code but after some thoughts and thanks to some comments (thank you dear readers) I conclude that this example is not 100% correct. Before Thread.start() could be started the object has to be fully initialized and initialization will be completed only AFTER the SomeClass object is created. This way final keyword doesn't add any value there. On the other hand I meant this piece of code:


public class Main {
    public static void main(String[] args) {
      LinkedHashMap map = // map creation
      // map initialization
      final ExecutorService service = Executors.newFixedThreadPool(1);
      service.submit(new SomeClass(map));
      // etc.
    }
}

In such case I'm pretty sure you cannot assume that e.g. second processor (in a multi-core platform) will wait before executing newly scheduled task until the object of class SomeClass is fully initialized, right? Correct me if I'm wrong because I feel have some mess in my brain now ;)
End of UPDATE

Whatever happens in the preProcessMap() method you can be sure that LinkedHashMap will be copied and preprocessed BEFORE starting the run() method by the new thread. This way any READ (get) operation on this map in all threads will be safe and will operate on completely and correctly initialized map object.

If only you remove final keyword from LinkedHashMap declaration you cannot be sure our previous assumption. It may happen that the thread will be started BEFORE copying the map and if in run() method you process this map in any way you may get NPE.

Let me use a bottle metaphor :) For me final keyword is like the guarantee that you closed the bottle of whisky before passing it to your crazy friend who likes juggling them. If you ensure that the bottle is closed (final) before passing it to your friend you can be sure he will not spill any of the precious whisky on the floor. On the other hand if you start passing the bottle to him while trying to close it you're not sure whether it's fully closed while he starts juggling it or not. It may be closed 100% of time when you meet in your place but when you go to your other friend's party he may spill very expensive whisky and break the bottle in the same time (test and production environments metaphor this time).
Conclusion? Close the bottle before giving it to anybody else.

I will not be original and simply copy Alex Miller's advice:

Final fields are ever so important and useful. Whenever possible, strive to make your fields final. One tip for this is to enable a Save Action in Eclipse (or your IDE of choice) that automatically attempts to make fields final if possible so you can’t forget!

Why Object-Oriented languages rock?

2009-03-24T14:13:00.001+01:00

I loved coming back from using T-SQL to C# or Java like I loved the view from the picture (view from the l'Esterel's mountains towards Cannes). Of course, SQL-like languages are very powerful and when you do some complicated computations that need many requests to the underlying relational database, this choice may be the right one. At least, as far as the performance is concerned.

If we leave performance concerns behind we still have the language itself. So, what did I miss in this language? I was thinking of enumerating things that I didn't like but after some thinking I conclude that I just missed object-orientation in general.

But what it means? Let's see... I was not able to use any design pattern (e.g. Factory, Strategy or Decorator) and I needed them very badly! Instead of using design patterns I had to add many ugly ifs.

Another thing I missed was abstraction i.e. I wasn't able to create abstract stored procedure and then implement specialized subprocedures. In T-SQL everything you can do is to add some more ifs :) This language is strongly data-oriented which is maybe good but it's not sufficient IMHO.

Yes - it's very vague but it doesn't make sense to compare stored procedures with OO constructs. T-SQL is just yet another language I've learnt and used after many years of writing Java/C++ code and I just wanted to express my feelings.

I don't know what are perspectives for such SQL languages like T-SQL or PL/SQL but I think they should evolve in an object-oriented way. It would be much easier to write stored procedures/classes in e.g. Java, test them and then deploy on the server. This would be a win-win situation. We would gain performance by having stored code invoked directly on the SQL server while we will still be able to use OO constructs and profit from unit tests, design patterns, etc.

Maybe there already are such languages but I just don't know them (most of them are procedural languages) - if you know some let me know.

Everybody Needs ... Unit Tests

2009-03-02T10:16:00.001+01:00

Picture courtesy of Lotus72@flickr

... please remember people, no matter who you are, and whatever you do to live, thrive and survive, there are still some things that make us all the same: you, me, him, them - everybody, everybody! - this is a fragment of Blues Brothers' "Everybody Need Somebody" song. Nothing to do with software development but when you change the next line of this song from "Everybody needs somebody" to "Everybody needs Unit Tests" it becomes closer to software development.

I intentionally used lyrics of this song to keep everyone's attention (just a little bit) to this post. If you are familiar with TDD and unit test your code you can disregard it - I'd like to address this post to all software engineers/developers/programmers/etc. that DO NOT unit test their code.

In this post I will try (again) to convince all software developers that still don't unit test their code that it's right thing to do and it's damn simple.

Assume that Agile software development is wrong, really wrong

You don't have to be the fan of Agile movement - you can even think it's stupid, immature and doesn't help software development at all. Maybe you're right, maybe not - it's rather a question of taste. You don't have to use or like Scrum, you may think that XP is crazy and an overkill - you have right to think so. I can understand everything but PLEASE do unit test your code! Forget about Agile, forget about processes and methodologies - just unit test your code.

I just took over another project that is quite complex (of course it is!) but has NO SINGLE unit test (or any test at all). More so, there is no documentation of any kind. Yes I can read the code and see what it does - that's simple but how the hell I could know it does what it should do? How could I know it's not a mistake made by the developer who created this creature?

This code lacks Unit Test, but fortunately day after day I add more and more and I'm more confident this piece of software works.

Unit Testing once again

If you want to know something more about unit testing check out our previous articles. I will just recapitulate the most important, in my opinion, qualities of unit testing:

unit tests document internal and external architecture of the software system

unit tests help you and other developers to immediately see whether code "improvements" broke already existing code

unit tests help making your software bug-free - all issues should be addressed by writing a failing test first

unit tests together with code coverage tools improve quality of your software

unit tests can be used as a presentation platform - you can present hard to visualise issues using unit testing green bar - green means that it works

unit tests significantly improve team confidency in progress - if all the tests pass it means that all features implemented so far work and nobody broke the system

unit tests improve efficiency - you don't have to perform laborious manual tests again and again after code changes, just start unit tests and watch the red/green bar

already there but so important that worth reminding - unit tests work as a DOCUMENTATION (auxiliary or main one)

I'm open for your propositions

I hope I convinced you to start writing unit tests - if you don't like or believe in Agile movement it's OK. Unit Testing is just an obligatory practice of every decent software developer. If you do unit test it doesn't mean you're good one but if you don't unit test you're a poor developer (I mean it).

I wonder when unit testing will be taught obligatory at every decent university. It does so many good and is so simple - people, please - write unit tests!

If you have any ideas how to convince every developer that unit testing is right thing to do I'm open for suggestions.

Originally published on AgileSoftwareDevelopment.com

Synchronized collections in Java 5+

2009-02-16T08:00:00.001+01:00

Picture courtesy of Swamibu@flickr

You probably encountered problem of which collection to use while writing concurrent applications. Which implementation to use when you have N concurrent writers and M concurrent readers? I will try to shortly answer this question on simple example.

The code below presents usage of different collection implementations while having 16 concurrent writer threads and 100 concurrent reader threads. At the same time writer threads modify the collection i.e. add new elements and reader threads iterate through actual collection state and operate on its elements.


public class ConcurrentCollectionsTest {
 public static final int READERS_COUNT = 100;
 public static final int COLLECTION_SIZE = 5000;
 public static final int WRITERS_COUNT = 16;

 static volatile int readExceptionCount = 0;
  static CountDownLatch latch;

 public static void main(String[] args) throws Exception {
  Map<String, Collection<String>> map = 
    new LinkedHashMap<String, Collection<String>>();

  map.put("ArrayList", 
      new ArrayList<String>());

  map.put("Synchronized ArrayList", 
      Collections.synchronizedCollection(
        new ArrayList<String>()));

  map.put("ConcurrentLinkedQueue", 
      new ConcurrentLinkedQueue<String>());

  map.put("CopyOnWriteArrayList", 
      new CopyOnWriteArrayList<String>());

  for (Entry<String, Collection<String>> e: map.entrySet()) {
   Collection<String> collection = e.getValue();
   latch = new CountDownLatch(READERS_COUNT + WRITERS_COUNT);
   readExceptionCount = 0;

   long start = System.currentTimeMillis();
   for (int i = 0; i < WRITERS_COUNT; ++i) {
    Thread t = new Thread(new Writer(collection));
    t.start();
   }

   for (int i = 0; i < READERS_COUNT; ++i) {
    Thread t = new Thread(new Reader(collection));
    t.start();
   }

   latch.await();
   System.out.println("-----------------");
   System.out.println("Stats for [" + e.getKey() + "]:");
   System.out.printf("Collection's size: %d\n", collection.size());
   System.out.printf("ConcurrentModificationException count [read]: %d\n", readExceptionCount);
   System.out.printf("Duration: %d\n\n", System.currentTimeMillis() - start);
  }
 }

static class Reader implements Runnable {
 private final Collection<String> collection;

 Reader(Collection<String> col) {
  this.collection = col;
 }

 public void run() {
  try {
   for (Iterator<String> i = collection.iterator(); i.hasNext();) {
    // additional computing operation just to consume some time
    String t = i.next() + "_added";
    if (t.length() > 10 && i.hasNext()) {
     i.next();
    }
   }
  } catch (ConcurrentModificationException e) {
   readExceptionCount++;
  }
  latch.countDown();
 }
}

static class Writer implements Runnable {
 private final Collection<String> collection;

 Writer(Collection<String> col) {
  this.collection = col;
 }

 public void run() {
  try {
   for (int i = 0; i < COLLECTION_SIZE; ++i) {
    collection.add(Thread.currentThread().getName() 
        + System.currentTimeMillis());
   }
  } catch (Exception e) {
   // ignore
  }
  latch.countDown();
 }
}
}

Results on my machine (numbers can vary on different hardware but problems/conclusions will be the same):
----------------- Stats for [ArrayList]: Collection's size: 77038 ConcurrentModificationException count [read]: 17 Duration: 1047 ms ----------------- Stats for [Synchronized ArrayList]: Collection's size: 80000 ConcurrentModificationException count [read]: 4 Duration: 1141 ms ----------------- Stats for [ConcurrentLinkedQueue]: Collection's size: 80000 ConcurrentModificationException count [read]: 0 Duration: 1046 ms ----------------- Stats for [CopyOnWriteArrayList]: Collection's size: 80000 ConcurrentModificationException count [read]: 0 Duration: 14344 ms
Conclusions
ArrayList (unsurprisingly) doesn't work at all - there is 17 ConcurrentModificationException on read attempt and what is more unacceptable this collection is in unpredictable state i.e. it's size should be 80000 instead of 77038 - it means that many items were not stored by the writer threads.

Synchronized ArrayList works better i.e. its state is correct but still you need additional synchronization in order to get rid of ConcurrentModificationException while iterating through it.

ConcurrentLinkedQueue works best in this case and does not add any performance penalties.

CopyOnWriteArrayList is the worst in this case because traversal operations does not vastly outnumber mutations.

The last conclusion would be to perform such test before deciding which implementation works best in your particular case. Also, read javadoc carefully to see implications and suggested use cases.

Java synchronizers - Semaphore

2009-02-09T08:00:00.004+01:00

Picture courtesy of matthewblack@flickr

In this part I will present good-old semaphore that was finally introduced in Java 5. Semaphores are applicable in many different cases. One of them is the case when you have limited number of resources that can be accessed at the same time (imagine that you have 5 printers and 100 employees who print their documents - possibly all at the same time). Users have to wait for their turn - however order in which they requested the resource is not important (someone can wait couple of turns while someone else can get the resource many times turn after turn).

The code below shows this "metaphor" with 3 printers and 5 employees:


public class SemaphoreTest {
 public static void main(String[] args) throws InterruptedException {
  Semaphore sem = new Semaphore(3);
  Thread[] t = new Thread[5];
  
  for (int i = 0; i < t.length; i++) {
   t[i] = new Thread(new Task(sem));
   t[i].start();
  }
  
  Thread.sleep(10000);
  
  for (int i = 0; i < t.length; i++) {
   t[i].interrupt();
  }
 }

 private static final class Task implements Runnable {
  private final Semaphore sem;

  private Task(Semaphore sem) {
   this.sem = sem;
  }

  @Override
  public void run() {
   try {
    while (true) {
     if (sem.tryAcquire() == true) {
      System.out.printf("%s acquired semafore - %d\n", Thread.currentThread().getName(), sem.availablePermits());
      sem.release();
     } else {
      System.out.printf("%s could not acquire semafore\n", Thread.currentThread().getName());
     }
     Thread.sleep(1000);
    }
   } catch (InterruptedException e) {
    Thread.currentThread().interrupt();
   }
  }
 }
}

The program above terminates automatically after about 10 seconds from start.

Results (one of the infinite number of possible results) on my machine:
Thread-0 acquired semafore - 1 Thread-3 could not acquire semafore Thread-4 could not acquire semafore Thread-2 could not acquire semafore Thread-1 acquired semafore - 0 Thread-3 acquired semafore - 1 Thread-0 acquired semafore - 1 Thread-4 acquired semafore - 0 Thread-2 could not acquire semafore Thread-1 acquired semafore - 0 Thread-0 acquired semafore - 1 Thread-3 acquired semafore - 0 Thread-2 acquired semafore - 1 ...
As you can see each thread at some point acquires the semaphore and accesses requested resources. Be careful though because when the number of consumers/producers vastly outnumbers semaphore's permits starvation is very likely (i.e. some of the threads will never have an opportunity to enter the critical section). You can even spot it on the presented listing where Thread-2 cannot acquire semaphore twice while Thread-0 is "lucky" and always gets it.

Seven Principles of Lean Software Development

2009-02-05T08:00:00.000+01:00

Lean Software Development has its roots in Toyota Production System and it helps software organizations optimize their processes and production methods in order to deliver their products to the market much faster and with better quality. Lean movement can be considered as a new development method that tries to identify and eradicate all problems and "disabilities" of old methodologies like Waterfall. Lean puts main focus on people and communication - if people who produce the software are respected and they communicate efficiently, it is more likely that they will deliver good product and the final customer will be satisfied.

Lean Software Development subsequently gave birth to Agile Software Development methods and its main branches like Scrum or Crystal Clear. For many people who know the subject Agile is just another word for Lean or Lightweight.

In one of the most popular books on Lean subject, namely "Implementing Lean Software Development - from Concept to Cash", Mary and Tom Poppendieck explain how to implement Lean by following seven principles - principles that are some kind of Lean commandments:

Eliminate Waste
- Provide market and technical leadership - your company can be successful by producing innovative and technologically advanced products but you must understand what your customers value and you know what technology you're using can deliver
- Create nothing but value - you have to be careful with all the processes you follow i.e. be sure that all of them are required and they are focused on creating value
- Write less code - the more code you have the more tests you need thus it requires more work and if you're writing tests for features that are not needed you are simply wasting time

Create Knowledge
- Create design-build teams - leader of the development team has to listen to his/her members and ask smart questions encouraging them to look for the answers and to get back with encountered problems or invented solutions as soon as possible
- Maintain a culture of constant improvement - create environment in which people will be constantly improving what they are working on - they should know that they are not and should not be perfect - they always have a field to improve and they should do it
- Teach problem-solving methods - development team should behave like small research institute, they should establish hypotheses and conduct many rapid experiments in order to verify them

Build Quality In
- Synchronize - in order to achieve high quality in your software you should start worrying about it before you write single line of working code - don't wait with synchronization because it will hurt
- Automate - automate testing, building, installations, anything that is routine, but do it smartly, do it in a way people can improve the process and change anything they want without worrying that after the change is done the software will stop working
- Refactor - eliminate code duplication to ZERO - every time it shows up refactor the code, the tests, and the documentation to minimize the complexity

Defer Commitment
- Schedule Irreversible Decisions at the Last Responsible Moment - you should know where you want to go but you don't know the road very well, you will be discovering it day after day - the most important thing is to keep the right direction
- Break Dependencies - components should be coupled as loosely as possible to enable implementation in any order
- Maintain Options - develop multiple solutions for all critical decisions and see which one works best

Optimize the Whole
- Focus on the Entire Value Stream - focus on winning the whole race which is the software - don't optimize local inefficiencies, see the whole and optimize the whole organization
- Deliver a Complete Product - teams need to have great leaders as well as great engineers, sales, marketing specialists, secretaries, etc. - they together can deliver great final products to their customers

Deliver Fast
- Work in small batches - reduce projects size, shorten release cycles, stabilize work environment (listen to what your velocity tells you), repeat what's good and eradicate practices that creates obstacles
- Limit work to capacity - limit tasks queue to minimum (one or two iterations ahead is enough), don't be afraid of removing items from the queue - reject any work until you have an empty slot in your queue
- Focus on cycle time, not utilization - put in your queue small tasks that cannot clog the process for a long time - reduce cycle time and have fewer things to process in your queue

Respect People
- Train team leaders/supervisors - give team leaders the training, the guidance and some free space to implement lean thinking in their environment
- Move responsibility and decision making to the lowest possible level - let your people think and decide on their own - they know better how to implement difficult algorithms and apply state-of-the-art software frameworks
- Foster pride in workmanship - encourage passionate involvement of your team members to what and how they do

If this brief introduction to Lean Software Development is still not enough for you I strongly recommend buying and reading Poppendiecks' book "Implementing Lean Software Development - from Concept to Cash".

Deadlock while terminating JVM - how to avoid it

2009-02-02T08:00:00.006+01:00

Quite recently I was implementing some very small command line tool in Java. In order to clean and release all resources in case user presses Ctrl+C I added shutdown hooks. Unfortunately by mistake I added:


System.exit(0);

line within the shutdown hook and my application couldn't close. When I used jps and jstack I saw a deadlock on exit() method? WTF - I thought - I assumed that this method tries to forcibly close the JVM. You now know what was the problem?

If you read documentation carefully you should know :)

If this method is invoked after the virtual machine has begun its shutdown sequence then if shutdown hooks are being run this method will block indefinitely. If shutdown hooks have already been run and on-exit finalization has been enabled then this method halts the virtual machine with the given status code if the status is nonzero; otherwise, it blocks indefinitely.

This explains everything.

Conclusions - never invoke System.exit() from shutdown hooks and READ javadoc even if you're absolutely sure you know Java well...

Seven Principles of Lean Software Development - Eliminate Waste

2009-01-31T13:36:00.000+01:00

Picture courtesy of Phil Romans@flickr

Peter Stevens in his newest post advices how to deal with current crisis using Lean methodologies. One of his advice is to eliminate wastes, not costs. I totally agree with it, more so it is one of the most important (at least for me) principles of Lean Software Development.

Software development organizations should always strive to produce the best products and to deliver only features that are of paramount importance to their customers. They should always try to develop those 20% of functionalities that represent 80% of the value. This need is more vivid and desired during such global business conditions. This could apply to all types of enterprises - you should eliminate all unnecessary steps and waiting periods; you should strive to get the value as soon as possible and to get only pure value without any waste.

In this post I will try to explain "Eliminate Waste" principle from "Implementing Lean Software Development - from Concept to Cash" book.

Provide market and technical leadership
Your company can be successful by producing innovative and technologically advanced products. Important thing here is that you understand what your customers value and you know what technology you're using can deliver.

You don't necessarily have to invent something that is new and unknown. Note that the richest companies just replicate good ideas adding just few features that are unique and that satisfy customers (e.g. Google Mail, JIRA issue tracker).

You can be the market and technical leader by improving existing ideas and fitting them so that the final product will attract more customers.

Create nothing but value
You have to be careful with all the processes you follow. Be sure that all of them are required and they are focused on creating value.

If you are e.g. creating lots of documents that have been always produced but nobody really knows why and you are pretty sure nobody reads them - it sounds like waste you have to eliminate. Another example of waste is when you have to wait for a long time for some other department or team to answer your questions or uncertainties and this waiting period always stops you from moving forward. Waiting is the most apparent and obvious waste - though it is not always easy to eliminate.

You should measure your process cycle efficiency and strive to keep improving it - it will probably never be perfect but it can be constantly getting better and better.

Write less code
This advice is quite aggressive and when you work in old-fashioned waterfallish organization saying that you should limit the features in your system to only those that are absolutely necessary and deliver value can be perceived as some kind profanation. Well.... it's not.

The more code you have the more tests you need thus it requires more work and if you're writing tests for features that are not needed you are simply wasting time. If you don't have tests (is it possible?) it's even worse - there is bugs in a code that is not used and they will appear in the least predictable moment. I personally remove a lot of code when I take over some projects. I focus on the required features and remove all stuff that I "may need in the future". The rule is simple: if you need it, add it - if you don't need it right now, remove it. Another argument standing for writing less code is that usually with less code the overall complexity is lower and the code base is easier to understand thus maintain and support.

Last thing - the easiest way to identify unnecessary code is to use code coverage tool. More details can be found at the provided link.

Eliminate Waste
I hope pieces of advice given above will make it easy to understand how to put "Eliminate Waste" principle, which is the most important one IMHO, in practice. If you need more detailed description with more examples and more sophisticated explanation you should definitely go to "Implementing Lean Software Development - from Concept to Cash" book.

PS. Six remaining principles described earlier can be found here:

Respect People

Deliver Fast

Optimize the Whole

Defer Commitment

Build Quality In

Create Knowledge

Seven Principles of Lean Software Development - Create Knowledge

2009-01-07T18:48:00.002+01:00

Picture courtesy of J.C.Rojas@flickr

"There is no fool like an old fool" says popular international proverb. One of the meanings of this proverb is that people can make mistakes but they should learn from them. You are not a fool if you make mistake but you become one if you make the same mistake again. It simply means that you're not learning.

The same rules apply to the software development teams - it's much easier to work in an environment that encourages you to learn and to create knowledge. Why should you create knowledge? Well, the best way to learn something is to make mistakes by yourself but it wouldn't be wise to let all the engineers invent the wheel all over again. It's much better to teach them about what we already know, what works and what doesn't. It's much better and safer (yet a bit less effective) to learn from someone else's mistakes.

In this post I will try to explain "Create Knowledge" principle from "Implementing Lean Software Development - from Concept to Cash" book.

Create design-build teams
It is essential that the architecture of the software you're building is easily "splittable" into logical modules that can be implemented by cross-functional teams. These teams should be provided with appropriate leadership and be encouraged to engage themselves in the project and provide intensive feedback. My translation of this is that leader of the development team has to listen to his/her members and ask smart questions encouraging them to look for the answers and to get back with encountered problems or invented solutions as soon as possible. These teams should be encouraged by the leaders to share early and often, to fail fast, and to learn constantly.

It is pretty easy to imagine the contrary i.e. team members just waiting for others to fail noting every mistake and using it during 360 degree yearly evaluation. It must not happen! Team members have to teach each and help other. They should install some Wiki and put there whatever they think will be needed in the future. They should encourage each other to experiment and not to be afraid of failures - failures are great as long as you learn something from them. The last thing, though, you can learn from your failures is not to try again. Failures should be accepted by applaud - the team would know then which way they cannot go and invest in - it can save some money.

Maintain a culture of constant improvement
As a leader you must create such environment in which people will not be blaming each other for writing crappy code or designing crappy class or package structures. You must create environment in which people will be constantly improving what they are working on. People should also know that they are not and should not be perfect - they always have a field to improve and they should do it.

You as a leader must know that even if everything works fine there is always places that could be improved - encourage your team to identify such places and fix them. Such "place" can be a policy or practice, not necessarily source code, that makes development cumbersome or is basically not necessary and doesn't improve the overall results.

Teach problem-solving methods
Your team should not only learn and teach each other - they should be able to solve all kinds of problems in a structured way e.g. Plan Do Check Act. Development team should often behave like small research institute, they should establish hypotheses and conduct many rapid experiments in order to verify them. Your team should also produce concise and valuable documentation keeping up to date knowledge base of what works and what doesn't.

Create Knowledge
I hope pieces of advice given above will make it easy to understand how to put "Create Knowledge" principle in practice. If you need more detailed description with more examples and more sophisticated explanation you should definitely go to "Implementing Lean Software Development - from Concept to Cash" book.

PS. Five principles described earlier can be found here:

Respect People

Deliver Fast

Optimize the Whole

Defer Commitment

Build Quality In

Originally published on AgileSoftwareDevelopment.com

Year 2008 in a nutshell

2008-12-24T19:42:00.000+01:00

Picture courtesy of foxypar4@flickr

Christmas and New Year is coming so it's time for some summary. I've never done this before but this year was quite awesome and I have something to be proud and loud.

Let's start from January when I moved from my hometown to the glamorous French Riviera i.e Cote d'azur (yes, the sea here is pretty much azure).

I accomplished some cool stuff at work and one that I can share with you is the new generation hotel search engine - Wallaby (which is working but still in beta phase).

In 2008 I started interesting in the theory of Agile Software Development (after few years of using it) which resulted in inviting me to contributing to the AgileSoftwareDevelopment.com blog. It was a huge challenge for me as I've never been a good or even decent writer (especially in my native language - Polish) but it appeared to be kind of success. I have on my account some pretty cool posts that appeared to be big hits like these (each of the following posts was visited more than 5k times):

Can unit testing be a waste?

Video tutorial: Test Driven Development in practice

Developers Aren't Gonna Read It

"I know it doesn't work but it's done" - a story about the definition of done

And the best posts were written pretty spontaneously in less than thirty minutes without any preparation. Just some kind of a thunder in my brain that triggered my grey matter to work - that was just it.

I'd like to thank Artem for inviting me to become a regular contributor - it was a good decision for me to join ASD. I hope I will keep at least the same level of inventiveness next year :)

During 2008 I've read over 18 technical books and I shortly reviewed some of them on my private blog.
More on numbers in 2008: my FeedBurner account shows quite an increase (260% from 25 to 65 - and counting) of regular readers of my private blog - that's cool. Thank you readers!

Last but not least submitting my old and fresh posts to DZone gave my private blog a lot of visibility and doing this I increased number of visits by 1000% from ~930 to over 9K in just one month (I will not be able to keep these numbers but it anyway shows me that what I write is interesting and a lot of people like it). This is also thanks to ASD blog which gave me more visibility.

And now some conclusions. As I wrote earlier I moved to southern France in January 2008 and this was caused mostly by my previous employer - Intel - who simply laid off our team at the end of 2007. My friends who I worked with at Intel opened their own company and I was supposed to be their partner but I decided to move to France with my wife. And this was one of the best decisions we made in our life.

We live in a great place, have awesome friends and every week we do some cool stuff like skiing, hiking or just visiting some beautiful places like Monaco or Eze. And this is much more important that money that I could earn staying in Poland.

I learned a lot during year 2008, the Sun hasn't burnt out my brain (yet) and consider it pretty awesome year privately as well as professionally. And I hope next year will be even better - but even if it will be a bit worse it will still be awesome :) Yes - I'm the optimist. I'm not going to reveal my professional plans for the next year but I will probably share with you the results in 365 days.

As this is my last post this year I'd like to wish all the people around the World:

Merry Christmas
and
Happy New Year 2009

Seven Principles of Lean Software Development - Build Quality In

2008-12-17T14:00:00.001+01:00

Picture courtesy of WayTru@flickr

I was doing a quick research for this article using Google trying to find arguments standing for the claim that "quality is expensive". I was trying to find some resources saying that care for quality in the early stages of the software projects doesn't make sense and doesn't pay - I find it very difficult and I cannot share with you any reasonable links (maybe except for this one which refers to another post saying that "Quality Sucks").

What does it mean? It means that software people know that quality is very important. Why then quality of software products sucks in more cases than it rocks?

In many big corporations following waterfall model there is a special team named QA (Quality Assurance). This team's responsibility is (not surprisingly) to assure appropriate level of quality in the software product. This is fine, this is perfectly OK, except that QA is considered as a separate activity. This is the "Verification" or "Testing" box in the waterfall model diagram.
The biggest problem with this attitude is that QA team gets the product after it has been implemented and this is the root cause of all evil. QA is considered as something separate - we first do the product and then we care about the quality. This way of thinking is wrong. You should care about quality from the day one, before you write a single line of code.

Software teams should "build quality in" their products and QA should not be considered as a separate activity. Quality Assurance should be constant process of improving the product - QA activities and people should be involved in the development of the product during the development, not after when the developers are moved to another projects or even teams.

In this post I will try to explain "Build Quality In" principle from "Implementing Lean Software Development - from Concept to Cash" book. I will present few practices that will help your team build software with quality built in.

Synchronize
In order to achieve high quality in your software you should start worrying about it before you write single line of working code. It means that you should write test first and use all the frameworks that will facilitate your test suite (e.g. use mock objects). Track your code coverage - don't be too anal about it because 100% code coverage should not be the goal of your software but use it as an indicator to see which parts of you system should be tested more. And use all other tools that you feel are necessary to test your software thoroughly - unit testing is very often not enough.

Reduce partially done work - tasks that are 90% done usually takes another 90% of total time to finish - keep focused on one task and make it complete, then you can go to the next one. Try not putting defects on a list - stop and fix them the moment you discover them. Known bugs residing in your software will cause more defects in the future - don't allow this to happen (however issue tracking system can be sometimes useful e.g. for collecting requests from your customers).

Integrate your code as soon and as extensively as possible - commit your changes to CVS, SVN, etc. at least once a day and ensure all the tests give you green light. Don't wait with synchronization because it will hurt - you will spend more time on integration than on development. And you will get frustrated.

Automate
Even the best engineers make mistakes - you cannot avoid it - they are not robots (that may make mistakes too, btw.). Eliminate risk of mistakes by automating everything that is routine-work. Almost everything that is a repetitive work can be automated. And you should do this as soon and as early as possible - the best is to have continuous integration engine before committing a single line of code.

You should automate testing, building, installations, anything that is routine, but do it smartly, do it in a way people can improve the process and change anything they want without worrying that after the change is done the software will stop working. Automate in order to make people feel comfortable improving the software, tests, installation process, etc. by changing whatever they feel is necessary.

Refactor
Code in your software product should be as clean and as simple as possible. You can easily ensure it using static code analysers - they really work and can be a real pain in the ass for lousy developers (that's good because they will learn how to write clean code and follow the conventions).

Eliminate code duplication to ZERO - every time it shows up re factor the code, the tests, and the documentation to minimize the complexity. Using modern IDEs it's pretty simple and gives developers fun.

Build Quality In
I hope pieces of advice given above will make it easy to understand how to put "Build Quality In" principle in practice. If you need more detailed description with more examples and more sophisticated explanation you should definitely go to "Implementing Lean Software Development - from Concept to Cash" book.

PS. Four principles described earlier can be found here:

Respect People

Deliver Fast

Optimize the Whole

Defer Commitment

Originally published on AgileSoftwareDevelopment.com

Null Object - design pattern

2008-12-12T15:01:00.008+01:00

Joshua Bloch in his excellent book Effective Java (2nd Edition) gives advice that you should never return null collection/map/array from your code i.e. instead of such code:


public List<String> returnCollection() {
  //remainder omitted
  if (/*some condition*/) {
    return null;
  } else {
    // return collection
  }
}

you should always use this pattern:


public List<String> returnCollection() {
  //remainder omitted
  if (/*some condition*/) {
    return Collections.emptyList();
  } else {
    // return collection
  }
}

This basically prevents caller of your code to get NPE while trying to do things like this:


if (obj.returnCollection().size() > 0) {
// remainder omitted

Robert C. Martin in his book Agile Software Development, Principles, Patterns, and Practices gives another very similar pattern but related to ALL objects, not only to collections/maps/arrays. This design pattern is called Null Object.

Here is the example - let's assume that you have an application that check whether the user is authenticated:


public class User {
  private String username;
  private boolean authenticated;
  // remainder omitted

  public boolean isAuthenticated() {
    return authenticated;
  }
  // remainder omitted
}

and the code that returns the reference to the User object looks like this:


public User getUser() {
  if (/*some condition*/) {
    return user;
  } else {
    return null;
  }
}

This way the code that checks whether our user is authenticated should look like the following snippet:


if (obj.getUser() != null && obj.getUser().isAuthenticated() {
  // allow
}
// remainder omitted

Checking whether the object is null is not only a boilerplate code but it can also give you a lot of bugs e.g. if you forget to check whether the object is null.

And here the Null Object can help you:


public class NullUser extends User {
  
  public static final NullUser INSTANCE = new NullUser();

  public static NullUser getInstance() {
    return INSTANCE;
  }

  @Override
  public boolean isAuthenticated() {
    return false;
  }

  private NullUser() {
  }
}

plus:


public User getUser() {
  if (/*some condition*/) {
    return user;
  } else {
    return NullUser.getInstance();
  }
}

plus cleaner client code:


if (obj.getUser().isAuthenticated() {
  // allow
}
// remainder omitted

I find this pattern very useful and really helpful. With this pattern you can really save yourself a lot of NPEs.

Still the question is whether User should be a class or an interface and then whether NullUser should extend the base class or implement the user's interface. But I will leave this decision for your consideration.

What do you think about Null Object pattern?

PS. Example I presented is not necessarily applicable in the real systems - it is here just to depict design pattern's idea. Please don't treat provided code as a solution (I can think of many improvements/changes to it depending on the context by myself) - think about it at the pattern level - not the code level.

"I know it doesn't work but it's done" - a story about the definition of done

2008-12-11T08:00:00.001+01:00

Picture courtesy of orinrobertjohn@flickr

Some time ago I was talking to engineers responsible for some part of the software and I was asking when they will be ready for production. In other words I asked them when their features will be ready. They told me that they are ready now. What was my surprise when I tried to test their software and discovered that about 50% of cases were not working at all. So I asked them why they told me that they are "done" when they haven't even implemented half of the planned features? They answered me: "We know it doesn't work but it's done - we implemented something so it's done. Now we have to work on quality i.e. implement rest of the features."

When I heard this I think I might have looked like the lady from the picture. I couldn't believe someone can think like this - if we implement one use case out of one hundred we can consider the project done? The rest is the "quality"? I don't think so.

In this post I'll try to explain once again what is definition of done and why it's so important to have the same definition at least among all the people involved in the development of a single project.

Let's define "Done"

I would say that there is no one, good and universal definition of done. You can find some discussions in the Internet about it but you can see that everyone has it's own variation. So do I - in my view the most important things are (I will use "user story" word meaning every variation of request, use case, user story, etc.):

user story has to be implemented, today (no 99.9% is accepted)

user story has to be tested and no known bugs should exist

user story is ready to go into production, today

user story has to be ready to be presented to customer, today

Some explanation

User story has to be implemented - means that the code has to be committed to the version control system (like CVS, SVN, etc.), documentation should be available on Wiki or in the VCS, etc. It means that the output of the work done (whatever the work to be done was) must be available for anyone in the company to be downloaded in some way and checked. There must not be "I have it on my box - will publish it soon". Work must be committed and available for others.

User story has to be tested and no known bugs should exist - means that if you know about any bugs in the user story you're going to deliver - it's not done. If it exists in some subpart of the user story and you really need to deliver the working stuff, maybe you should split this user story into two, smaller ones. You must not deliver bugs to your customer - I'm talking about bugs you're aware of.

User story is ready to go into production - it means that it is ready to be deployed at any time from the time you stop talking. The wise thing would be if the working software is already deployed and tested in the production system - if it works - it's really done.

User story has to be ready to be presented to customer - it means that within 30 minutes (max) you are able to prepare presentation of working software to your customer. Of course, it requires you to have list of acceptance test ready and you know how to demo your software. The last point of it is very very important. Remember about it when defining all your user stories - you have to know HOW TO DEMO USER STORY - it will probably help you defining acceptance tests (e.g. user adds new item to the database using HTML form, then goes to the search panel and is able to find newly created item by its name, ....).

Wrap up

As I mentioned above good and universal definition of done probably does not exist but at least many resources agree on base principles. My definition of done is simple but I consider it quite powerful.

If you are interested in diving more into the subject I would recommend those two links from ScrumAlliance:

Are We There Yet?

Definition of Done: A Reference

What do you think about my definition of done? If you have your own I would gladly read about it. Please share your opinions here.

Originally published on AgileSoftwareDevelopment.com

Postcard from Auron

2008-12-10T14:00:00.003+01:00

Christmas time is coming and just have to decorate my blog with some snow :)

Enjoy!

Developers Aren't Gonna Read It

2008-12-04T08:00:00.000+01:00

Picture courtesy of austinevan@flickr

Developers are the customers - from time to time. They are the customers for product definition/specification team that is preparing technical specification documents. It doesn't really matter whether you work in an agile or non-agile environment - I'm sure you have some technical documentation and the main goal of it is to answer developers' questions on technical issues (e.g. how to configure some components to work with others, how to map fields from GUI forms to XML message, etc.) It also helps test or QA team to prepare acceptance tests and to verify whether what developers implemented is what was specified (I know it stinks a bit the waterfall but stay tuned - I will say something about agile documentation soon).

I would suggest you reading an article on TAGRI (They Aren't Gonna Read It) by Scott W. Ambler - it's really great. And in my post I'm going to give you real-life example from what I experienced regarding documentation. I will share with you my opinions of what kind of documentation sucks and what documents are really cool and useful (btw. my dream documentation is the one with which I'm able to find accurate information of my interest quickly and be able to put this information into my head in less than 10 minutes - the picture you see is a total contradiction of my dream - it's a waterfall process).

Developers won't read it

At the beginning of the project I met with the specification creators in order to discuss what we are going to deliver. They described me the project goals, business value and the requirements, of course. They also showed me two big books (sorry, specification docs) with around 300 pages in total. And the project was quite simple - it was basic CRU (Create, Read, Update) with one type of objects to be stored and queried. Believe me - the system was simple.

I was responsible for the web UI part of this system (100 pages) but had to understand also how the backend works (200 pages). Wow! That's a lot of, for a simple system, to read - imagine how much time you need to read this. And it's nothing comparing to how much time and effort it cost to produce it - and you still have no single line of code working.

So, did I read the documentation? I didn't.

I didn't have to because I preferred direct communication with the guys who know the system throughout. I didn't have to read the documentation because the guy who specified the GUI prepared screen mockups and I knew exactly how the pages should look like. And I didn't need 100 pages of documentation for it. I just needed couple of screen mockups - it was enough for me to deliver the software.

Tests and examples are the best documentation ever

And they are not only the best documentation - they also define a design of the system to great extent. When I was integrating UI forms with the XML to the backend I was still not referring to any documentation. I just asked guys responsible for specification and preparation of tests to give me example messages (requests and responses). I got what I needed and basing on the examples I was able to integrate UI with the backend - simple. Here I will give kudos for the specification that explained how to map UI form fields into XML message (XML schema was not self-explanatory in many places e.g. I could store the phone number in different XML tags - I had to know which one I should fill). I also used a piece of documentation that explained what should be the format of input data - I had to validate user's input somehow. But that was all I needed - I used 5% (roughly) of the overall documentation stack.

KISS (Keep It Stupid Simple)

In that simple project we had three different documents and I was finding discrepancies between them almost every day - yes, some of the fields, data types, etc. were specified in all three documents (often differently). The part developed by me was quite resistant to this because I was still keeping my one source of information. If I didn't know something I just asked specification team - I wasn't looking for the answer in the documentation because it would take more time and it appeared that I was asking questions that were mostly not covered by the documentation. Again, verbal communication was the best choice for me - I was able to fix some mistakes in the documentation on the fly - because I reported every technical problem I had to those guys - they were adjusting details according to technical constraints.

To summarize this point - keep it simple. Have one source of information and you will win. I do - every time I follow this rule.

Not all documentation sucks

That's true - some level of documentation is necessary, as I showed it before. Some mappings are sometimes required, list of required fields, error codes, etc. Many of this can and should be covered by unit tests and automated GUI tests - but it's pretty hard to record GUI tests not having it. Usually in waterfall-like process documentation is considered as a good - very important good. I don't understand why? It doesn't represent any business value for the customer (except the user's guides) but a lot of people value documentation more than the working software. I think this is the reason they fail so often.

In my opinion the best documentation is the one that is very easily changeable. For example Word documents stored in Lotus Notes folders is a bad idea. Developers cannot easily change or even comment the specification/documentation - they should. I often faced the problem that I saw something really really stupid in the docs and I wanted to change it but I wasn't able. Wiki is the best option here - it's easily searchable (through many projects) and changeable. And if you want to have a Word doc - your business - just export the page to the format of your choice.

Solution

My main recommendation is to communicate verbally with people who know your product as often as you need; have one source of information - you will not be surprised by different requirements for the same thing; start with unit testing your code and keep your UI tests up to date; automate as many acceptance tests as possible - consult the results with customers (or customer proxies).

I'm not going to copy-paste the whole article but I totally support the Solution part of TAGRI article - I strongly recommend you reading it. I have nothing more to add.

What is your experience with documentation? Do you read huge stacks of papers your team lead gives you? What is the value of the documentation? Please, share your opinions here.

Originally published on AgileSoftwareDevelopment.com

Video Tutorial on "Java Stack Traces with jps and jstack"

2008-11-27T08:00:00.001+01:00

In one of the previous tutorials named "Java concurrency is easy: Video Tutorial on >>Detecting and debugging deadlocks<<" I made a little mistake but thanks to "Anonymous" user who commented that post I'm publishing small appendix to it. This short extension shows how to use jps and jstack tools together. Using these tools you don't have to have an access to your program's console in order to see the stack traces.

Enjoy the video!

If you want more information about jps and jstack applications refer to their home pages:

http://java.sun.com/javase/6/docs/technotes/tools/share/jstack.html

http://java.sun.com/javase/6/docs/technotes/tools/share/jps.html

Customer Team Member - a way to winning together

2008-11-26T16:22:00.002+01:00

Used by permission.
Copyright by Paweł Niewiadomski

One of the core XP practices is having a Customer Team Member. It means that development teams have access to the newest information from the customer's side and they know about all changes in the requirements very quickly. Having on-site customer (or customer proxy for commercial products with lots of potential customers) ensures that requests change informally, the process becomes flexible, and saves the cost of formal overhead.

In this post I would like to take another look at this practice. I would like to share with you my thoughts why having a Customer Team Member can help the development team win together with their customer. And I would like to take a "social" view on this practice.

Customer knows better
Some developers say "It would be better if the software wasn't made for the customers - we wouldn't have any change requests then and we wouldn't have to reimplement once delivered features". Good point, however in this case we wouldn't earn any money because no customer = no money.

XP is very clear about customers - customer should be a team member. XP says that the development team is able to consult with them all unknowns just-in-time and they don't have to assume anything or wait for their response (sometimes quite long which can result in development blockage).

Customer as a team member is very helpful - as I wrote earlier, communication is much faster and efficient as it is mostly informal. If a developer, or tester or someone else from the development team has some questions he/she goes directly to the Customer and gets the first-hand information instantly. No waiting - no wasting - no assumptions - Customer knows better.

"Social" aspect
My colleague who I worked with for last couple of years used to tell his best experience with Customer Team Member from his previous job. The project's goal was to deliver some software for the civil aircraft cockpit and the main users of this software were airline pilots. Airline pilots were the Customers.

My colleague worked for the whole project closely with Customer according to the XP practices and always says that this project was the most successful in this company. And I have to mention that the company was quite big - actually it was one of the biggest airlines in the World.

When I ask him why it was such a big success one of his main points is that when the team worked together with pilots they felt responsible for the project. They felt responsible for project's successes and failures - they were involved in the development process thus it was their product. They couldn't blame the team - they were the team!
More so, they saw how hard developers worked and that they weren't spending their time on drinking coffee and talking to each other. Pilots really appreciated the work developers were doing - they saw it, they were co-builders of the product. Customer Team Member = involvement, ownership and trust.

Customer proxy works well too
Leaving my colleague's example I have my own. I'm now working on the web GUI application for the XML-based backend and my "customer proxy" is a product definition (PD) team (my customer should be marketing directly but I don't set the rules here). They are supposed to know everything about the project - I mean requirements. To be honest it should be transparent for us as a development team who they are - they act as the Customer for us and they are on-site.

I noticed that we are the first team that works so closely with PD team but I see huge advantages. Actually I see the same advantages I was told about with airline pilots. PD sees how we work, they are involved in the development process, they know about our problems and we can adjust some requirements to the technical constraints on the fly without the overkill process of changing the documentation first.
PD team really appreciates work we do and they feel responsible for the project - it's not like "We give you the specification and now it's your business to make it work". They are helping us in adjusting the specification - but this is because we communicate them all our problems and we involve them in our development process.

It really works!

Just do it!
I hope I expressed the main points in just few words - it is very very important aspect of Customer Team Member (IMO).

And what should you do when you have problems in having an on-site customer? Robert C. Martin in his book "Agile Software Development, Principles, Patterns, and Practices" answers this question - just have one.

In practice it can be more difficult than that but it will pay off - you will win this together. Try to convince your Customer to work with you - if you succeed you will not regret it (will you?).

Do you have any experiences with Customer Team Member? Do you find it useful or useless? How do you manage to work with your customers?
Please share your opinions here.

Originally published on AgileSoftwareDevelopment.com