java.think()

A simple load test in Terracotta...

2009-10-01T08:45:00.001-07:00

This is a response to the following blog in which the author wrote a micro-benchmark and got some pretty bad results using Terracotta http://zion-city.blogspot.com/2009/10/terracotta-as-distributed-dbms-bad-idea.html.

Since the commenting system on blogger doesn't allow code, I am posting the response on my blog with code attached for reference.

So my approach was to try replicate the author's implementation, to see what kind of performance a straightforward micro-benchmark might achieve.

Reader beware - micro-benchmarks are never a good idea, and not usually indicative of real-world performance. In this case, based on real-world results I have seen, my results appear to be a lower bound for the kind of performance one should expect since the test isn't concurrent and is running on a single machine - hardly the kind of environment a real world clustered app would exist in)

So, with that said, I wrote a simple load test against a ConcurrentHashMap, and put 100,000 objects into it.

My results show:
Avg TPS: ~3,000
Instantaneous TPS as high as: ~7,000

Here's the code:

import java.util.Date;
import java.util.Map;
import java.util.concurrent.*;

public class Main
{
  static Map<Integer, Foo> map = 
    new ConcurrentHashMap<Integer, Foo>();

  public static class Foo
  {
    public String name;
    public String name2;
    public String name3;

    public Foo(String name)
    {
      this.name = name;
      this.name2 = name + " 2";
      this.name3 = name + " 3";
    }
  }

  public static void main(String[] args)
  {
    long start = System.currentTimeMillis();

    for (int i = 0; i < 100000; i++) {
      map.put(i, new Foo(new Date().toString()));
    }
    System.out.println("elapsed: " + (System.currentTimeMillis() - start));
  }
}

And here's the tc-config.xml:

<?xml version="1.0" encoding="UTF-8"?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-5.xsd">
  <application>
    <dso>
      <instrumented-classes>
        <include>
          <class-expression>Main$Foo</class-expression>
        </include>
      </instrumented-classes>
      <roots>
        <root>
          <field-name>Main.map</field-name>
        </root>
      </roots>
    </dso>
  </application>
</tc:tc-config>

I took a screenshot of the dev console running during the test, to give you an idea of the instantaneous TPS achieved:

Great customer service in the cloud

2009-09-02T19:38:00.000-07:00

It's interesting to see providers moving to and, by proxy, the rest of us relying on, the cloud.

I just spent a few hours at VMWorld, and judging by the size, sophistication, and variety of providers, vendors, products, and companies, virtualization technologies, and in particular, cloud computing, is here to stay.

Today, Google apologized for it's GMail outage yesterday, with a completely forthright, mature, and encouraging response.

Gmail's web interface had a widespread outage earlier today, lasting about 100 minutes. We know how many people rely on Gmail for personal and professional communications, and we take it very seriously when there's a problem with the service. Thus, right up front, I'd like to apologize to all of you — today's outage was a Big Deal, and we're treating it as such. <read the rest of Google's post...>

And, in a twist of fate, just a few days ago I received an email from Netflix. Apparently they had some trouble with their network while I was trying to watch a tv show using my XBox 360. Not only did they figure this out, but they sent me an email that offered to discount my monthly fee by 3%. This is fantastic customer service! Here's the email in it's entirety:

We're sorry you may have had trouble watching instantly via your Xbox

Dear Taylor,

Last night, you may have had trouble instantly watching movies or TV episodes via your Xbox due to technical issues.

We are sorry for the inconvenience this may have caused. If you were unable to instantly watch a movie or TV episode last night via your Xbox, click on this account specific link in the next 7 days to apply your 3% credit on your next billing statement. Credit can only be applied once.

Again, we apologize for any inconvenience, and thank you for your understanding. If you need further assistance, please call us at 1-866-923-0898.

-The Netflix Team

Failures do happen. And today's scaled-out architectures are designed to be resilient to these failures. But the fact is that even though these designs exist, and are generally very resilient against failures, giving these services availability times numbered in the 9's, mistakes in design, implementation, or execution still do happen.

I say, give these guys massive cred for owning up to their mistakes, and dealing with the consumer in an open and honest way. That's the way to build solid relationships, and I for one will not look twice when Yahoo! or Blockbuster sends me that next request to join up on their service. These guys have it figured out, and have sold me as lifelong* customer, even if they're not perfect.

If only legacy infrastructure (power, cable (grrr Comcast), telephone (AT&T - I'm looking at you)) and the like could understand the value in this approach.

* lifelong in tech years, which is only about 5 years or so ;-)

How To Optimize Performance (or how to do Performance Testing right)

2009-06-17T01:37:00.001-07:00

Optimizing performance requires you to performance test.

I'm just going to say it - performance testing is hard. Really hard.

Ask anyone that's done it before, and they will agree. If you haven't done it before, well, yeah, sorry. It's not easy. But you've got to do it anyway - because the most important thing you will do as a software engineer is performance test. It's a bit like when your Dad told you "when you grow up to be my age <insert age old wisdom here>" and you didn't believe him?

And now you're old enough that you realize, hey, the guy might have had a point?

Yeah, trust me. Performance testing is both the hardest, and most important, thing you will ever do in your software engineering career. Get it right - you'll be a rockstar. Don't do it - well, I promise you, you'll always be griping about why the amazing software you write is never actually used for "production" apps.

So here you go, simple steps to performance engineering:

1) Set goals - what are you trying to accomplish
2) Measure a baseline
3) Identify a bottleneck
4) Fix said bottleneck
5) Repeat until you meet your performance goals

Did I miss anything? Ahhh...yes. TAKE NOTES.

Let's try again:

1) Set goals - what are you trying to accomplish
1a) Take notes
2) Measure a baseline
2a) Take notes
3) Identify a bottleneck
3a) Take notes
4) Fix said bottleneck
4a) Take notes
5) Repeat until you meet your performance goals

Step 6) -- Report to your boss how much better your application is. But because of Step 1, you'll be able to tell him/her why it matters, right? :)

Terracotta and Spring - Powering High Throughput JEE Applications

2009-04-24T14:31:00.000-07:00

Recently, Terracotta did a webinar with Spring founder Colin Sampaleanu.

The webinar starts out by covering the benefits that a Spring+Hibernate+Terracotta application can deliver for your Java JEE application. The latter half is dedicated to running through a reference application that provides a solid starting point as you explore all that Terracotta+Spring can provide.

Examinator

The application demonstrated in the webinar is called "Examinator", and was jointly developed by SpringSource and Terracotta. Briefly,

Examinator is a full stack reference application which demonstrates with code how to build a higly scalable, highly available application using Spring, Hibernate and Terracotta

Highlights include:

Frameworks: Spring MVC, Spring Security, Spring Web Flow, Hibernate

Scale: 16 application servers, 20,000 concurrent users

Latency: Max of 5 ms response time

Find out more

For a full-length recording of the Webinar available for free visit http://terracottech.webex.com.

For a complete reference—everything you need to know including full documentation, how install and run Examinator— visit http://www.terracotta.org.

You can also access a live demo of Examinator at reference.terracotta.org.

SpringOne 2009

Speaking of Spring, I'll be attending the Terracotta booth at SpringOne Europe 2009 this coming week (April 27-29, 2009) in Amsterdam. Stop by if you're attending.

A simple tip for new Terracotta users - always run the Terracotta Developer Console

2009-04-22T08:16:00.001-07:00

With the release of Terracotta 3.0, I hope many of you have or are considering checking out Terracotta to see if it can help with scalability and availability of your Java application.

Of course www.terracotta.org - in particular the tutorials section with many simple recipes for exploring the many uses of Terracotta is a good place to get started.

But before you do any of that, I'd like to point out a best practice for learning and working with Terracotta. So, here's my tip for whenever you are working with Terracotta:

TIP: ALWAYS RUN THE TERRACOTTA DEVELOPER CONSOLE
It's easy to do, so I recommend before you run any samples, try an recipes, or work with your application, make sure to have the Developer Console running at all times.

How to run the Terracotta Developer Console

Running the Developer Console is easy. There are many ways depending on your context:

From the Welcome Application: Click the "Developer Console" link

From the command line: Run $TC_HOME/bin/dev-console.sh|bat

From Maven: Run $ mvn tc:dev-console

From Eclipse: Select Show Developer Console from the Terracotta menu.

Once you've got the Developer Console running, make sure to select the Connect automatically checkbox before connecting—this option will automatically connect the Developer Console to your cluster meaning you don't have to select "connect" every time you run a new cluster instance. This is very useful during experimentation (running sample demos and recipes) and integration testing.

Why should you run the Terracotta Developer Console?

We designed it so that you have access to a large array of information right at your fingertips. In particular, let's look at the user interface which is new in Terracotta 3.0:

One thing that I hope jumps out at you immediately is the presence of a new array of "speedo" dials - somewhat like the array of instruments that greets you when you step into the driver's seat of an automobile.

The resemblance isn't accidental. Those dials are there to give you up-to-the-second information about what's going on in the cluster - and to help pinpoint a problem - if there is one. Let's take a closer look:

Making use of the Speed Dials

As you can see, the dials are arrayed from left to right, giving you vital statistics about the cluster. The dials are separated into two groups:

Write Transacions

Impeding Factors

The Write Transactions dial contains a measure of the number of write transactions that are occurring in the system. Read transactions with Terracotta are exceedingly cheap (so cheap in fact that we don't measure them). Write transactions are a good measure of work being done in the cluster - so this measure is effectively a measure of how fast your application is running.

The Impeding Factors set of dials shows you a set of seven dials that show you statistics about other types of activity in the system. The activities displayed include such statistics as Object Creation Rate/s — the amount of new objects being added to the clustered heap per second — and Lock Recalls/s — the amount of lock requests that are being transferred from one client node to another.

Making use of the Runtime Statistics

Another very useful feature is the Runtime statistics panel. You can access this feature from the left menu tree by selecting the Runtime Statistics node.

The runtime statistics give you access to a wealth of realtime data with historical views. Unlike the Speedo Dials, the runtime statistics are kept for a longer period of time and graphed for you so you can see a historical view of how your application has been behaving.

Putting it all together

The Speed Dials give you instantaneous information - so they are visible all the time.

Look at the Write Transactions to measure your speed, and monitor the Impeding Factors to make sure nothing is slowing you down.

If there's something worth looking at in more detail, switch to the runtime statistics for more detailed information.

If there is a problem worth investigating, use the Diagnostics tools such as the Lock Profiler or Cluster Wide Thread Dump to debug further.

In other words ALWAYS RUN THE TERRACOTTA DEVELOPER CONSOLE!

Simple Java Messaging

2008-12-14T10:45:00.000-08:00

Following up on my recent post Java Distributed Lock Manager, sometimes you just need a simple way to pass messages between Java processes.

Messaging is a very useful pattern in Enterprise Integration, and there are many ways to do it. Apache Camel is a great tool when you need the flexibility and power to manage complex messaging patterns, including routing, filtering and the like.

If you just want to do something simple, though, that can be a challenge. The most common solution, JMS, requires quite a bit of boilerplate code, and requires selecting and running a JMS provider, which means selecting a J2EE container, Apache ActiveMQ, or others.

So what if you just want a drop-dead simple way of adding messaging to your application? Terracotta gives you that. (And also integrates well with other solutions, like Apache Camel if you need more power later on).

Simple messaging in Terracotta is built on the notion of clustering a LinkedBlockingQueue. Just as a LinkedBlockingQueue is used to pass messages between threads in a single JVM, it will be used in combination with Terracotta's JVM-level clustering to provide message passing between JVMs.

To demonstrate, here is a simple example.

import java.io.*;
import java.util.concurrent.*;
import java.util.concurrent.locks.*;

public class SimpleMessage
{
    private static ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
    private static BlockingQueue<String> queue = new LinkedBlockingQueue<String>();

    public static void receive() throws InterruptedException
    {
        System.out.println("Receiving messages...");
        while (true) {
            String msg = queue.take();
            System.out.println("msg >> " + msg);
        }
    }

    public static void send() throws Exception
    {
        while (true) {
            System.out.print("Enter a message> "); System.out.flush();
            String msg = new BufferedReader(new InputStreamReader(System.in)).readLine();
            queue.put(msg);
        }
   }

   public static void main(String[] args) throws Exception
    {
        // we use the presence of a lock to distinguish receiver from sender
        if (lock.writeLock().tryLock()) {
            receive();
        } else {
            send();
        }
    }
}

The app consist of two modes - a receiver mode and a sender mode. Normally, you would have an application specific mechanism of choosing whether you wanted to send messages or receive messages. For this example, we use a simple lock (for more information on using a ReentrantReadWriteLock with Terracotta, read the ReentrantReadWriteLock recipe). When free, the lock indicates no processes are receiving messages, so the process takes on the "receiver" mode. All subsequent processes take on the "sender" mode when the lock is held.

So let's run it with Terracotta and see how it works. First, we need to "cluster" the app. We need the lock and queue objects to be the same cluster-wide, which in Terracotta is called a root. So our Terracotta configuration file looks like:

<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">

  <application>
    <dso>
      <roots>
        <root>
          <field-name>SimpleMessage.lock</field-name>
        </root>
        <root>
          <field-name>SimpleMessage.queue</field-name>
        </root>
      </roots>
    </dso>
  </application>
</tc:tc-config>

Now, let's run two JVMs with Terracotta. First, we start a server instance:

$ start-tc-server.sh
2008-12-14 10:26:18,246 INFO - Terracotta Server has started up as ACTIVE node on
  0.0.0.0:9510 successfully, and is now ready for work.

Then, we start our JVMs.

JVM 1:

$ dso-java.sh SimpleMessage
Receiving Messages...

JVM 2:

$ dso-java.sh SimpleMessage
Enter a message>

Here, we enter a message, and see that it is printed in JVM 1:

JVM 2:

$ dso-java.sh SimpleMessage
Enter a message> hello world

JVM 1:

$ dso-java.sh SimpleMessage
Receiving Messages...
msg >> hello world

Further exploration

Try starting another JVM and see that they can both send messages to JVM 1. Try killing the receiver JVM and send messages to it. Then start another JVM. Since the lock is no longer held (Terracotta automatically releases any locks held by a JVM that exits the cluster) the new JVM will take on the receiver mode. Any messages sent while there was no receiver will have been queued, and will be printed on the startup of this new node.

And of course, you can see all the activity in the cluster. Try taking the receiver down again, send some messages using the sender nodes, then run the admin console. You'll be able to inspect the messages in the queue using the clustered heap browser.

This is just a demonstration of course - so to keep it simple I used a String as the message - but you could use any class.

For more fun with Terracotta, try the helpful "recipes" at Terracotta.org.

(Note, I've blogged about simple coordination in the past using Terracotta, which is similar)

Java Distributed Lock Manager

2008-12-11T22:14:00.000-08:00

Sometimes you just need a simple way to coordinate activities across more than one java process. There's a lot of choices out there. The database, JMX, distributed caches, JMS, filesystems. It would be nice if there was a simple, easy way to get distributed locks in a J2SE, J2EE, Web, SOAP, or AJAX application? There is.

Terracotta provides one of the easiest ways to get a distributed lock manager in your Java application. Terracotta plugs right in to normal Java threading constructs—synchronized, wait/notify, java.concurrent.locks.ReentrantReadWriteLock, and even java.concurrent.CyclicBarrier, which means you basically already know how to use Terracotta as a lock manager.

To demonstrate, let's work up a simple locking example and then drop Terracotta in. Our app will consist of acquiring a lock, "do some work" in a simple loop, and repeat. Here's the code (LockExample.java):

public class LockExample
{
    private static Object lock = new Object();

    public static void main(String[] args) throws Exception
    {
        while (true) {
            System.out.print("Waiting for the lock..."); System.out.flush();
            synchronized (lock) {
                System.out.print("I got the lock, doing work");
                for (int i = 0; i < 4; i++) {
                    Thread.currentThread().sleep(1000);
                    System.out.print("."); System.out.flush();
                }
            }
            System.out.println("done");
       }
   }
}

Simple. If we run this on the command line, we get:

$ javac LockExample.java
$ java LockExample
Waiting for the lock...I got the lock, doing work....done
Waiting for the lock...I got the lock, doing work....done

During the "work" part the "."'s are added 1 every second for four seconds. Fancy.

Let's add Terracotta. We need a tc-config.xml file which tells Terracotta how to provide the appropriate clustering behavior to our application:

<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">

  <application>
    <dso>
      <locks>
        <autolock>
           <method-expression>void LockExample.main(..)</method-expression>
        </autolock>
      </locks>
      <roots>
        <root>
          <field-name>LockExample.lock</field-name>
        </root>
      </roots>
    </dso>
  </application>
</tc:tc-config>

Now, let's run two JVMs with Terracotta. First, we start a server instance:

$ start-tc-server.sh
2008-12-11 22:26:18,246 INFO - Terracotta Server has started up as ACTIVE node on
  0.0.0.0:9510 successfully, and is now ready for work.

Then, we start our JVMs.

JVM 1:

$ dso-java.sh LockExample
Waiting for the lock...I got the lock, doing work....done

JVM 2:

$ dso-java.sh LockExample
Waiting for the lock...

It's a bit hard to demonstrate in a blog post, but the lock ping-pongs between the JVMs. That's it!

For more fun with distributed lock coordination, try these helpful "recipes":

ReentrantReadWriteLock

CyclicBarrier

Sales 101: The 5 sales archetypes

2008-11-22T20:38:00.001-08:00

There's a lot they don't teach you in school. As an engineer, most of the things I rely on day to day they never even so much as mentioned when I was in college, things like bug tracking, revision control, heck even writing error messages.

If you're an engineer, and in sales, it's even worse. I have spent the better half of my career in pre and post-sales as a "solutions architect" before moving on to product management. One thing that I have found to be useful is how to identify the person you are talking to on the other end of the phone.

I am not talking about figuring out if you are talking to a developer, a manager, an architect or a CEO. I mean what kind of person is this - what problems do they have, what kind of ego do they have, but most importantly, are they actually going to spend money. Like it or not, these considerations make a major contrbution to chances of your success.

So here's the 5 kinds of people you are most likely to meet, how to identify them, and what they mean to the bottom-line.

1. The beard-tugger

Summary: The beard-tugger thinks he is smarter than everyone else, and is committed to proving it. Thus he will spend the entire sales-pitch showing you just how smart he is.

Tell tale signs you have a beard-tugger: For every feature you talk about in your product, the beard-tugger will analyze it in 5 different ways and 10 different contexts. If he sees the slightest hole in your theory or implementation, he's bound to ask a question about it.

Pros: If you like to get into the guts of the product with someone, this is a great person to do it with. Remember though that if you are trying to sell this person, you need to let them "win" - e.g. never show them up to be lesser than the audience, or you risk losing them as a champion of your product.

Cons: There is no faster way to sink a pitch than to rat-hole on some minor detail of your product. The beard-tuggers middle name is rat-hole, so be careful.

2. The tire-kicker

Summary: The tire-kicker is out for a good time. They will take a sales call from anyone, and don't really have an agenda. They just want to see what you have. Often, this person thinks of themselves as "knowledgable about the market" so they will talk to you just to get a feel for your product to reinforce that feeling.

Tell tale signs you have a tire-kicker: if your product can be used in 10 different ways, he wants to know about every one of them. If you ask them about the specific problem they are looking to solve (and you should) the answer will be vague or none at all.

Pros: none really. This person may ultimately be a champion for you when they come around to solving a problem, so don't shun them. But you should get on with your life as soon as you can. Pitch your wares succinctly and move on.

Cons: big time suck. If you are not careful this person could come back time and time again, without a real problem to solve. That could be a big time sink for no real opportunity for business, in other words, a real waste of your time.

3. The science-experiment

Summary: Similar to the tire-kicker, but a little more involved, the science-experimenter is likely to have a problem in mind and wants to solve it, but there's generally no business behind it. This person is likely to be "exploring" technologies, but has picked yours as a likely candidate for a solution.

Tell tale signs you have a science-experiment: this person is allocating resources to a project to test out your product - e.g. run a POC (Proof of Concept). But if you press them, you will find there are no hard and fast requirements, so everything is either made up or guesswork. This person is likely to be enthusiastic both about their use case, and your product. They are probably even more enthusiastic about the opportunity for the mutual relationship to grow.

Pros: If you are successful, this could blossom into a sale, and the science-experimenter will probably tell you that at least 5 times in the span of 20 minutes.

Cons: A science-experiment goes nowhere 9 times out of 10. Stay away because there are no real requirements, and even if you succeed (which is unlikely, given the lack of a real requirement or business driver) there is unlikely to be a pot of gold at the end of this relationship, despite the claims of the science-experimenter to the contrary.

4. The delusional

Summary: This person is trying to use your product either for an outlandish use case, in an extreme way, or worse, in every possible way (e.g. they think you have the silver-bullet to every problem known to man).

Tell tale signs you have a person with delusions of grandeur on the line is that they will be obsessed with how successful their product will be. That is to say, they have either no product, or a very small product at the moment, and this person is most likely obsessed with the massive growth they are just about to incur.

Pros: None. This is one of the most dangerous people to engage with. Get out fast.

Cons: Very likely to be enthusiastic with your product and want to POC it. This is the most dangerous of people, because they will be pandering to your ego, which means you will be very easily swayed by them because they believe in you, and they love your product. They will spend a very long time in a POC with your product, because the requirements will likely be unrealistic bordering on ridiculous. If you bend to their whims, you will likely be adding features/fixing bugs that have no bearing on real business.

5. Your customer

Summary: This is the person you want to sell to. They have a business case, and likely some pain. When you discover what that pain is, you should have a product that solves that pain, for less money then they are currently spending.

Tell tale signs you are talking to your ideal customer: at the end of your initial call with this person, they should have asked you how much your product costs, and they should be wary that your product can actually solve their problem (afterall, nothing else has to date), but be slightly optimistic that maybe you can solve their problem, and be willing to try. They should have a use case that is within the bounds of what your product can solve, and they should be interested in what the next steps are after the call.

Pros: Sell them quickly - this is where you should be spending your time.

Cons: None - your only challenge is to identify this person. If you don't, you're losing money.

Now, go out there and make some money! :)

Grails + Quartz + Terracotta

2008-11-17T21:58:00.002-08:00

1) Grails recently added plug-in support for Terracotta.

2) Grails recently added plug-in support for Quartz.

3) Terracotta supports Quartz

So....wouldn't it be possible to demonstrate Grails, Quartz and Terracotta all working together? Seems like a fun project.

What is a Memoizer and why should you care about it?

2008-09-21T15:59:00.000-07:00

A Memoizer is a class created by Doug Lea/Brian Goetz. It's basically a create-and-store cache. That is, it has one very useful property:

A Memoizer is a Generic class that encapsulates the process of creating a value, and remembering the value (memoizing it) so that subsequent calls to get the value will return the memoized value, and not call the Factory method to create it.

Why do you care?

A Memoizer encapsulates the process of "getOrCreate()" which is highly useful when you are getting a resource that does not change by a key (think cache) and which is generally considered to be expensive to create.

But, you ask, I already can do that, why do I need a Memoizer? Glad you should ask. Memoizer is unique in its solution, because it caches a value by key, but its implementation is very efficient. It has the following additional properties:

Calls to get items are concurrent

Calls to get items for a particular key are guaranteed to call the Factory method for that key once and only once

It's important to understand that what the Memoizer does is both efficient, and correct. It's relatively easy to roll your own "getOrCreate()" method that has only one of those properties (and most people do). It's not so easy - or obvious - to do both and that's why you should use a Memoizer - and not roll your own.

To review, let's see the strategy most people would use for calling the Factory method once and only once. Here's how to call the Factory method once and only once:


/*** SUB OPTIMAL IMPLEMENTATION THAT IS CORRECT, BUT NOT CONCURRENT ***/
private final Factory<A,V> factory = ...;
private final Map<A, V> map = new HashMap<A, V>();

public V getOrCreate(A key) {
    synchronized (map) {
        if (map.contains(key)) { return map.get(key); }
 
        // create
       V value = factory.create(key);
       map.put(key, value);
       return value;
    }
}

But of course, the line synchronized (map) should be a tip-off that this implementation is not concurrent
since there is only a single lock protecting access to our map. Can we do better? Sure. Let's use a ConcurrentHashMap to make our getOrCreate() method concurrent (but as we'll see, will not have the once-and-only-once property):


/*** SUB OPTIMAL IMPLEMENTATION THAT IS CONCURRENT, BUT NOT CORRECT ***/
private final Factory<A,V> factory = ...;
private final ConcurrentMap<A, V> map = new ConcurrentHashMap<A, V>();

public V getOrCreate(A key) {
    if map.contains(key) { return map.get(key); }
    
    // map doesn't contain key, create one -- note that first writer wins,  
    // all others just throw away their value
    map.putIfAbsent(key, Factory.create(key)); 

    // return the value that won
    return map.get(key);
}

So, those are the two implementations, each implements either correctness (once and only once) or performance (concurrent puts and gets) but neither accomplishes both simultaneously. Of course, that's the whole point of this article. To introduce you to Memoizer. So how does Memoizer ensure correctness (once and only once) while being simultaneously concurrent?

The trick is to delay the call to the Factory create method into the future. We already know how to make the operation concurrent, but making the operation concurrent means that the first writer will win, and all the other writers will lose. So we delay the Factory create call by wrapping that call in a FutureTask. In other words, instead of putting the actual value into the map, we put a Future (which is a wrapper for getting the value sometime in the future) into the Map.

(Note: If you have not yet become familiar with the Future and FutureTask classes introduced in the JDK 1.5, you should)

By putting a Future into the map - and not our actual value - we can move the work of calling the Factory create method into the future - specifically we can move it until after the winner of the race to put the value into the map has been determined. That enables us to call get - which calls create on the Factory - on the one and only Future instance that is contained within the Map.

The full code for Memoizer illustrates this trick. Note that the Computable interface specifies a generic "Factory". Also note that I have not been able to find a library or canonical reference for Memoizer. The best I have is here: Memoizer.java.

Here it is re-created for your convenience (note that I have adjusted it slightly, for example allowing the user to specify the number of segments created in the underlying ConcurrentHashMap):


public interface Computable<A, V>
{
    V compute(A arg) throws InterruptedException;
}

public class Memoizer<A, V> implements Computable<A, V>
{
    private final ConcurrentMap<A, Future<V>> cache;
    private final Computable<A, V> c;

    public Memoizer(Computable<A, V> c)
    {
        this(c, 16);
    }

    public Memoizer(Computable<A, V> c, int segments)
    {
        this.cache = new ConcurrentHashMap<A, Future<V>>();
        this.c = c;
    }

    public V compute(final A arg) throws InterruptedException
    {
        while (true) {
            Future<V> f = cache.get(arg);
            if (f == null) {
                Callable<V> eval = new Callable<V>() {
                    public V call() throws InterruptedException {
                        return c.compute(arg);
                    }
                };
                FutureTask<V> ft = new FutureTask<V>(eval);
                f = cache.putIfAbsent(arg, ft);
                if (f == null) {
                    f = ft;
                    ft.run();
                }
            }
            try {
                return f.get();
            } catch (CancellationException e) {
                cache.remove(arg, f);
            } catch (ExecutionException e) {
                LaunderThrowable.launderThrowable(e);
            }
        }
    }
}

You will of course need the LaunderThrowable implementation to compile this example. For a full code listing, hop on over to Terracotta, where I show not only a full working example, but demonstrate how it works with Terracotta across two nodes:

Full Memoizer Example with Terracotta

What Is Terracotta?

2008-09-19T13:10:00.001-07:00

There are so many ways to answer this question - of course our website has it's own way. I recently wrote an e-mail to someone who asked me about Terracotta, and I figured why not share it on my blog?

Here is my response (updated a bit to be more appropriate for a blog)...

Terracotta clusters at the level of the JVM. It is a 100% Java solution. We use the Java API and Memory Model as an abstraction layer into which we inject clustering.

There are some specific differences with the way that we implement clustering and the way that others do (in fact, in that regard, Terracotta is an entirely unique solution, I do not believe there is anything else like it).

There are two main differences in Terracotta -the programming model and the performance (the two go hand in hand, one cannot be had without the other).

For the programmer , Terracotta is injected at the Java level, meaning that programming a "distributed" application with Terracotta is no different than programming a multi-threaded or concurrent application. Terracotta makes use of all of the concurrent facilities built in to the Java language and API so that the definition and operation of those facilities are extended across a cluster - in other words each node that you add to the cluster simply becomes more threads available to your application.

Put another way, you program plain POJOs, and Terracotta manages replication services of those POJOs, maintains the identity of those POJOs and provides locking services using either synchronized/wait/notify or java.util.concurrent libraries e.g. ReentrantReadWriteLock - all across the cluster. (Again the model is simply threads in one node are no different than threads in another node - all standard Java operations "just work")

Of course this doesn't mean that programming across nodes separated by a network is free. Terracotta doesn't believe that an architect doesn't have to know about that interaction. We do think one has to architect for Terracotta, but we do not believe you should have to *program* to it. The analogy is much like that of the garbage collector - you don't program to the Java Garbage Collector, but you do architect your application around it's presence.

From a high level architectural level, Terracotta uses a tiered architecture. All application nodes talk to the Terracotta Server using TCP (never multicast, and never to one another, P2P is provably not scalable to provide coherent locking). The Terracotta Server can be clustered (called the Terracotta Server Array) for availability and scalability. It's a lot like the Database in that regard - the Terracotta Server (Array) is the composer in the symphony, coordinating the actions of the application nodes, and storing the data safely - all the way to disk in fact just like a database (and transparently from the application nodes perspective). When you need more availability or scalability, you just add more Terracotta Servers (no changes to your application are required).

The replicated data in Terracotta is 100% coherent across the cluster, and always stored safely to disk. This feature is unique to Terracotta given the performance levels it can achieve, which is the other main difference between Terracotta and all other solutions in the same space.

Other clustering solutions in the same space claim to have linear scaling, coherent data, and high performance - all delivered at the same time. That's basically a lie - none of them come even close to delivering all three at the same time (e.g. coherent data is possible but it's really slow, high scale can be achieved, but only for non-coherent data). And no product in the same space delivers the same performance that Terracotta does - it is simply in a class of its own since it is the only product that does not rely on brute force replication techniques such as Java Serialization. Coupled with some really innovative ways to eliminate or reduce network latency for locking, Terracotta provides a solution that can give data coherence guarantees, with amazing performance.

What all of this translates into is a solution that is flexible (a programmer can pick and choose his/her own programming stack and domain model), fast (no other clustering solution can send delta updates over the network), and more importantly, manageable. The Terracotta architecture is not an accident - it is an intentional design decision that ensures that managing a Terracotta Cluster is simple and efficient. Just like the proven 3-tier architecture of web application nodes and database servers, Terracotta stores application clustered memory in a well known location - the Terracotta Server Array. Loss of application nodes in a Terracotta Server Cluster, like in a 3-tier application, does not risk data-loss in any way, and likewise, loss of the Terracotta Server process(es) does not jeopardize the data in any way.

Cluster Deadlocks ROCK with Terracotta

2008-09-09T10:46:00.000-07:00

Am I crazy or what?

No not really. You see I just happen to have seen more than one customer run into a cluster deadlock, and it turns out that solving the issue with Terracotta is awesome (actually, Terracotta can automatically detect it in an upcoming version, but shhh don't tell anyone I told you that)

It's funny, really, because I have been hearing this dumb idea that somehow clustered deadlocks with Terracotta is actually this really scary thing -- ooooh watch out for that complicated Terracotta thing, it uses LOCKING (oh gosh) and that can lead to CLUSTERED DEADLOCKS. Oh my. (Anyone know where I can get a clustered deadlock costume, it's almost halloween!)

Really. It's like a bad rumor I keep hearing over and over again. What do they call that when people try to scare other people with rumors that aren't true .... F.....oh nevermind. Here's why deadlocks truly are better with Terracotta:

First, what do we get with Terracotta?
- Kill a JVM, release its lock.
- Kill a JVM, don't lose your state

Why does that matter? Well what do you do when you see a deadlock with a regular Java application? Since it's pretty much hosed, you have to restart it (usually you probably debug the hell out of it first and try to fix the deadlock). But the app is hosed. Unless you happen to have coded a "stateless" app - you've also lost your app state. Bummer :(

Well, not so with an app running on Terracotta. First of all, you don't have to kill the whole app. In fact, if you do actually get a clustered deadlock, you just have to kill one half of the deadlock (because the locks are released, get it?) and the other half will actually get to complete it's operation. How do you do that? Well since the app state is highly available, you can kill any node at will.

So it's simple to resolve a clustered deadlock with Terracotta - just do a rolling restart of every client JVM. That's it. When you hit one half of the deadlock, and kill that JVM, the lock that the other side of the clustered deadlock wants will be freed, and it will go on its merry way.

Now of course, you still need to debug the hell out of your app :). When you fix the app, just update it in place, do another rolling restart, and voila! Fixed deadlock with no downtime.

So to summarize, deadlocks with Java:
- Have to restart
- Lose app state
- Downtime BAD

Deadlocks with Terracotta (e.g. Clustered Deadlocks):
- Rolling restart of application nodes
- Preserve application state
- No downtime GOOD

How can I make sure Terracotta is wired into my application?

2008-09-08T06:57:00.001-07:00

Per the title of this blog post, I'm going to show you how to make sure that Terracotta is enabled in your application.

One of the funny things about Terracotta is that applications typically don't know it's there. This is normally a really good thing - it means you can use the 'ole binary search debug trick. (Just remove Terracotta from one half of the application one debug session at a time to quickly narrow down the issue).

But of course, in production, we want to make doubly sure Terracotta is actually running - yeah sure the application might be happy as a clam churning out txns, but we want to make sure those txns are getting replicated for high availability, right? :)

Fortunately, finding out if Terracotta is wired into your app is really easy. We can make use of the fact that most of the regular Java classes are instrumented by default, so we can use reflection to interrogate one of them. I chose String since I think it's a well known class. Here's the code:


   public static final boolean isTCEnabled()
   {       
       try {
           String.class.getMethod("__tc_decompress");
           return true;
       } catch (Exception e) {
           return false;
       }
   }

That's it! We just test to see if a special Terracotta method is present, and if so, we know Terracotta is wired in. All you have to do is put that into your application startup somewhere, and complain loudly if the method returns false :)

Return to civilization...re-launch of Terracotta.org

2008-08-20T00:18:00.000-07:00

It's been a bit since I posted...we've been really busy cranking away on a new site at Terracotta, and it's finally out.

I'm really pleased with the site, I hope our users are too. The goals for the site were:

Simple
Clean
Professional

and of course, useful! :)

What's New

Everything, really :). Well, not everything, but a lot. On the graphics side of things, we added a lot of JQuery magic. Not so much for the magic itself, but to make the user experience more pleasant. Where possible, popup windows and user transitions have been replaced by images that zoom in place, and drop-down panels that help keep the focus where it should be - on the task at hand.

Also of note, although it's of little practical use now, I implemented a nice CSS effect for our menus that allows for them to look 3d and stylish but only requires one simple transparent png (no, I don't care about IE6. It's disgusting). It's basically a minor modification of the transparent text effect described here in this blog post. I'll probably write this up as a separate post.

But really, we didn't focus that much on the look or the feel, but the design. The look and feel came primarily from the design, and the goals, so we knew when something worked, and when it didn't. If it was distracting, complicated, or busy, it didn't make the cut. I bet I'll be blogging about the design process before long.

Here's a brief preview of some of the new design elements:

Clean Simple Menu

Pretty self evident I think.

Process Oriented Site Flow

Well, pretty hard to miss, really. I hope it doesn't get any easier than 1) 2) 3) 4).

Simple Controls, My Terracotta

Main controls are on every page, easy to find, but hopefully unobtrusive until you need them. Also this is the preview of "My Terracotta" - expect more.

Drop Down Panels

As I mentioned, drop down panels help get stuff done without leaving the context. I think a site shouldn't need Help, but we added it anyway. We all really hope it's unnecessary, but if it helps just one person, it's worth it. We're really committed to getting everyone successful with Terracotta.

Lots of new content

As I said, we worked really hard on trying to capture how someone should come to Terracotta, learn and understand it, integrate, test, tune, deploy and operate it. The design and the process are integral to the site - so much so that we even embedded some process diagrams to anchor where the user is in the site.

Architecture Patterns

Finally, we put some massive effort into capturing and describing successful architecture patterns, and how they work with Terracotta. There's a whole section devoted to describing these patterns, and there is a lot more on the way.

Go check it out for yourself, we're live at http://www.terracotta.org. We'd love to hear your feedback.

Shortcuts using JIRA

2008-07-14T11:36:00.001-07:00

At Terracotta, we use JIRA for issue tracking (see here at jira.terracotta.org).

Today, I stumbled on a really nice feature. On a whim, I thought "I wonder if they implement shortcut (hotkeys)?". So I tried it out - sure enough "Ctrl+e" edits a JIRA issue, and "Ctrl+s" saves it. NICE!

+1 for Atlassian.

Chronicles of a Terracotta Integration: Compass

2008-03-31T08:57:00.000-07:00

Last week, I met up with Shay Banon, author of Compass, at the The Server Side: Vegas conference. We thought it would be great to see if we couldn't crank out an integration between Terracotta and Compass. You can read more about our integration from Shay himself.

I wanted to write a log of our efforts, because I thought it might provide some insight for anyone considering integrating Terracotta into their own project. I was particularly happy with our effort, because it outlines what I feel is the best approach for developing with Terracotta. The approach is actually quite simple. Because Terracotta adds clustering concerns to your application using configuration, you don't write code directly to Terracotta. Instead, you just write a simple POJO application without Terracotta, and then add the clustering later.

So the approach I recommend is the following:

Figure out how to implement the solution using a single JVM. NO TERRACOTTA. Use just simple POJOs and threads.

Implement and test your solution.

It helps to have envisioned, beforehand, what part of your implementation will become a Terracotta root. But it's not necessary. If your application is stateful, it will have a root.

Using the root, start with a basic Terracotta config file, and build up the appropriate config file to cover all the instrumentation and locking.

Test your application again, with a single jvm, but this time with Terracotta.

Tune your implementation.

Move to 2 or more JVMs.

That's it. So how did this play out for the Compass integration? Here is my rough recollection of the action.

10:00 am - Shook Hands - Shay and I met up at the conference.

10:05 am - Started coding - First we chatted a bit about our strategy. It seemed easiest to start with the existing Lucene RAMDirectory implementation and tune it up a bit.

10:30 am - Strategy decided - Based on my knowledge of Terracotta, and Shay's knowledge of Lucene/Compass, we decided on the following:

Start with the Lucene RAMDirectory implementation, but rewrite it as necessary to fit a simple POJO model

Since RAMDirectory is mostly unmaintained, we knew we had to just go through the implementation and clean it up. It comprises about 4 classes total, about 100 lines long, so the task was feasible.

Because Terracotta can just "plug" in to a well written application, and Shay has a comprehensive unit test suite (over 1,000 tests), a load test, and a concurrency test, we'd write the implementation first in POJOs

After verifying that the implementation works as expected in pure POJOs, then we would work on the configuration to inject Terracotta clustering

After running the solution with Terracotta, we would tune it.

And finally, we would wrap up various bits and pieces into a Terracotta Integration Module (TIM)

11:30 am - POJO Implementation done - We ended up rewriting the RAMDirectory, which was fine because it was in need of an overhaul anyway. Rewriting its implementation meant we now had a good understanding of the implementation.

Just a quick note - it was a real joy coding with Shay. He is a super smart guy, and it's great to work with someone like that. Of note, he really understands synchronization, which is really important to write applications correctly. Even better, he really got the principle of writing better code by writing less code. We went through the RAMDirectory implementation with a weed wacker, and what was left was about 1/2 the code. That was more readable and more maintainable. And is better performing. That was fun.

12:00 pm - Unit Tests pass - With some minor corrections, we had unit tests passing. We were both running out of power, and hungry, so we took a break to eat lunch, and agreed to resume in the afternoon.

1:30 pm - Write the Terracotta config file - While writing the POJO implementation, we already knew the key concepts we were going to need for writing up our config file. We added the appropriate instrumentation. We added the locking. A few config statements later, we had a working Terracotta configuration.

2:00 pm - We had Compass running on Terracotta! - Approx. time elapsed? 2 1/2 hours (most of which was spent rewriting the RAMDirectory implementation)

2:30 pm - Tuning Time- At first Shay threw me - he said oh man it looks like it's running really fast. Except it turns out he wasn't testing the right thing. And then he tells me oh man its really slow!

Now don't misunderstand this. I know Terracotta can go really fast. But I wasn't in the least bit surprised. And you shouldn't be either. How many pieces of code have you ever written that compiled and ran correctly - on the first try? Right. One, if you are lucky.

Terracotta is kind of like that. The first step is to get it right. And that means synchronization, and locking, and once you have all that, your application runs correctly, but slowly.

Fortunately, it's easy to fix.

And so I taught Shay how to tune up his Terracotta integration. Or rather, I showed him the tools he needed, and he went to town. I just sort of stood by watching, giving the occasional comment or two.

This was the fun part. It was time to take out the Admin console. The Terracotta Admin console gives you a wealth of information about your application. Of note:

You can browse your clustered data in realtime

You can monitor realtime statistics - including Terracotta txns/sec, Java Heap Memory, and CPU

You can access lock statistics using the lock profiler

You can snapshot over 30 metrics using the Statistics Recorder and visualize them using the Snapshot Visualizer

We started first with the object browser. Once convinced that we had the right data in the cluster, we moved on to performance.

On our first run, we measured the Terracotta txns/sec. I was actually pretty impressed to see that our server on his MacBook Pro was cranking out 10k/sec. But I knew we wanted this number to be lower. A lot lower.

So here comes the first rule for tuning Terracotta: adjust your locking to match your workload. It turns out that we had enabled an autolock for every single byte being written to the Lucene "files" - and this was hurting us pretty bad. Because we already had a lock on our byte array that we were writing to, we actually just deleted the synchronization, and the lock config from the method that wrote bytes into the "file" - and we observed a big drop in the Terracotta txns/sec. We went from the aforementioned 10k/sec to about 1750/sec.

Now what this means is that the Terracotta server was working just about 10x less for the same workload. And that means we were doing more work/transaction, and so our performance improved accordingly. You get the same effect with Hibernate - it batches up a bunch of little POJO updates into a single SQL statement - and that means you can do more real work because each SQL statement has more data in it. Lots of little SQL statements means lots of overhead, and maybe more SQL queries executed/sec, but much less application txns/sec. Same concept here with locking.

How did we identify what lock(s) to target?

That's the second rule of tuning with Terracotta: USE THE ADMIN CONSOLE

We used the lock profiler feature included in the Admin Console to determine the exact stack trace that generated these locks. The process is simple:

Enable lock profiling with stack traces in the Admin Console,

run your application,

then refresh the view to get a count of the lock acquires/releases/held times etc.

sort on # of lock acquires, and now you know what lock is being requested the most, what stack trace caused that lock, and what Terracotta config was responsible for making that lock.

Armed with this knowledge, Shay set about eliminating most of our superfluous locking. Turns out that creating a Lucene "file" is a single threaded affair, so were able to create a single lock to cover the entire process of "writing" to a file, and that cut out about 90% of our locking.

At the end we got down to about 750 Terracotta txns/sec, which improved the application performance quite a bit.

Still not satisfied, we moved on to the Terracotta Statistics Recorder. This is a new feature in Terracotta 2.6.

Turning on this feature records just about everything you ever wanted to know about your application, Terracotta, the JVM, and your system (including CPU, disk stats, and network stats). You can export these stats as a CSV file, and import them into our Snapshot Visualizer Tool. The SVT gives you a view like so:

4:30 pm - TIM time - We were pretty satisfied with the performance. Even though we wanted more, Shay felt it was best to focus on turning Compass into a TIM (Terracotta Integration Module).

5:30 pm - Time to call it quits - We had hacked up the ant build.xml file to get ourselves a TIM in no-time - except that it wouldn't quite load correctly. (Later we learned we had just specified the filename wrong - easy fix).

Overall, I thought we had a pretty good day. We wrote and tuned a Terracotta integration in about 6 hours flat. With a few more hours of work, Shay was able to complete the integration.

I was really happy to use some of the recent tools we have been building, like the Lock Profiler and the Statistics Recorder. Seeing the real-world use of those was invaluable, and confirmed that our commitment to enabling the developer to self-tune by providing enhanced visibility is spot on.

I am looking forward to people downloading 2.6, trying out these awesome tools for themselves and providing feedback!

Fun with Distributed Programming

2008-03-30T20:55:00.000-07:00

Something about the nature of distributed programming makes it quite divisive. You either love it or despise it. It's rare that I've run into someone who is ambivalent about it.

Those that love it, love it because it's hard core. They're proud to know all the ins and outs of dealing with failures, at the system, network, and application level. All of that specialized knowledge is also what turns off the rest of us.

It's kind of like database programming. There are only a select few who really like it. The rest of us only do it because we have to.

Well, I honestly think that Terracotta changes the game. The key is that Terracotta makes distributed programming fun because it takes away most of the distributed programming part, leaving you with just the fun part.

It reminds me of when Linux came out. Everyone loved it because they could just tinker with different schedulers, and not have to think about building an OS from scratch, just to try out a new idea. That's what Terracotta is like. It manages all the hard networking and distributed programming parts, so you get to just play with the algorithms.

Interested? Let's look at a real (if contrived) example. Let's suppose that you have to build the following:

a service that executes periodically to do some work

you don't care where this service runs, only that it runs

it has to run, but one and only one system can run it

you've got a cluster of n systems, you'd like any one of them to be responsible for running the service

If it were a single JVM, you could do a thousand things, like use a java.util.Timer, or Quartz, or even your own simple Thread with a delay loop in it.

But in a cluster? The choices for synchronizing the behavior of a number of JVMs across a cluster quickly eliminate the fun part, leaving just tedious, boring, and mundane work to be done. Cluster synchronization you're thinking. What should I use? TCP? Multicast? Shared file system locking? A shared database? RMI? JMS? EJBs? Oh dear.

But wait. Terracotta provides synchronization primitives that work across the cluster just like in a single JVM. So that means getting this right in a single JVM means getting it right across the cluster. Could it really be that easy? And fun? Yes!

Let's have a look. For the sake of simplicity, let's do the simple thing. We'll write the delay loop version. We'll implement it as a Singleton that implements Runnable, so we can pass the Singleton to a Thread. Here it is:

public class SimpleWorkRunner implements Runnable
{
    // mark as a Terracotta root
    private static SimpleWorkRunner singleton = new SimpleWorkRunner();

    // singleton pattern - private constructor so there is only one
    private SimpleWorkRunner() { }

    public static SimpleWorkRunner instance() { return singleton; }

    public synchronized void run()
    {
        while (true) {
            // do work
            ...
            try { Thread.sleep(2000); } catch (InterruptedException e) { }
        }
    }
}

That's it! In every JVM, kick off a new thread against the singleton:


...
new Thread(SimpleWorkRunner.instance()).start();

And we're done!

You might have noticed one thing - the run method is synchronized. In a single JVM, this will mean that more than one Thread executed against this Singleton will result in only one Thread winning the synchronization race, and executing the run() method.

In a single JVM, this may not be that important, since there might only ever be one Thread. But with more than one JVM, we will always start at least one Thread per JVM, and that means we have to ensure, per our requirements, that only one Thread ever enters run() at a time.

Terracotta takes care of that for us. We just write the synchronized block, and Terracotta converts that into a cluster lock. And just like in a single JVM, only one Thread - across the cluster - will win the race to enter the run() method.

(Also of note is that this particular implementation assumes that one and only one Thread should assume control and never relinquish it. That was the purpose of the implementation, if you wanted to "bounce" the control around the cluster then we should implement the run method differently depending on the requirements.)

The Terracotta config for this class is trivial. We need to tell Terracotta that the singleton should be a Terracotta root. A Terracotta root will always be the same instance across the entire cluster, which is exactly what we want for a singleton. And we need to autolock the run method so the synchronization is applied to the cluster, not just a local JVM. Here's the config for that:

<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">

  <application>
    <dso>
      <locks>
        <autolock>
           <method-expression>void SimpleWorkRunner.run(..)</method-expression>
        </autolock>
      </locks>
      <roots>
        <root>
          <field-name>SimpleWorkRunner.instance</field-name>
        </root>
      </roots>
    </dso>
  </application>
</tc:tc-config>

We didn't have to worry about the dirty details. Teracotta did. And that means distributed programming becomes fun again!

Find out more:

Terracotta.org - home page
Quick Start - download and see the demos
Cookbook - simple recipes that demonstrate Terracotta in action

Note that this example is very similar to the Single Resource recipe. Try it out first to get started.

Extra Credit
How does another JVM in the cluster gain control of the task? (Hint: Is it possible for more than one Thread to enter the critical section in run()? In normal Java - no. But what happens in Terracotta with more than one JVM?)

A Clustered ClassLoader

2008-03-22T15:56:00.000-07:00

If you're building a distributed system, or contemplating building a distributed system, you might have run into this one before:

You write and compile your classes in Eclipse
You try out your classes on your laptop -- they work (woohoo!)
Its a distributed system so you need to make sure your classes work in a true distributed environment
You publish your classes to the distributed systems
You try out your classes -- and they don't work (boo!)
You fix the problem.
You publish the classes again.
Rinse, repeat.

After doing this a few dozen times, you find that publishing your classes to distributed systems is a total PITA that you would rather avoid altogether.

Or, you might have an application, like Master/Worker in which you deploy some part of the application at deploy time, but you deploy other parts of it during run-time. In the Master/Worker case, you deploy the Master and the Worker, but the Work comes and goes, and you'd like to be able to deploy new Work easily and trivially. In the Master/Worker case, since Masters are usually in control, and there is a farm of Workers, you'd like to deploy some new work to the Master, and let it send the Work to the Workers. Knowing about the Work up front on the Workers is a non-starter.

Some solutions to this problem?

Java has dynamic code loading capabilities already. Deploy your class files to a shared filesystem like NFS, and deploy your code to a shared directory.
Java also supports loading code from URLs (thanks to it's Applet heritage) so deploy your code to an HTTP server and you're set
Factor your application such that new classes aren't needed - just make the new definitions "data" driven
Embed a scripting engine, so you can pass Strings and interpret them as code - BeanShell, Jython, JRuby, Javascript, and Groovy all come to mind here...

Those are all fine solutions, but it never hurts to have more tools in your toolbox does it? Especially if you're already using Terracotta, wouldn't it be nice if there was some way to just leverage Terracotta's core clustering capabilities to build a clustered classloader?

I've done just that. Here's how it works:

Your application tries to instantiate a class, which means it asks the currently in scope ClassLoader to instantiate the class (by name)
By launching the application under the clustered classloader, it is in scope.
The clustered class loader has a Map<String, byte[]> that correlates classnames to bytes
The clustered class loader looks in this Map, if the classname is found, it uses the byte[] to create the requested class using defineClass()
If the class wasn't found in the Map, then it looks in the filesystem to find the class
If the class bytes are found on the filesystem, then it reads them into a byte[], and stashes them in the clustered Map<String, byte[]>
If the bytes aren't found, it just delegates to the parent classloader

I've omitted some of the finer details. The Map used is actually a Map<String, ClassMetaData> where ClassMetaData is a class that holds a long modified and byte[] bytes.

Let's have a look at the important parts of the ClusterClassLoader:


public class ClusterClassLoader extends ClassLoader
{
    private static final String NAME = "ClusterClassLoader";
    
    private static Map<String, Class> classes = new HashMap<String, Class>();
    private static Map<String, ClassMetaData> bytes = new HashMap<String, ClassMetaData>();
    private static transient boolean loaded;

    ...

ClusterClassLoader is defined to extend ClassLoader. It has a NAME field, which will be used to give a name to this classloader. This is a requirement for a classloader used by Terracotta. Normally Terracotta does this for you, but we are defining a new classloader, so we have to follow the naming rules for Terracotta (naming gives ClassLoaders across the cluster a unique identity).

A classes field is defined, which caches the result of the defineClass operation in the local JVM only. A bytes field is defined. This field is marked as a root, so that it can be shared with every other instance of ClusterClassLoader in the cluster.

The constructor detects if Terracotta is loaded using some reflection, and if so registers the classloader and sets a flag to enable cluster classloading features:

    public ClusterClassLoader()
    {
        super(ClusterClassLoader.class.getClassLoader());
        try {
            Class namedClassLoader = findClass("com.tc.object.loaders.NamedClassLoader");
            Class helper = findClass("com.tc.object.bytecode.hook.impl.ClassProcessorHelper");
            Method m = helper.getMethod("registerGlobalLoader", new Class[] { namedClassLoader }); 
            m.invoke(null, new Object[] { this });
            loaded = true;
        } catch (Exception e) {
            // tc is not present, so don't do anything fancy
            loaded = false;
        }
    }

Next, the definition of loadClass is overridden:

    @Override
    public Class<?> loadClass(String name) throws ClassNotFoundException
    {
        return findClass(name);
    }

and so is findClass:

     @Override
    protected Class<?> findClass(String name) throws ClassNotFoundException
    {        
        if (!loaded) {
            return getParent().loadClass(name);            
        }
        
        Class result = null;
        synchronized (classes) {
            result = classes.get(name);
            if (result != null) { return result; }

            result = loadClassBytes(name);
            if (result == null) { return getParent().loadClass(name); }
            classes.put(name, result);
        }
        
        return result;
    }

This is the bulk of the algorithm. The loaded flag is set when the class loader is instantiated. It used a bit of reflection to determine if Terracotta was even present in the JVM. If not, it is set to false, and the ClusterClassLoader just delegates to the parent class loader.

If Terracotta is present, then it checks to see if the class has already been defined. If so, it is returned directly from the classes cache. If it has not, then it gets the bytes from the loadClassBytes method. If that cannot find the bytes, then it asks the parent class loader to load the class.

The bulk of the implementation is done in the loadClassBytes method:


    private Class loadClassBytes(String name) throws ClassNotFoundException
    {
        ClassMetaData metaData;
        
        synchronized (bytes) {        
            try {
                File f = null;
                metaData = bytes.get(name);
                URL resource = ClassLoader.getSystemResource(name.replace('.',File.separatorChar)+".class");
                // if resource is non null, then the class is on the local fs (in the cp)
                if (resource != null) {
                    f = new File(resource.getFile());        
                }
                
               if (metaData != null) {
                   // if it's cached, but not on the fs, return it.
                   // if it's cached, but on the fs, check to see if it's 
                   // up to date
                   if (f == null || metaData.modified >= f.lastModified()) { 
                       return defineClass(name, metaData.bytes, 0, metaData.bytes.length, null);
                   }
                }
                
                // load from the fs
                byte[] classBytes = loadClassData(f);
                Class result =  defineClass(name, classBytes, 0, classBytes.length, null);

                try {
                    result.getDeclaredField("$__tc_MANAGED");
                    // it's managed so cache it
                    bytes.put(name, new ClassMetaData(f.lastModified(), classBytes));
                } catch (NoSuchFieldException e) {
                    // not managed don't cache it
                }
                return result;
            } catch (IOException e){
                return null;
            } 
        }
    }

This method looks for the cached bytes, and for a file that corresponds to the class. If both are found, then it compares the modified date of the two. If the modified date of the bytes are greater than or equal to the file, then it returns the bytes in the cache. Otherwise it loads the bytes from the file. Once the bytes are loaded, defineClass is called to turn the bytes into a class file.

At this point, the ClusterClassLoader can check to see if the class is instrumented by Terracotta. Every instance of a class that is shared by Terracotta must be instrumented, so it's not necessary to cache class bytes for classes that are not instrumented by Terracotta. If the class is instrumented by Terracotta, then the ClusteredClassLoader stashes the bytes into the class bytes cache.

Click here if you would like to see the source code to ClusterClassLoader in its entirety

UPDATE: This project has been included in the tim-tclib project, and is a runnable sample. More details can be found in the sample readme.html

I've put the whole thing together as a simple runnable example. You just have to check out the source for the project, and run a few simple Maven commands. You can get the demo from here:


$ svn checkout http://svn.terracotta.org/svn/forge/projects/labs/tim-clusterclassloader clusterclassloader
$ cd clusterclassloader

The demo defines a main project, and two sub projects. The first sub project, sample, is responsible for putting classes into a queue. The second sub project, sample2, reads from the queue. To test the effectiveness of the cluster class loader, the second sample of course does not have the classes from the first sub project.

To run the demo:

Build the project:
```
$ mvn install
```
Cd to the sample directory, compile and start a tc server:
```
$ cd sample
$ mvn package
$ mvn tc:start
```
Start the sample process:
```
$ mvn tc:run
```
In another terminal, cd to the sample2 directory:
```
$ cd sample2
```
Compile, and run the example:
```
$ mvn package
$ mvn tc:run
```

If you did everything correctly, you should see:


[INFO] [node] Waiting for work...
[INFO] [node] This is Callable2 calling!

In the second terminal (sample2). The message printed ("This is Callable2 calling!") is printed by a class that is only present in the classpath of the first instance (sample).

Stupid Simple JVM Coordination

2008-03-17T20:39:00.001-07:00

If you think cross-jvm coordination is easy - then this post is not for you. If it makes you cringe inside, just trying to remember the JMS interfaces, or JGroups api, java.io classes, or figuring out how to mess with a database, then carry on, intrepid reader. This post is for you.

I'm going to show you how stupid simple it is to use Terracotta to send a message from one JVM to the other. We'll use two JVMs - a producer and a consumer. I want the producer to create and send a message to the consumer. I want the producer to wait for the consumer to consume the message. When the message is consumed I want the producer to use the return value from the consumer.

This would be stupid hard if it weren't for two amazing technologies. The first is the java.util.concurrent package. The second is Terracotta JVM Level Clustering. Putting them together gives you stupid simple JVM coordination.

The scenario I outlined is actually ridiculously easy in a single JVM using the java.util.concurrent package. It was built to handle these scenarios and more at the flick of a wrist. Instantiate a queue, fire off a couple of threads, use a FutureTask, and you're done.

And you know what? Could it get any more simple than writing one line of code to cluster that queue, and move from two threads in one JVM to one thread in two JVMs. It can't.

Here's the main method that does it all:


public class Main
{
   public static final Main instance = new Main();

   private AtomicInteger counter = new AtomicInteger(0);
   private BlockingQueue<FutureTask> queue = new LinkedBlockingQueue<FutureTask>();

   public void listen() throws InterruptedException
   {
       while (true) {
           queue.take().run();
       }
   }

   public void run() throws Exception
   {
       if (counter.getAndIncrement() == 0) {
           System.out.println("Waiting...");
           listen();
           return;
       }
     
       FutureTask task = new FutureTask(new MyCallable());
       queue.put(task);
       System.out.println("Task completed at: " + task.get().toString());
   }

   private static class MyCallable implements Callable
   {
        public Object call() throws InterruptedException
        {
            System.out.println(new Date().toString() + ": Sleeping 2 seconds...");
            Thread.sleep(2000);
            System.out.println("Hello world");

            return new Date();
        }
   }

   public static void main(String[] args) throws Exception
   {
       instance.run();
   }
}

And the Terracotta config:


<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">

 <application>
   <dso>
      <instrumented-classes>
        <include>
          <class-expression>Main$MyCallable</class-expression>
        </include>
      </instrumented-classes>
     <roots>
       <root>
         <field-name>Main.instance</field-name>
       </root>
     </roots>
   </dso>
 </application>
</tc:tc-config>

That's all there is to it. Output looks like this:

Node 1:


$ javac *.java
$ start-tc-server &
$ dso-java Main
Waiting...
(after starting other node...)
Mon Mar 17 17:53:45 PDT 2008: Sleeping 2 seconds...
Hello world

Node 2:


$ dso-java Main
Task completed at: Mon Mar 17 17:53:47 PDT 2008

I've actually written this entire example up as a Recipe on Terracotta.org. Full details and instructions are listed there in the FutureTask recipe.

The Trouble With Data Partitioning (cartoon)

2008-03-08T13:33:00.000-08:00

It's the little things that matter...

2008-03-05T14:22:00.000-08:00

It's often the little things in a design that make the biggest difference. Sure you have to get the big things right too, but all too often products suffer from a great idea implemented poorly.

So at Terracotta, I often have conversations along these very lines. The job we've carved out for ourselves - clustering the entirety of the Java Virtual Machine, is pretty big. That's why it's such a great place to work - the challenge we face is enormous, and it's enormously fun to tackle it. Let me tell you right now, clustering the VM itself isn't going to happen if you don't get the big ideas right. I'll wager that we have, but only history can prove that one right. But just as important is getting the little things right.

Today I just happened to discover one of those little things. What is it? Well, if you don't already know, Terracotta maintains Object Identity across a cluster of JVMs. That in itself is an amazing feat (no other piece of technology I have ever run into can do this). So Object Identity is the big thing. What's the little thing?

Here goes.

First, my sample code (Main.java):

public class Main
{
    public static final Main instance = new Main();

    private Map<Object, Object> map = new HashMap<Object, Object>();

    public void run() throws Exception
    {
        Object key = new Object();
        Object value = new Object();

        while (true) {
            synchronized (map) {
                map.put(key, value);
            }
            Thread.sleep(500);
        }
    }

    public static void main(String[] args) throws Exception
    {
        instance.run();
    }
}

Those of you not familiar with Terracotta might wonder what's so interesting about this. Well, with Terracotta, you can cluster any Java object, so with the following bit of config, I have done just that:

tc-config.xml:

<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">

<application>
  <dso>
    <locks>
      <autolock>
         <method-expression>void Main.run(..) </method-expression>
      </autolock>
    </locks>
    <roots>
      <root>
        <field-name>Main.instance</field-name>
      </root>
    </roots>
  </dso>
</application>
</tc:tc-config>

The map in the Main class listed above is now a clustered map (because the root field, instance, holds a reference to it, and therefore transitively it becomes clustered). Anything I put in the map is clustered (transitively again), meaning every object I put in the map is available to all other JVMs in the cluster. That's pretty cool in it's own right (I happen to think), but how is that different from a normal get/put API in a traditional clustered cache, say EHCache, JCS, or OSCache?

Well, I monitored the number of transactions the little test above generated. How many would you guess? 1? 100? 1 every 500ms?

The answer is actually : 1. Because of Object Identity, after the first iteration through the loop, Terracotta knows that it can optimize out the subsequent calls - there is no need for Terracotta to "re-put" an object for a key that already has that same relationship in the map - so it can save the roundtrip work to the server.

In the clustering world, anything you do on the network is orders of magnitude slower than main memory, so every little thing you can do to keep operations local means a big improvement in latency and throughput. So it may be a minor optimization, but it's got a big effect on the latency and throughput of this application. My application may be trivial, but consider if that map was an HTTP Session Context, or a distributed cache.

Furthermore, this optimization is simply not possible with serialization based solutions (which must implement a copy on read, copy on write strategy), because it's simply not possible for a serialization based approach to track object identity, or changes to objects, and optimize out this kind of a scenario.

However because Terracotta is at the VM level, it knows implicitly when objects change, because of Object Identity, so it is not necessary for a caller of the map to "re-put" objects into the map to make sure it's updated (and it's thus valid to eliminate the subsequent put calls that are superfluous). So in the end - Terracotta would work exactly the same with or without this optimization - the correctness is unaffected by it - but with it, it can, depending on your usage, make your application run orders of magnitude faster.

So, in summary, you gotta get the big things right. Object Identity is the big thing. But it's in getting the little things right - for example optimizing away unnecessary network calls by eliminating redundant map.put() calls, that turn out to take a great idea and make it truly impressive.

Note that I can't take credit for this, since I had nothing to do with creating the feature or even suggesting it. I just happened to have realized that it's trivial to test to see if it's implemented or not, and I did test it and hoped that you would find the results interesting too.

To find out more,

Read about Terracotta

Check out some bite-sized code samples in the cookbook section

Or just download it already :)

Note that the code posted in this demo is 100% runnable - just

save it to Main.java and tc-config.xml in a new directory

type "javac Main.java",

start the Terracotta server - start-tc-server.sh&

run the program - dso-java.sh Main

Time for a Maven Best Practice

2008-02-12T15:06:00.000-08:00

I really like Maven. I think the idea is really solid, and I love the outcome of a project that is factored so well that you can have a working demo in an 80k download. That's really powerful! I use this power to my advantage all the time, building demos for Terracotta that can demonstrate with almost no effort how Terracotta gives you a high performance clustered cache, or can give Hibernate a performance boost.

That's awesome stuff, so awesome that I find I just can't code a project with "hard" dependencies anymore. If I can't download your dependency from my pom.xml, sorry man, it doesn't exist.

Which is where the frustration comes in - and a time for a best practice to start. Have you ever come across a new library you want to try out, but cannot find the groupId, the artifactId, or the version that corresponds to the dependency download? For a while I thought I was the only one...If I just knew these three "coordinates" I could get on with the job at hand...Instead I get bogged down hunting through some obscure project website, svn'ing code and the like. Ughh! Turns out some colleagues at Terracotta agreed, and we realized there has to be a better way.

Well, it's not rocket science, but we figured just printing out the info on the site would do. So, I present to you the Terracotta Maven best practice, implemented for your convenience on the Terracotta Forge. Every project published by Terracotta is required to have a "Maven Coordinates" section in the index page.

Terracotta Maven Best Practice

Print the groupId, artifactId, and version of the published artifact on the index of the site in a human readable form
Print the groupId, artifactId, and the version of the published artifact on the index of the site in a pom.xml compatible form

Here's what it looks like in practice:

Stupid JVM Tricks - Read locks from just synchronized

2008-01-22T17:50:00.000-08:00

I bet if you asked someone in an interview what the synchronized block in Java does, they are likely to answer something along the lines of "protected sections" of code, if you are lucky they might answer with a deeper technical understanding, with terminology such as "atomicity," "mutual exclusion," "monitors," and the like.

I will further bet that if you asked if it's possible for Java's synchronized keyword to produce read lock semantics (e.g. concurrent access), you would hopefully get a resounding no. Ok, well, if your interviewee is clever enough, they might tell you that you could build a read lock mechanism, but that synchronized itself only provides you with write lock semantics - that is to say, mutual exclusion.

So if I ask you the very same question - what's your answer?

Mine is ... YES!

Let me show you how. First, here's my code (in a file called Main.java):

public class Main implements Runnable
{
    public static final Main instance = new Main();

    public void run() 
    {
        enterMonitor();
    }

    public synchronized void enterMonitor() 
    {
        System.out.println("I'm in the synchronized block"); System.out.flush();
        try { Thread.currentThread().sleep(5000); } 
        catch (InterruptedException ie) { System.out.println("Interrupted"); }
    }

    public static void main(String[] args) throws Exception
    {
        for (int i = 0; i < 10; i++) {
            new Thread(instance).start(); 
        }
    }
}

And here's my output:


I'm in the synchronized block
I'm in the synchronized block
I'm in the synchronized block
I'm in the synchronized block
I'm in the synchronized block
I'm in the synchronized block
I'm in the synchronized block
I'm in the synchronized block
I'm in the synchronized block
I'm in the synchronized block

(Note that all statements above were printed at the same time, not serially)

Now, I hope you are wondering how I did that. If not, this post really isn't for you. If you are, ok, ok, I admit it. Vanilla synchronized really can't give you read semantics, but with a bit of sleight of hand, Terracotta can do it for you.

The way I made the above output happen is that I configured Terracotta to treat the synchronized block as a read lock. The rest is simple - Terracotta manages the locking, and since it was told to make the particular lock in question a read lock, lo and behold, I was able to get fully concurrent access to the code inside the synchronized block.

I know what you are thinking now - holy synchronized block batman! That could be a disaster for code that wasn't expecting to be run concurrently! And you'd be right - except Terracotta is smarter than that. Let's suppose I had accidentally surrounded some code that made field updates (performing a write) to my object with a synchronized block, and told Terracotta to make that into a read lock. If I do that, and run the code with Terracotta, it will tell me (by throwing an Exception) at the point of modification that I made a mistake and my code was mistakenly trying to make an update to an object under a read lock. In other words, no, using read locks with Terracotta is not dangerous at all, and it can really boost the performance of your app.

Update: To see how simple this is, let me post the relevant bit of Terracotta config that converts the above synchronized into a read lock:

<locks>
  <autolock>
    <method-expression>void Main.enterMonitor()</method-expression>
      <lock-level>read</lock-level>
   </autolock>
</locks>

For all the details, see my full write-up in the Terracotta Cookbook - a really great place to see simple examples that demonstrate the power of Terracotta.

In today's world, what is the role of the Application Server?

2008-01-17T09:31:00.000-08:00

Undoubtedly, you noticed a few changes in the Enterprise world:

1) Sun bought MySQL for a reported $1B

2) Oracle bought BEA for a reported $8.2B

I've already been asking this question to various individuals, with these recent announcements it occurred to me that I'd be interested in what you, the blog reading public have to say.

My question is this:

Given the likes of Spring and Hibernate, how do people consume the Application Server? Is it all you can eat with JEE still delivering on the promise of writing business logic and letting the App server deal with the infrastructure? Or do things like Spring, Hibernate, JPA etc. change the game - are people choosing a best of breed approach, with lightweight frameworks (Spring, ActiveMQ, Hibernate etc) and lightweight containers (Tomcat, Jetty etc) allowing developers to build the stack they need, and no more?

How do the acquisitions mentioned above change the game? Is BEA still delivering value, or is the App server market completely commoditized?

I would love to hear your comments!

Delicious - Just the Tip of the Iceberg

2007-10-31T10:53:00.000-07:00

I had a long conversation with my long time friend James yesterday. We spent some time chatting about a link I sent to him via del.icio.us, which posited that social bookmarking is essentially broken. I like the kind of analysis the author provides - it's nice to see someone thinking about a problem, so I promptly commented on the blog, noted that for future reference, and sent it to James.

Unlike James, I like to write my blogs in narrative form. For me it's just more fun. If you read through his blog you'll see that he spends a lot of time carefully crafting well written, instructive and informative entries. I see the value in it - I write emails that way sometimes (and should do it more often) but my personal preference is to tell a story. My (rambling) point is, if you re-read the entry paragraph, there's actually some interesting information I purposely hid to see if you picked up on it the first time 'round. By now you've found it no doubt, so let's point it out.

I said: "noted that for future reference" and "sent it to James."

Those are actually two distinct actions that I took, and I used del.iciou.us for both. Getting back to the title of this post, that's just the tip of the iceberg. But let's start there, because it's farther than most people take del.iciou.us on any given day.

Action 1: noted that for future reference

I actually stored a reminder for myself on del.iciou.us using a tag called "commented". This is an action tag, it's something I have done. I'll get to the difference between an action tag, and a classification tag in a minute. For now you just need to know that "commented" means that I made a comment somewhere - a blog, a forum, etc.

Action 2: sent it to James

I sent the link to James using the built-in "for:" functionality in del.icio.us. This is the only instance I know of that illustrates del.icio.us thinks of itself more than just a gigantic classification system for the web. More on that subject in a minute. For now, you just need to know that if I tag a link with "for:jameselee", James will see that tag in his "links for you" section in del.iciou.us. Here's are links tagged for me: http://del.icio.us/for/tgautier. What's really handy about this is you can subscribe to an RSS feed in your favorite reader, meaning you don't need to constantly check del.iciou.us for new tags, just let your RSS reader take care of that chore for you. This kind of tag is an actionable tag. Now lets discuss what I'm talking about.

What's a Folksonomy, and why do I care?

First, we need to understand a little bit about what exactly is del.icio.us.

Everyone and their brother that uses del.icio.us today is familiar with using it as a classification system. Looking at the list of popular tags on del.iciou.us, they are primarily of the type that adds meta-data about the target link that relates to what the tagger thinks that link is. That's classification. It's what biologists do when they talk about a certain Monkey belonging in a particular Species, Genus, Family, Order etc. The difference between a biological taxonomy and a tag taxonomy (or more precisely, folksonomy) is that a traditional taxonomy has well-defined rules, definitions and categories to place things. It's often hierarchical, as in the case of biology, and it is rigid and doesn't change much. Folksonomies are flexible, and don't follow well-defined rules, they depend on emergent behavior to derive value. The del.iciou.us popular tags link shows you what people think of the web.

But we can be more rigid about our folksonomy, (there aren't any rules which means I am free to make up my own, right?), in fact what I propose is that there is in fact very strong similarities to these activities of classification, and they all fall into a just one part of a larger taxonomy of folksonomies. That part is the "classifying" part. They aren't adding any data about their relationship to the link (e.g. I am the author, I commented on it, it makes me feel warm, I like it, I hate it, etc.) and they aren't associating any actionable items with those links either.

To put it another way, it seems everyone on del.icio.us is busy classifying data according to what they think it is, because the particular part of the folksonomy they see themselves in is just the "classification" part. It's as if you let a horde of botanists free in a field full of flowers, all of the same Genus, and let them start coming up with Species on their own, no matter who stumbled on what and then tried to sort out all of the data later (the current theory being that a majority of the botanists would converge on a given answer for each specie).

But there are so many other kinds of meta-data that can be associated with a link, that I think del.icio.us, and the folksonomy community at large, is missing out on the big picture. Let me try to illustrate it better:

What else can I do with link meta-data? The bigger picture

Let me suggest a rudimentary "higher-order" taxonomy which is super-imposed on smaller order folksonomies. In this way we can begin to extract some useful, and interesting, new ways to use the meta-data being collected by the masses:


+ Link Meta-Data Taxonomy
  - Classification Folksonomy (e.g. tutorial, howto, tip, webdesign, etc.)
  - Relational Folksonomy (e.g. authored, commented, etc.)
  - Actional Folksonomy (e.g. forterracotta)

Classification Folksonomy

The first one is pretty obvious, it's how everyone uses del.iciou.us today. The goal here is to try to describe what the content of the link is. So if it's a tutorial on web design practices, you might tag the link as "tutorial webdesign".

Relational Folksonomy

Less well known. Describe how you are related to, or see your relationship with, the link. If you wrote it, you might tag it with "didauthor". If you commented on it, you might use "didcomment". James suggested the use of "didxxx" which I like because it tells me right away that this is a relational tag, not a classification tag.

Actional Folksonomy

I've not seen much of these around. We use them for our buzz system on http://www.terracotta.org which I blogged about in detail several months back. The basic strategy here is to use tags that indicate to other tools scraping your data how to view the link. So, in the context of the buzz engine, if I encounter a link on the web that is buzz worthy, I simply tag it "forterracotta" and it pops out on the Terracotta .org site. Similary, I filter important links into my Google homepage via an RSS widget for del.icio.us by tagging them with a combination of "my startpage". This means I want my iGoogle page to show the particular set of links tagged with "my" and "startpage" to show on my Google startpage.

What next?

The title of this post is "the tip of the iceberg". I highly doubt that the taxonomy of folksonomies is limited to just three categories. I'd love to see this idea pushed forward, to find more interesting and creative uses for meta-data about the web. Del.icio.us is a start, but I suspect we will look back on it in 5 years and find it horribly primitive.