The Making of MarkMail

A Tale of Two Search Engines

2008-07-18T00:44:00.000-07:00

If you're local to the Bay Area, you may be interested in attending an upcoming talk from the SDForum Software Architecture & Modeling SIG on August 27th. It's titled A Tale of Two Search Engines and will be given by our own John Mitchell, one of the developers on MarkMail. Here's his abstract:

Betwixt the rigid structure of relational databases and the unbridled chaos of random content lies the world of search engines. Search engines shine in the middle ground where the messy complexity of reality makes everything harder than we imagine.

While the soap operas of general-purpose search engines dominate the news, specialized search engines are coming to dominate their vertical niches. Special-purpose search engines can aggressively leverage domain-specific intelligence to return highly relevant results.

This talk will present the architecture, implementation, and stories behind the creation of two specialized search engines for code and email: Krugle and MarkMail.

If you're interested in MarkMail I think you'll enjoy it. (And no, John doesn't really use words like "betwixt" in daily conversations.)

The Perl Review: Now with Video

2008-07-02T18:15:00.000-07:00

The folks at The Perl Review recently enhanced the interview I mentioned here Monday with a new screencast video showing MarkMail in action. The intro is terrific. There's a guy hitting his Mac with a hammer!

It's a strange (happy) feeling to have others produce advertising videos for you. Thanks, brian d foy!

Interview with The Perl Review

2008-06-30T12:20:00.000-07:00

The Perl Review, a quarterly newsletter about all things Perl, recently published an interview with us where we discuss several topics relating to MarkMail:

How we load mail
Our choice between Java and Perl
Our model of permalinking
Comparative community sizes
What's in store for the future
How this is different than Google

It's a more technical interview than some of the ones we've done previously with Apache, The Content Wrangler, and InfoQ.

Diacritics, or should I say dịẫçritícs

2008-06-10T13:59:00.000-07:00

We changed our indexing this week regarding how we handle diacritics -- those accent marks you see on vowels and some consonants in many languages.

Previously we resolved all queries in a diacritic insensitive manner. That meant that a search for "francois" would match both "francois" and "françois", and a search for "françois" would do the same. Basically we specified in our MarkLogic Server configuration that the c versus ç difference be ignored.

Now we've changed the configuration so the diacritic sensitivity choice depends on the search term. A term containing diacritics will trigger a diacritic sensitive match, while a term without diacritics will remain diacritic insensitive. That means a search for "francois" will match with and without diacritics (the same as before), but a search for "françois" will respect the ç character constraint and won't match "francois" anymore.
To summarize: If you care enough to type a diacritic, we'll care enough to match it for you!
This is a particularly helpful change as we've expanded from English-only content into lists written in Japanese, Vietnamese, Spanish, German, Italian, Dutch, Portuguese, Slovak, Polish, and Farsi. We even have one mail in Frisian. Who knew!

On a per-message basis, we get more traffic from these lists than our English lists. Perhaps they're underserved by other email archive systems? Maybe the other systems have issues hosting messages with the non-ASCII characters. We've definitely had trouble finding "clean" historical archive records for non-English lists, ones where the diacritics were reliably preserved. Luckily for us, being built on XML, we have native support for all Unicode characters.

We hope you find the new indexing logic helpful.

Loaded TAXACOM: A List about Biodiversity

2008-05-21T17:17:00.000-07:00

A couple days ago we received a request from Roderic Page to load TAXACOM. He describes it as:

...a mailing list that dates back to the early '90's, and is a forum for taxonomists and other researchers interested in biodiversity. It is lively, with some long conversations. It's also featured as the source for sociological research, such as "Systematics as Cyberscience: Computers, Change, and Continuity in Science". Given interest in the Encyclopedia of Life (see also Ed Wilson's TED talk ), which could be viewed as one response to the issues raised on TAXACOM posts over the years, I think it would be a very timely addition to MarkMail.

With a description like that, how could we resist! So I'm happy to say we've loaded the list, and (for trivia buffs) it even sets a new earliest list record in MarkMail, with archives starting back in 1992.

For more, see Rod's blog and email announcements.

Loaded OpenMoko: An Open Source Smartphone Platform

2008-05-19T20:41:00.000-07:00

Last week we received on our feedback form a request to load the OpenMoko mailing lists. These folks are creating an open source smartphone platform, very cool stuff. Along with the request, the requester explained the benefit he saw in having MarkMail archive the OpenMoko lists, from the perspective of a project participant. I've reprinted it here with permission:

I would LOVE to see the OpenMoko lists get into MarkMail.. for 2 reasons..

1. From a developer perspective, I'm new to the OpenMoko platform and still learning the build system, etc. and am eager to start writing my own applications. But there's only so much info on the wiki and like many young communities all the juicy info is buried in the Mailing lists. So I'd love to be able to search all the lists for things like installing the sim card, what new hardware bugs they've found on the dev list, how to modify the dialer application, etc. This is where MarkMail really shines and is the best platform out there for this type of information gathering from community lists. If these lists were in MarkMail it would be one of the ONLY places one could find some of this information because of the advanced search functions in MarkMail. I think this holds true for a lot of young open source communities and MarkMail can really help out.

2. I think it would help the community in general by giving users an avenue to find the information they need to start better participating and contributing back. There would be less duplication of questions which distracts everyone on the list and hopefully more "I see how things are being done because I got all caught up by searching MarkMail, how about we do it this way.." etc.
Of course we loaded the lists for him.  Here's the activity chart:

Loaded Perforce: High-End Revision Control

2008-05-12T11:38:00.000-07:00

Recently we loaded seven mailing lists dedicated to the Perforce SCM system. If you haven't heard of Perforce, they're a high-end revision control system, with a long list of corporate clients. They're known for speed and features.

I've been using a Perforce system to manage my own files for over a decade now, appreciating their free individual license.

Their mailing lists have a lot of technical Q&A discussion, so I hope having these lists more easily searched will help people find the answers they need. Here's the historic traffic pattern:

Loaded Eclipse and NetcoolUsers

2008-05-01T17:08:00.000-07:00

Yesterday we loaded the email archive histories for two new communities: Eclipse and NetcoolUsers. Normally I wouldn't talk about these two communities in the same blog post, but after the load it occurred to me that both projects (coincidentally) relate to IBM. More about that at the end.

Eclipse (eclipse.org) is an extremely popular open source development tool project, initiated by IBM back in 2001. (The name was widely seen as an attack on Sun.) It took off and lots of Java developers use it as their IDE. They have a beautiful growth chart:

NetcoolUsers (netcoolusers.org) is a user community focused on IBM Tivoli Netcool. For a single list it's quite hopping (25 posts/day):

The fact both these communities relate to IBM is purely coincidental, but it's also interesting because it reflects the direction of pull we're seeing from the MarkMail user base. It's an early sign of what you can expect in the future: more technical content beyond open source.

Technical lists come in many flavors: pure open source (Apache, JDOM), corporate-sponsored open source (Eclipse, Xen), standards development (W3), technical user groups (NANOG), and groups focused on proprietary technology (NetcoolUsers). We plan to expand along each of these axes.

If you have a list you'd like us to load, let us hear about it.

Loaded NANOG: North American Network Operators' Group

2008-04-29T13:49:00.000-07:00

Today we loaded a new list, NANOG, a discussion forum for the North American Network Operators' Group. In its 100,000 messages it holds some fascinating discussions about internet operations. The chatter around 9/11, Katrina, and y2k stand out especially.

The list extends back to April 1994, two months earlier than any list we previously loaded. It's always fun to break little records like that. It could be a while before we break this one again.

If you're intrigued by internet operations, the NANOG FAQ has lots of good factoids.

Loaded Python: A Cool Million Messages

2008-04-15T17:49:00.000-07:00

Happy news: We've just finished loading the Python Software Foundation mailing lists. (Python is a popular programming language, overseen by the PSF.) With this load we're breaking a few records:

Weighing in at 1,022,479 total messages, Python is now the largest community ever loaded since our initial launch. (We went live back in November with roughly 4 million Apache messages.)
Half of those million mails are from a single list, python-list. That means python-list holds the new record for Crazy Huge What The Heck Can They Talk About So Much list. (And, would you believe, there's even more python-list histories from 1992-1995 still to load.)
This puts our total combined MarkMail message count above 10,000,000. There was much hooting and hollering (and page refreshing) around here as the numbers clicked over.
It's the biggest community ever loaded by our new hire Evan Paull. OK, it's the only community ever loaded by Evan. He started just a couple weeks ago. We figure after he's wrangled together a million message history, everything else will look easy.

Among the million mails are the archives for the Mailman project, something I'm especially happy about because much of our work here involves interfacing with Mailman, and this should help us understand it better.

As always, here's the traffic chart:

Loaded GNOME: 750,000 emails

2008-04-07T15:22:00.000-07:00

Over the weekend we loaded the mailing list history for the GNOME project. GNOME is a immensely popular GNU project, a free software desktop environment and development framework. Their message traffic shows it has a vibrant and active community. They boast a history of 750,000 emails across more than 200 lists:

The peak in 2007? That's because in 2007 they started a new svn-commits-list (a list that captures emails about code check-ins) and it's been archived while the older cvs-commits-list wasn't. If we add -type:checkins to the query, we can graph the history without that list:

It took a fair amount of work to load the GNOME history because the archives had more spam and virus mails than could suitably be removed by hand. We had to use procmail and SpamAssassin to remove the junk.

One neat factoid: It's easier to remove spam from mail sent in 2004 than mail sent today. Spam blocking has always been a competitive arms race, but in this case we're fighting yesterday's war with today's technology! Even running in offline mode, SpamAssassin did a darn fine job.

I just wish it ran faster. If anyone out there is a SpamAssassin performance guru, please let us know.

Our thanks to Jeff Waugh for helping us get the histories.

World Wide Web Consortium Lists: 400,000 emails

2008-03-17T20:52:00.000-07:00

HTML 4.0, XML, PNG, CSS, DOM, and XQuery: These are but a few of the technologies to come out of the World Wide Web Consortium, commonly referred to as the W3C. We're proud to announce that MarkMail (which by the way uses all of those technologies!) has loaded the full W3C public mailing lists. They start in 1994 and cover 400,000 emails across 200 mailing lists.

With such a long and deep history it's fun to do a little archaeology: You can find the first mention of XML back in 1996. I tried to find the formal "XML 1.0" announcement and saw there wasn't one, but on launch day (February 10, 1998) you can find people complaining about rendering issues with the spec. Isn't that always the way with mailing lists? By the way, it's fun to use XML to search on the birth of XML.

Google first came up as a topic in August 1998, back when its domain ended "stanford.edu". That beats any other list by 5 months. The first mention of XQuery didn't come until January 2001, well after xml-dev and other lists were talking about it. I expect there's more chatter in the private W3C archives.

Finally, the first mention of MarkMail came in December 2007. And what a great post it was! :)

Loaded Perl: 530,000 emails

2008-03-13T01:04:00.000-07:00

Perl is the duct tape of the internet. Created by Larry Wall in 1987 and made famous with his Programming Perl "camel book" published by O'Reilly, it's the tool sysadmins use to keep things running.

We're proud to announce we've finished loading the Perl.org mailing list history into MarkMail. A total of 530,000 emails across 75 lists. The lists don't go back to 1987 (boy that'd be cool if they did). But that's all right; who really needs tech support against Perl 1.000?

What we have here is traffic starting with the migration to the Perl.org setup in 1999:

Enjoy! And if anyone has earlier archives, let us know.

New Search Feature: "opt:nostem"

2008-03-11T22:13:00.000-07:00

In the science of Information Retrieval there's a constant tug of war between precision and recall. As Wikipedia defines the terms, precision is the fraction of the documents retrieved that are relevant to the user's information need, and recall is the fraction of the documents that are relevant to the query that are successfully retrieved. Or as I define the terms, precision is how much of what you wanted you actually got, and recall is how much of what you got is what you wanted.

MarkMail increases recall by running stemmed searches. This loosens the query constraint so that searching for proxies will match proxy as well. Sometimes this is good, and sometimes we hear from users who don't like the behavior all that much! They want more precision.

So we're happy to announce a new feature, opt:nostem, that when added to the search string turns off stemming for that query. You can try it for yourself:

http://markmail.org/search/?q=proxies
http://markmail.org/search/?q=proxies+opt%3Anostem

Average Load Time: 0.1 Seconds

2008-03-07T11:41:00.000-08:00

There are many challenges in running a high-traffic web site. Performance is a challenge we particularly focus on at MarkMail because users get frustrated if they have to wait more than a second for a reply.

The challenge in maintaining performance increases as more of a site's content gets built dynamically -- meaning on the fly in response to user requests rather than ahead of time where it can be directly served (like a McDonalds hamburger).

With MarkMail we build every page dynamically using XQuery. Even a page that at first blush seems as if it could be pre-built, like an individual email message, we actually build dynamically because we want to highlight the search terms from your query.

All this is why I was so happy to notice that Alexa.com calls us a "Very Fast" site...

Markmail.org has a traffic rank of: 128,666 (UP 745,248)
Speed: Very Fast (99% of sites are slower), Avg Load Time: 0.1 Secs

Here's some background on how Alexa tracks performance.

New Feature: Top 10 expands to Top 100

2008-02-27T23:57:00.000-08:00

Every time you do a search on MarkMail the leftmost pane shows you the top 10 lists, senders, attachments, and message types for all emails matching your query. OK, it's not always 10 that you see. Sometimes it's more, sometimes less. Exactly how many you see depends on your browser size. But even if you're the proud owner of one of those new 17" MacBook Pro laptops with the 1920x1200 screen, the view maxes out around 25.

We've added a new feature to help improve this. When there are more values than will fit in the selection box, you'll see a "View more" link in the top right corner.

Clicking on "View more" shows the top 100 in an overlay.

Clicking on any of the values in the overlay will limit your search, same as clicking on a value in the short list. Enjoy!

A Place for Xen

2008-02-25T21:06:00.000-08:00

At MarkMail you can now find your Zen. Or, to be more accurate, you can find your Xen.

Xen is an open source "hypervisor" (similar to VMWare) that enables operating system virtualization. It's supported by Citrix and used by Amazon EC2, among others.

I can joke about finding Xen at MarkMail because we recently loaded a bit over 100,000 messages from the Xen community. If you're into virtualization, enjoy!

If you want to compare VMWare with Xen, you'll find some good discussion in the archive.

PostgreSQL: More Traffic than MySQL (and a first Google spotting)

2008-02-20T06:00:00.000-08:00

When we announced back in December we'd loaded the MySQL database mailing lists, we heard from several people who asked us to load the PostgreSQL lists also. We said we'd be happy to, and MarkMail now has 635,000 PostgreSQL emails loaded and searchable.

Comparing PostgreSQL and MySQL is kind of interesting. With all the talk about the LAMP (Linux/Apache/MySQL/PHP-Perl-Python) architecture you'd think MySQL had a lock on the open source database market, but based on simple message traffic analytics, PostgreSQL has a much higher level of community involvement. Looking at January 2000 onward, the MySQL lists have amassed 340,000 messages with about 3,000 new messages each month:

In the same time period, the PostgreSQL lists have hit 583,000 messages with 7,000 new each month:

I wouldn't have thought it, but there it is.

Also in the PostgreSQL lists we find the very first mention of Google in all of the messages loaded so far! The first Google sighting was on the pgsql-interfaces list, January 28, 1999, in a post by James Thomson:

"I've been using the Oracle Pro*C precompiler manual. I don't have the URL here at work but I found an online copy using www.google.com"

The first mention in another community happened on the xml-dev list a couple months later, March 10, 1999, in a post by Andrew McNaughton:

"You need a new search engine. I've recently been using www.google.com with results an order of magnitude better than what I got from altavista (though altavista still has it's place for more complex query definitions)."

Here's the query if you want to look for yourself:

http://markmail.org/search/google+order:df

Maybe as we load more community archives we'll get even earlier sightings.

Announcing an Informal Partnership with Codehaus

2008-02-13T16:17:00.000-08:00

We're happy to announce we've developed an informal partnership with Codehaus to load all their mail archives and receive automatic notification of new Codehaus lists as they get created.

The automatic update is particularly important because Codehaus is a fast-growing home to open source projects with new lists being created all the time. How fast is Codehaus growing? Looking at the traffic chart, it shows a beautiful upward trend line. For comparison, it has the same level of activity as Apache had in late 2000.

Previously we loaded the Groovy, Mule, and XFire archives from Codehaus. We now have the archives from Grails, Castor, Mojo, JRuby, Plexus, PicoContainer, Cargo, Drools, OpenEJB, and XStream as well as almost a hundred more. In total we're archiving 400,000 emails across the Codehaus lists.

P.S. Curious what happened in May 2006? They had some ISP troubles that month. And of course the last month is short since February has only just begun.

Squid Cache: Searching our own Dog Food

2008-02-13T13:19:00.000-08:00

Yesterday we loaded 115,000 messages from the Squid mailing lists. We're particularly pleased about this because Squid plays a prominent role in the MarkMail site architecture and we plan to use these searchable archives to help with our own development.

Squid is probably the most famous caching proxy out there. It's been around for years, is fire-tested, and has oodles of configuration options. At MarkMail we use Squid as our "reverse proxy cache". In case you're not familiar with that term, let me explain.

On the web a "proxy" is a piece of software that sits between the user and the web server. When a user wants a web page, the user makes the request to the proxy and the proxy makes the request to the web server. Simple proxies provide a means to poke through firewalls, mask user identity, and things like that.

A "caching proxy" is a proxy that remembers the traffic passing through it, so later requests for the same content can (subject to configurable rules) be delivered to the user without actually connecting to the destination server. Schools, companies, and even countries use caching proxies to reduce their bandwidth costs and speed their users' web browsing. For example, once any user has pulled a logo image from a web site, every other user at that organization can just pull the proxy's version. Caching proxies make web browsing better, faster, and cheaper.

A "reverse proxy cache" is a caching proxy that runs on the server-side instead of the client-side. It gets first crack at each user request. In many cases, like when the requested page is already in its cache, a reverse proxy cache can handle the user request on its own and reduce the load on the actual web server.

On MarkMail, Squid sits in front of our MarkLogic Server instance (our web server) and gets first crack at all user requests. It handles several tasks:

URL rewriting. This lets us present friendly URLs like /message/xyzzy to our users, while we actually serve the content from a .xqy XQuery page in the MarkLogic Server back-end. We use a Squid plug-in called Squirm for this. It lets us map public URLs to private URLs.
Caching. Almost every page in MarkMail is dynamic, even the home page with all those count statistics, but that doesn't mean we should regenerate the page on every request. We let Squid cache the results of each page for a few minutes. If you're looking at a page that anyone else saw recently, we're probably serving it to you from cache.
Connection pooling. On the web there's a feature called Keep-Alive that lets users hold open connections to the web server in case they make later requests. A common Keep-Alive period is 30 seconds. Keep-Alive saves the cost of opening up new connections but holding all the open connections can be resource intensive for a web server. By using Squid, we let Squid hold all the Keep-Alive connections to end users (hundreds of connections) while MarkLogic Server only talks to Squid. This reduces the load on the actual web server, leaving it free to focus its energy on searching, rendering, counting, etc.

Of course there's more we'd like Squid to do for us. We'd like some help in blocking abusive users, automatically gzipping content, and things like that. We'll probably look for those features in a new load balancer. More on that later.

In the meanwhile, hope you enjoy the Squid archives.

New Feature: Sweep the Chart to Select a Date Range

2008-02-09T11:14:00.000-08:00

People often write us saying they want to click or sweep on the chart to select a date range. We're happy to announce that's now possible.

To demonstrate, if you search for JavaOne you see a repeating yearly spike which correlates to the dates of the annual Java developer conference. Lets say you want to investigate what people said in just the last few years about the show. You can click and swipe your mouse over the time period of interest:

This adds a date: constraint to the query and automatically updates the search results. You can remove the date constraint by clicking on the "Remove date refinements" link in the top right of the graph.

You can sweep to select or click on individual months, and if you hold down control (command on a Mac) it toggles the selection, enabling you to create non-contiguous selections.

Saxon: Loaded 10,000 emails about XSLT and XQuery

2008-02-04T17:11:00.000-08:00

We've recently begun archiving the saxon-help mailing list, with its 10,000 emails about the famous XSLT and XQuery processor written and maintained by Michael Kay.

Michael's a great guy, in person and online, and he writes long detailed emails answering people's questions. He stays considerate even when the receiver is being a little "dense", ignoring people's help and exasperating the others who try to help out. A recent quote from Michael on the xquery-talk list:

I've spent five or ten minutes writing this response in the hope that you will learn from it and not make the same mistake again, which will save everyone time in the future. If you come back with another query showing the same error in a week's time, I shall give up.

I hope that by making the saxon-help archives more easily searchable than the built-in SourceForge search we'll be able to save him and the readers even more time.

Extra tidbit: If you admin a project on SourceForge and want your archives in MarkMail, there's an easy way to work with SourceForge to make that happen. Just let us know.

Give us a Date, and We'll Search It

2008-01-29T18:33:00.000-08:00

In MarkMail we like to have both a search box way to do something (easy for experts) and a graphical way to do something (easy for novices). Recently we added support for date-based query constraints. At the moment it's only available in the search box, but we thought it worth talking about anyway. A graphical version will be coming soon (we know, we can't wait to click and swipe the months on the chart either).

With the new feature you specify a date or date range by adding a date: term to a query. For example, lets say you'd like to investigate the cause of the sudden spike in messages from the PHP lists for the query "register globals". You see this histogram:

So let's satisfy that curiosity. In April of 2002 things really start to heat up and it remains a pretty hot topic until roughly November 2003. To restrict our query to these dates all we have to do is add "date:2002/04-2003/11" to our register globals query, yielding "register globals date:2002/04-2003/11". Two dates separated by a hyphen indicate a range. This gives you only the matching messages from April 2002 through November 2003. The chart even highlights the selection range:

Looking at the statistics for this query, most of the messages were posted to the discuss list so it's likely that users are having problems. We might assume that something changed with the language so let's add "release" to the query. Sure enough, looks like they changed the default behavior of register globals in the 4.2.0 release which was released on April 22nd, 2002.

We support more than just date ranges. Here's a just a few of the formats that we support:

date:today	Messages posted today
date:"last week"	Messages posted in the last 7 days
date:"last month"	Messages posted in the last 30 days
date:lastmonth	Spaces are optional, for convenience
date:2008/01/26	Selection by day
date:20080126	Slashes are optional, if you prefer
date:2008/01	Selection by month
date:2007	Selection by year
date:2005/06-	Everything from June 2005 onward, because of the trailing hyphen
date:-2005/06	Everything up to the end of June 2005, because of the leading hyphen
date:2007-2008	This year and last
date:"July 4th, 2007"	Human readable formats are supported too
date:"Julio 4th, 2007"	For all of you Spanish speakers
date:t90d	Messages from the last 90 days, don't forget the t
-date:2008	Negation is also allowed, put the hyphen in front of the date:

Don't worry if you can't remember all this. The question mark graphic next to the search box will pop up a reminder. So now while we continue to work on allowing you to interact with the graphs, give this a spin and let us know what you think.

P.S. Jason really likes this because it lets him examine all the changes between JDOM 1.0 and JDOM 1.1.

Ruby vs Groovy: What Can List Traffic Tell Us?

2008-01-27T13:13:00.000-08:00

Over the weekend we loaded the main Ruby lists from ruby-lang.org, about 300,000 messages across the last six years. The ruby-talk list alone weighs in at 245,000 messages and is our new second place traffic champ, trailing only php-general.

With both the Ruby and Groovy archives loaded, we have an exciting opportunity to compare the two communities. In my experience talking with developers at Java conferences, they often look to both Ruby and Groovy as possible next languages to learn. The Java developers have a natural desire to go toward Groovy because it lets them keep their Java stack, but they're concerned about Groovy's level of support relative to Ruby, which has been around much longer and has a larger community.

How much larger? Is the community growing or shrinking? It can be hard to tell with open source, having no revenue numbers and with download counts skewed by bundling. I think looking at email list traffic patterns are about as good a gauge as anything.

Below you'll see the a composite graphic showing the traffic from the five Ruby lists compared to the five Groovy lists, with the Groovy lists inlaid at matching scale:

The Ruby lists are more active, by about double. In the months before the Groovy 1.0 launch in January 2007, the spread was even larger. Both communities seem to have plateaued in 2007. I look forward to seeing what 2008 brings.

P.S. The Ruby lists are half English and half Japanese. Yukihiro (Matz) Matsumoto who created Ruby is Japanese, and the language first took off in Japan. If you speak Japanese, feel free to search for Japanese words. It should work but do let us know if you spot any issues.

Groovy: Traffic Doubled with a Formal Release

2008-01-24T17:29:00.000-08:00

A few days ago we loaded the Groovy lists and their 70,000 messages. The list traffic chart helped change my mind about the language.

The groovy project calls itself, "an agile and dynamic language for the Java Virtual Machine". I'd call it a cool scripting language that compiles to Java bytecodes and so lets you write in a scripting language while accessing the vast set of Java libraries out there.

The first time I saw Groovy, years back, I got very excited -- but then it didn't seem to be catching on, and I thought it was slowly on the downturn. In fact it's not like that at all, it just takes time to develop a language. Look at the shape of the message traffic histogram:

Looks like the project caught some fire. Guillaume Laforge blogged that the big jump you see here starting in January 2007 was due to the release of Groovy 1.0. I see they're on Groovy 1.5 now, as of a month ago. The 100+ messages per day rate will probably continue. My friend Scott Davis was right.