Liminal Existence

Velocity

2008-05-12T08:53:00.000-07:00

If you're interested in scalability and performance as it pertains to internet applications, or if you're involved in or looking to build a reliable internet-based system, go to Velocity.

I was fortunate enough to attend the O'Reilly Velocity Summit back in January, which was a Foo-Style event designed to provide some context leading into the actual conference in June. If the conference is one tenth as good as the summit, it will be amazing. Jesse is an amazing organizer and a man with a vision.

Anyhow, instead of reading my off-the-cuff posts about obvious things that are much more fun to talk about while drinking in Miami, go to Velocity and learn this stuff for real.

Scalability

2008-05-11T18:57:00.000-07:00

Updated: Go read Steve's Dynamic Languages Strike Back. It's a longer read, but it's much more interesting, and he's much smarter than I am.

LOL. <-- this is a link. Read Ola's post, first.

For all those who don't get it, languages don't scale, architectures do.

Now, some languages are faster than others. That means that to complete a given operation, it costs less, everything else being equal. Costing less is a good thing. But developers also cost money, so if you have to spend money on developers' time porting from one language to another then you might not be saving any money at all, and really you're just treading water.

Once upon a time, Shell Scripts were used to write CGI applications. With the correct architecture, and enough money, you could build Google with tcsh. No, really. It wouldn't be fun, and you'd be dumb, because there are much cheaper ways to do it. But then again, if you stuck with it, perhaps you'd optimize tcsh to be really fast at spawning and serving up web requests. Faster than Java, faster than <insert your favourite language here>. Faster means cheaper, it doesn't mean more scalable.

I point to exhibit A. Perl used to be slow. Now it beats JoCaml with the bestest concurrency (re: “Scalability”) around. What was Perl built for? Parsing text. Lots of it. All the time. It's fast. Does it mean that you can't build Wide Finder with another language? Absolutely not. Does it mean that you couldn't build Wide Finder to scale out to a trillion documents with gawk? If you answered “yes”, go back to the start of this post and read again! :-) If you're still answering “yes,” try reading some more. Leonard, Ted, Joe, Cal, and Theo are good places to start.

If you answered “no,” congratulations! Pat yourself on the back for knowing what scalability means.

FoWA Miami Rocked.

2008-03-08T06:59:00.000-08:00

I've been busy getting [Twitter] ready for SXSW, and have completely failed at email and, well, everything except work, since then. Before I actually land get drunk in Austin, I wanted to make a quick post about how great FoWA Miami was.

After giving my workshop, I attended Joe's workshop on scalability, which was an amazingly thorough discussion, and I highly recommend attending anything that Joe does in the future (including the panel that he, Cal, and others are on at SxSW. I didn't catch many of the session talks on Friday, as I spent much of my time talking to attendees, but the "Building a Web App in 45 minutes" panel was a fun experiment, and both Cal and Gary were energetic, brilliant, amazing, all that good stuff. If you get a chance to see either of them speak, don't pass it up. Especially Gary, as you're likely to get free wine, even if he makes you eat dirt beforehand.

I had some hiccups during the demo / discussion portion of my workshop, but the first part went well, I think, and I'm looking forward to some great applications that incorporate Jabber soon. My talk, "Bringing your web app to the masses" went well, except for the part where Twitter went down in the middle of it, and I got a call from work (which I waited until after the talk to answer).

Some heckling ensued, but I was happy to be able to address the audience's questions about Twitter in an open and honest way. The atmosphere that Carsonified has managed to foster at FoWA helped a ton.

Mel, Ryan, Lisa, Keir, and Elliot did a fantastic job organizing everything, and Tantek and Brian put together an amazing lineup of speakers and workshops. Seriously inspiring stuff.

If they'll have me, I'll definitely be going to any conferences they hold in the future. They've come a long way since the FoWA San Francisco in Fall 2006, and it looks like they'll just keep getting better.

Google Entaglement

2008-02-21T10:51:00.000-08:00

I've done a fair bit of security work, and generally try to care about the finer details of privacy and security. However, one of the things that I've learned is that more often than not, no amount of digital security past a certain point is going to help, since usually the threat model isn't an advanced technological attack, it's a social one.

Thus far, Google has done a pretty good job of keeping private things private and public things public. I've spoken to people on the Google Reader team, and the main reason they haven't added support for private feeds is their acute concern for privacy.

Today Google announced a limited trial of storing health records online. This seems reasonable and doable in a secure way, but I'm sure they'll get lots of unwarranted flak for the long-awaited project.

However, there will and should be some warranted flak. It turns out that they're using your regular Google account to store this information, and will provide access to it using your regular password, no doubt through yet another Google login page. I've heard concerns that OAuth supports phishing (from Google people), but project infighting and power struggles at Google that result in tens of login pages, all slightly (or dramatically) different, all using the same credentials supports phishing much moreso.

I strongly support patients' rights to access their medical information, and Google is probably one of just a handful of organizations that can do the necessary coordination work and stand up to invasive organizations at scale. However, they need to stop thinking of this data as theirs, because it's not — it's your data. Using the same password as your email to access your health records is something that should be actively discouraged. If Google wants to present a unified interface, they should expose an API and use OAuth or AuthSub, just like any other third party that would consume the data.

Now, I may be over-reacting, but I had an interaction yesterday that suggests to me that I'm not. Someone using GTalk sent a chat request to blaine@twitter.com; this email address has an MX record that resolves to mail.twitter.com, and the corresponding JID resolves to jabber01.twitter.com. However, I have claimed my blaine@twitter.com address on GMail, and associated it with my primary GTalk ID (romeda@gmail.com). When I accepted the chat request, the response came from my GTalk account, romeda@gmail.com.

In effect, Google had done something clever, and in so doing broke the Jabber spec, ignored my own self-hosted Jabber server, and exposed my personal email address without asking my permission.

In this case, it wasn't a big deal, I don't care, etc. Others might, though, and I only knew that it was happening because the person on the other end of the chat was tech-savvy enough to realize what had happened. Also, email addresses and connections between them are hardly closely-guarded secrets. The thing I take away from this is that Google is being sloppy. There's a lot going on, and it's hard to keep track of it all. That your health records are being tied to your Google account just reeks of some power struggle where the Google account people want to bolster their product's internal importance (or have managed to do so that they get veto power where they shouldn't have it), and it's simply not a pragmatic choice. There's a reason your health records aren't stored at the DMV, and it's not out of convenience. Just sayin'.

FoWA Miami

2008-01-20T23:19:00.000-08:00

Just a quick note that I'll be giving a workshop on building real-time web applications using Jabber at the Future of Web Apps in Miami, on February 28th. The conference runs from the 28th to the 1st of March, and should be a lot of fun.

We've been gradually improving the Jabber stack on Twitter, and we're now sending millions of messages every day, doing things that just don't fit into the polling-based world of Atom feeds. There are a ton of extremely awesome things that can be built, and so far we've just scratched the surface.

More to come; if I don't start blogging these things in small pieces, they'll never come.

Stability

2007-09-01T18:39:00.000-07:00

Most of the Twitter team's work in the weeks leading up to launching Blocks was to ensure that it wouldn't fall over as soon as we released it. It's an extremely punishing application, loading 10 timelines on every occasion that someone looks at it. So far, the servers haven't even noticed.

There have been a number of Twitter hiccups in the past few weeks, but they've all been weird, random bugs. Which is not to make excuses, but rather to say that in spite of (very time-consuming) challenges along the way, we've been myopically focused on making the site faster and more reliable. As evidence, here's a graph of page load times, as seen from an external observer:

We're going to keep building a faster and more reliable Twitter. We're also going to add some awesome new features, and soon. Possibly better than contact search and GMail, even! Finally, we'll have more visualizations from the Stamen folks. Britt is off to Berlin for RailsConf mid-September. We'll then have more details about what we're doing to push Rails and Twitter.

These are the people ...

2007-09-01T15:58:00.000-07:00

Folly: "In architecture, a folly is an extravagant, frivolous or fanciful building, designed more for artistic expression than for practicality." – via Tom Coates, by way of Tom Carden.

We just released Twitter Blocks, a nice little visualisation done by the good folks at Stamen Design. It's fun! Go play!

Stamen's recent work highlights the playfulness inherent to Twitter. I can't wait to release more of these interfaces, and hope that it inspires similar work. Sam Ruby, Tim Bray and others have recently weighed in with their long bets. I'm willing to put down that playfulness — of the sort that Stamen, Schulze & Webb and Jane McGonigal explore and invent daily — is so important to who we are as people that the tech world won't be able to ignore it for much longer.

Not exactly a risky bet, but too often the tech industry just ignores these things, so there it is, just for kicks.

SELECT * FROM everything, or why databases are awesome.

2007-06-21T12:44:00.000-07:00

I've just committed a patch to ActiveRecord that prevents a large number of very, very bad queries from hitting your database. Go update your code, ASAP.

We've made some pretty significant progress towards scaling Twitter, and we're now at the point where the majority of requests that hit our site complete in less than 70 ms (mostly API requests), and the really complicated front-end pages that we display complete in less than 160 ms. There are still a lot of hiccups, so the average is higher than that, but we're constantly working on getting it down.

One of the consistent problems we've been facing is errant queries. We've been seeing (off and on) queries like:

SELECT * FROM statuses WHERE user_id = 234223 ORDER BY created_at

If you know anything about relational databases, this is a very bad thing, especially when you have users that have more than 20,000 statuses.

One major downside of having an object-relational mapper is that you don't always control what goes on behind the scenes. In tracking down this problem, first we investigated all our code, and weren't able to find the source of these problems. Switching tactics, we isolated some test cases that replicated the problem and brought out the big guns: print. This pretty quickly brought us to an obscure corner of the ActiveRecord source (three cheers for source code!), where it became apparent that Rails was doing these gigantic loads from the database every time we saved even a single field in a related object. There are a bunch of mitigating circumstances that mean that this bug doesn't get triggered all the time, but it's still really really bad.

Thankfully, the patch ~~will be committed soon~~ has been committed (32 minutes patch-to-commit!), and no-one will have to deal with, as Coda put it: "Arg stabby stab stab stabbity fuck stab" anymore. The fact that no-one noticed really speaks to how freaking awesome relational databases (in our case, MySQL) are these days.

Perhaps underlying all of this is the simple fact that most of the time, ActiveRecord and Rails in general is pretty solid, and Ruby underneath is a fully sound language with which to build high-volume services. Kevin over at PowerSet has more on the topic - they've recently announced that they'll be doing their front-end development in Ruby (up until now, it's just been a glue language internally).

Twitter at RailsConf

2007-05-18T03:30:00.000-07:00

Sadly, I won't be attending RailsConf in Portland this weekend. I'll be finishing up at XTech, followed by a (well-deserved) week relaxing / adventuring in Morocco. Photos will follow.

In the meantime, if you're one of the lucky ~~few~~ many attending RailsConf, Alex and Britt will be speaking 10:45 AM on Sunday. They may or may not being doing a reprise of my Scaling Twitter talk, but I'm sure it will be fantastic in any event.

Wish I were there, have fun all!

Social Software for Robots Slides Up

2007-05-17T01:58:00.000-07:00

I've uploaded the Social Software for Robots slides for the messaging/jabber talk that Kellan and I gave yesterday at XTech 2007.

I think the talk went well, aside from a hiccup getting all the stars aligned to show the visualization that Tom helped us put together. For those who missed it, I'm hoping that Kellan or I or both will give reprises at other upcoming conferences. There's a lot of potential here, and things are finally reaching the point where real-time APIs are not only becoming a reality, but a necessity.

XTech has been great fun so far, and there are a number of talks that I'm looking forward to today and tomorrow. Huge thanks to Edd and everyone else who's helped put it together.

Slides for Scaling Twitter talk, XTech next up.

2007-04-22T12:49:00.000-07:00

The slides from the talk are available on SlideShare. They don't spell out the talk, so if you have questions, please do ask.

I'm really excited about XTech, coming up in a few weeks (May 15th-18th) in Paris. I'll be speaking with Kellan about using Jabber to build Social Software for Robots. Lots of awesome people and talks there, should be a lot of fun.

Scaling Twitter, The Talk.

2007-04-13T12:51:00.000-07:00

Simon Willison linked to an interview with Alex Payne, one of my co-workers on Twitter. This caused a bit of a stir, so apparently there's some interest in our experience scaling Twitter, and Rails.

We've been extremely happy with Rails, and make use of the multitude of helpers that it offers us - like any application on any stack, though, providing fast response times to a (rapidly) growing number of users is a challenge. The solutions are often tightly coupled to the application and its characteristics, and while scaling the most trafficked Rails site in the world, we've run into situations where existing solutions weren't enough.

This process has led us to build a number of tools that help us deal with our load, and just as soon as we find some spare time, we'll be releasing many of them. In the meantime, you can find out first what sorts of challenges we've encountered and solutions we've come up with at my talk at the SDForum Silicon Valley Ruby Conference next weekend (April 21-22nd).

I'll be focusing on ActiveRecord and database optimization, caching, and of course, Messaging. I'll also touch on some areas where we haven't had great successes (yet), and hopefully someone from the audience will shout out that there's some totally obvious and awesome thing that we haven't thought of, and it'll save us weeks of work (no, I'm serious. Does someone want to take bets?).

MapReduce in 36 lines of Ruby

2007-04-01T11:48:00.000-07:00

This has been burning a hole in my head since August, after Joel's post made it blindingly obvious that Ruby is the perfect language for distributed programming. I have some code that properly implements partitioning, etc, but never got around to finishing it sufficiently for a proper release. Here's the core idea; if anyone wants the partitioning code, ping me at romeda@gmail.com. mapreduce_enumerable.rb:


require 'rubygems'
require 'ringy_dingy'
require 'ruby2ruby'

module Enumerable
  def dmap(&block)
    self.each_with_index do |element,idx|
      ring_server.write([:dmap, Process.pid, block.to_ruby, element, idx])
    end

    results = []
    while results.size < self.size
      result, idx = ring_server.take([:dmap, Process.pid, nil, nil]).last(2)
      results[idx] = result
    end

    results
  end

  def ring_server
    return @ring_server if @ring_server

    ringy_dingy = RingyDingy.new nil
    @ring_server = ringy_dingy.ring_server
  end
end

mapreduce_runner.rb:


require 'rubygems'
require 'ruby2ruby'
require 'ringy_dingy'

ringy_dingy = RingyDingy.new nil
ring_server = ringy_dingy.ring_server

loop do
  pid, block, element, idx = ring_server.take([:dmap, nil, nil, nil, nil]).last(4)
  begin
    result = eval(block).call(element)
  rescue Object => err
    result = err
  end
  puts "Got #{result} from #{element} for #{pid}."
  ring_server.write([:dmap, pid, result, idx])
end

From the shell:

$ sudo gem install RingyDingy
$ sudo gem install ruby2ruby
$ ring_server &
$ ruby mapreduce_runner &
$ ruby mapreduce_runner &

From irb:


> require 'mapreduce_enumerable'
> (1..100).to_a.dmap { |v| v * 2 }

Google Code Love

2006-12-22T13:32:00.001-08:00

Google Code Love
Originally uploaded by liminalists.

My Jabber::Simple project, extracted from Twitter's Jabber/IM support, got some love today from Google Code (thanks, eric!)

The Year in Cities, 2006.

2006-12-21T16:06:00.000-08:00

Here's the list of cities I stayed in during the course of 2006. Memes away. I've only included cities in which I spent at least one day and one night. I've preemptively added a few spots I'll be visiting over Christmas, too.

San Francisco, CA
Fresno (FresYES!), CA
Tahoe, CA/NV
Seattle, WA
Vancouver, BC
White Rock, BC
Victoria, BC
Whistler, BC
Reykjavik, Iceland
Ólafsvík, Iceland
Belfast, Northern Ireland
Aberystwyth, Wales
Norwich, East Anglia
London, England
Lewes (Brighton & Hove), East Sussex
Barcelona, Spain
Portland, OR
Grande Prairie, AB
Santa Cruz, CA
Mojave Desert & Las Vegas, CA & NV
Zion National Monument, UT
Tucson, AZ
[Near] Calistoga, CA
Chilliwack, BC

All in all, pretty good, though I'm hoping for maybe somewhere more less-European for 2007. We'll see.

The Weather, by Twitter.

2006-12-20T13:12:00.000-08:00

A few weeks ago, Kellan and I whipped up what is tentatively known as "WeatherBot". It's a simple tool to provide weather updates to Twitter users. Every morning at 7:00 AM and 3:45 PM (local time, that's 07h00 and 15h45 for those of you across the pond), our fleet of WeatherBots will send a weather update to Twitter.

If you follow these bots, you'll receive those updates wherever you normally get your Twitters; IM, Phone, RSS, or just on the web. So far, we have bots for the following cities (links are to the bots' twitter profiles, for easy friending action): Boston, Brighton, Chicago, Helsinki, London, Los Angeles, New York, Paris, Portland, San Francisco, Seattle, Singapore, and Vancouver. If you'd like to see another city, just ask and we'll provide.

The source code is available over on Google Code as Twitter Weather. Kellan and I will be providing follow-up posts (and hopefully talks!) about how we built this, and updates on future improvements, so watch this space, and if you're not already on Twitter, sign up already! We're building an awesome space, and want you to have as much fun as we are.

On Twitter

2006-12-08T11:55:00.000-08:00

Kathy Sierra has a great post, talking about Continuous Partial Attention and Twitter. Before you read my response, go read Kellan's excellent post on the matter.

Twitter is really nothing new, it's just new to the web (and IM/SMS). IRC has been around for ages, and functions in much the same way. Campfire, 37signals' group chat application is a similar application, but more task-focused. So to say that 2006 is the year of the singularity is probably too much of an exaggeration.

I held off on using Twitter for a long time, even though I work at Obvious, because it was too much of an attention drain for me. I have a 5 year old Nokia cell phone, and the SMS experience is one of mind-numbing pain when you receive 15 or 20 SMSes per day (not to mention expensive!). IM, on the other hand, is seamless. Like Kellan, I have verging-on-sub-conscious Growl notifications for IM, and as such I spend less than a second processing each incoming Twitter message.

A phone call, on the other hand, can easily eat half an hour for me, because I don't "do" the phone very often. I find it intimidating, because most of the time the interactions involve long hold times, anti-human menus, horrible bureaucracy, etc. Email is more distracting than IM or Twitter, and so on. Calling friends and family is also a major time commitment, especially if we haven't talked in a while.

I'm more than willing to put up with certain levels of interruption in order to have that sense that I "know" what's going on around me. I have friends who live far away, and whom I don't speak to on the phone, email, or see regularly. Twitter perfectly fills the gap, and allows us to keep each-other "in the loop", bringing us closer overall.

Sure, I could close myself off, and just work all day - but I feel like all these emerging (emergent?) "ambient" communication technologies actually help to make me happier, and feel less alone in our highly abstract and disconnected world. Having "community" is extremely important; far more so than our productivity. Maybe it's just sad that we allow ourselves to live in this paradoxically disconnected online world, "cyberspace" as it were. I like to think of it as just another progression of human adaptation. To say that people are lonelier or happier, more or less fulfilled, busier, or more productive than they were 10, 50, 100, or 500 years ago is a falsehood. We're different, and we're all just trying to make the best of where we find ourselves.

Announcing Jabber::Simple

2006-11-09T12:14:00.000-08:00

Jabber::Simple [src, doc] is a simple (duh!) Ruby library that aims to make the implementation of basic Jabber functionality trivial. It is an extraction of the Jabber support that was added to Twitter, and is released under the GPL by Obvious. A line of code is worth a thousand words, so here is the complete code for sending a simple message to a Jabber user:


jabber = Jabber::Simple.new('rex@friendosaurus.com', 'password')
jabber.deliver("bront@friendosaurus.com", "Hey! I'm thinking of going Vegetarian - Any suggestions?")

Getting incoming messages is just as easy:


jabber.received_messages do |msg|
  puts "#{msg.body}" if msg.type == :chat
end

You can also set your status, and get information about your friends' statuses:


jabber.status(:away, "Eating at the Tree Cafe. I need a ladder.")

jabber.presence_updates do |update|
  friend = update[0]
  presence = update[2]
  puts "#{friend.jid} is #{presence.status}"
end

Installation


sudo gem install xmpp4r-simple

or download the package from RubyForge. Source code is also available, licensed under the GPLv2.

Yet Another Jabber Library?

There are a number of existing Jabber libraries for Ruby (jabber4r, xmpp4r, and Net::XMPP), so why Jabber::Simple? First off, Jabber::Simple does not aim to replicate any core XMPP protocol functionality present in these libraries — in fact, Jabber::Simple depends on xmpp4r and the Jabber::Simple#client and Jabber::Simple#roster methods expose all of xmpp4r's awesome functionality. When I started building in Jabber support for Twitter I'd used various Jabber clients, and even set up a simple Jabber server. Writing my own client, however, was a bit more complex. It turns out that the seamless experience of "adding a friend" and chatting with them is (unsurprisingly) comprised of a series of disjoint steps, and fraught with the peril of threads, XML streams, and arcane magic. The available libraries handle these tasks and many more admirably, but lack in elegance. My hope is that Jabber::Simple provides a sufficiently obvious interface with which to develop tools that use the Jabber protocol.

But Wait! There's More!

Now, you might shy away from writing that really cool chat-bot you've been meaning to write, saying "Wow, this is great, but setting up a Jabber server is a pain." --- but fear not! Go over to Google Talk and sign up for an account. Once you're done, use your Google Talk username and password, and start Jabbering. No really, it's that simple.


jabber = Jabber::Simple.new("you@gmail.com", "password")

assert_before

2006-11-06T23:21:00.000-08:00

I'm working on a Ruby Jabber client library to hide the numerous machinations and relatively steep learning curve that xmpp4r requires. It's nearly done, and I'm writing tests (it's an extraction from Twitter's Jabber support). A problem I was running into was the latency involved with Jabber requests; sending a message is almost instantaneous, but always requires at least a one second sleep before the assertion, sometimes much more (roster updates can be time consuming). I started sprinkling five to ten second sleeps throughout my code, but all of a sudden my tests were taking upwards of a minute to run, and there are only five tests! Here's a snippet that should help with all your variable-latency tests:


def assert_before(seconds, &block)
  error = nil
  begin
    Timeout::timeout(seconds) {
      begin
        yield
      rescue => e
        error = e
        sleep 0.5
        retry
      end
    }
  rescue Timeout::Error
    raise error
  end
end

use it in your tests like so:


def test_something_time_consuming_should_succeed_in_at_least_10_seconds
  assert_before 10.seconds do
    assert true, time_consuming_task()
  end
end

and as soon as your tests pass, the block will exit and continue on, completing your tests as quickly as possible.

Borat, We Like.

2006-11-06T20:26:00.000-08:00

I still haven't seen it, but here's what the Guardian has to say:

While there had been much early enthusiasm for the film following festival screenings, there were fears that the appetite for a bumbling anti-Semitic, pro-incest fake Kazakh who engages in an extended bout of vigorous nude wrestling with his producer might not be mainstream enough to translate to the box office.

Huh? How could there not be enough of an appetite for that?!

The Knife

2006-11-03T22:45:00.000-08:00

After seeing this video, I'm totally bummed about missing The Knife. Ray's been talking about it for weeks, but unfortunately they don't do many shows and tickets were selling for (at least) $200. Anyhow, for your viewing enjoyment:

.irbrc

2006-11-03T21:40:00.000-08:00

irb is a wonderful tool; with a few tweaks and additions, though, it becomes essential. I spent too long getting by with just readline support, but today, finally, I added a few of the extensions that I've seen floating around on various ruby blogs. Now my tab completion and history run in overdrive, and the number of extra terminal windows I keep open has been reduced greatly. A quick summary of the features present:

Full readline support and tab completion
Copy-friendly prompt
Auto-indentation
Persistent history
Object#local_methods, to view methods unique to an object
Colourized output (via wirble)
Method finder. e.g., "hello".what?(5) #=> [:length, :size]
Colourized inline ri support (either Object#ri(*methods) or ri Object, works best with Ruby 1.8.5's much improved ri
Output spooling, via more, less or most. e.g., less { puts really_long_string }
Simple regular expression helper (/an/.show_match("banana"))
Textmate and Vi launching

You can get my .irbrc here: ~lattice/.irbc.

MacPorts, a PSA

2006-11-03T12:58:00.000-08:00

When upgrading your ports on OS X, do not use the following command:


port upgrade all

It doesn't do what you expect. Unless, of course, you want TeXmacs and SimGear. And five hundred of their closest friends. Do this, instead:


port upgrade outdated