Sriram Krishnan

At the Web 2.0 Expo from the 22nd to the 25th

2008-04-21T09:34:00.000-07:00

I'll be at the Web 2.0 Expo at SF from the 22nd to 25th. If you're nearby and want to meet up, drop me a line at sriramk@microsoft.com/mail@sriramkrishnan.com

Yet another Flickr change, yet another mutiny

2008-04-10T20:56:00.001-07:00

Flickr related change - check
User revolt with protest groups - check
A blog post from Zooomr CEO Thomas Hawk, supporting the protest - check

Sometimes, I feel really sorry for Flickr. They can't seem to make *any* change without drawing the ire of some really vocal users. Here are some incidents from the top off my head

- Flickr getting acquired by Yahoo
- Flickr moving to using Yahoo's login system
- Flickr potentially getting acquired by Microsoft

Now, I love Flickr. I think Stewart Butterfield and Caterina Fake have created something wonderful. I've gifted Pro accounts to friends and I have some valuable photos up there. But I think there's a line between 'being vocal fans' and 'harming the site's progress'. It's almost as if a lot of people have a frozen mental image of Flickr as it was when they first joined it and want to preserve that site forever, at the cost of the site growing.

"But shouldn't Flickr stick to what it does best? Photos?"

No.

Let me illustrate with a short history lesson from Microsoft's past. Microsoft was originally a languages and tools company - BillG wrote the first BASIC interpreter. If it had stuck to just doing one thing well - languages, it would have never built Windows and Office. Similarly, a decade ago, Microsoft was primarily known as a consumer company and didn't have a credible presence on the backend. SQL Server, Exchange, IIS and the Windows Server products changed all that. At each point in time, if Microsoft had stuck to what users thought it did best, it wouldn't have grown.

In my view, Flickr users are doing Flickr a disservice if they want to shoehorn into an online photo site. I would rather see Flickr evolve beyond photos into an friendly online forum where people post creations, regardless of their nature. Flickr's management seems to understand this well - the 90-second limit on the videos is a master-stroke. It stops people turning Flickr into another Youtube and forces them to some extent to post original content.

The Thomas Hawk Affair

Thomas Hawk's involvement in each of these protests makes me a bit uncomfortable since I'm a fan of his photography and enjoy reading his blog. At the end of the day, he is the CEO of a company that competes with Flickr. For him to lead every Flickr user protest and claim that there is no conflict of interest is a bit...stretched. A friend compared this to Tom Anderson (hypothetically) protesting Facebook's beacon implementation. Even though Thomas' arguments are probably being made on good faith, the fact remains that he stands to gain from Flickr users leaving to join his service.

Popfly - now using Cacheman

2008-03-30T02:13:00.000-07:00

John beat me to the post. Popfly is now running on top of Cacheman and we're seeing anywhere between a 2x-6x perf improvement depending on the mashup you're running. This is also the first time Cacheman has been put to real production usage and so far, it has been handling all of Popfly's traffic without breaking a sweat.

This has to be the first time one of my pet projects has proven remotely useful so I'm pretty pleased :-)

Slicehost

2008-03-23T08:31:00.001-07:00

I spent a lot of my weekend moving my website from Dreamhost to Slicehost. Slicehost is a VPS provider with a platform based on Xen. This was not really due to any problem with Dreamhost (they're really good and I would recommend them instantly) but more due to how good Slicehost was. In fact, I have to say that Slicehost, at this early stage, is by far the best host I've worked with.

Shared hosts vs Virtual Private Servers vs dedicated servers

A quick primer on shared hosts vs VPS vs dedicated servers (most of you can probably skip this section - but you'll be surprised how many people confuse these).

A shared host, like the name implies, means that you get a machine that is shared. Your website will run on the same box as several other websites with some security ACLs in place to make sure you don't trample over someone else's files. Depending on the host, your access to the machine might range from just FTPing files over to a shell account where you can log in and run programs yourself. However, you'll never have root/administrator access to the box and you'll usually need to go through cPanel/some-really-ugly administrative control panel to request changes to your configuration or setting up new software. However, these security settings can't really protect against other type of bad behavior from individual websites. For example, if your website happens to be on the same machine as another website undergoing a Digg/Slashdot-effect, your performance will be affected as there is no real effective throttling mechanism. Most shared hosts will either kill the website hogging resources or perform some manual magic, like moving it to a different box.

A dedicated server is the other end of the spectrum - you get a full machine to yourself. Depending on which company you work and/or how much money you pay, this might be from a low-end provider or in a colo facility or in a big datacenter along with thousands of other boxes. You don't share the machine with anyone and you can party on it to your heart's content. However, since these require a 1:1 mapping between customer and hardware box, they're typically expensive.

A VPS falls in the middle. They offer you most of the benefits of dedicated servers (full root access on the box) without the 'bad-neighbour' problems of shared hosts. They tackle the latter problem by using some form of virtualization and use a hypervisor or a virtual machine to act as a sandbox and throttle resources. In Slicehost's case, they use Xen. In the screenshot below from my slice, you can see the custom Xen kernel in action as well as the 256mb limit imposed on my slice though the actual underlying hardware probably supports multiple GBs.

Some random notes

I picked the base 256mb $20/month slice. I was tempted to get a beefier configuration but forced myself to hold off for now. This is also more than double what I was paying Dreamhost
Slicehost has an awesome feel of community and sincerity around it. Their documentation and wiki are great, they know their stuff and they seem genuinely *honest*. That's hard to find in the overselling-world of web hosting.
I installed Ubuntu 7.10 on my slice. I would really like to have installed Windows Server 2008 but I doubt that Slicehost will offer that as an option anytime soon. :-)
I'm running lighttpd as my web server. I ran nginx for some time but switched to lighttpd when I needed to get mercurial to talk to my web server over fastcgi. Nginx rocks but I was just too lazy to figure out the spawn-fcgi magic to get it to talk with mercurial (and the cgi options seems unreliable).
After a lot of iptables and lighttpd configuration file hacking, I realized how much I had been spoiled by MMC add-ins back in Windows land. I wish there was a way to remote-connect an configuration GUI over SSH (I'm thinking of the ability to connect to remote servers through MMC).
I can't wait for the day when I get mod_rewrite style rules right at the first attempt. Or on the tenth attempt. On a tangential note, I like nginx's mini-language inside the configuration file.
I'm writing a Python web app and I'm bewildered by the options for running it. mod_wsgi, mod_python, fcgi, etc. For every blog post with some graphs supporting one option, I can find another saying the exact opposite. I'm also amused at how a lot of people measure server performance under load by essentially hitting the server in a tight loop. All the interesting stress related problems only show up after a few days (as I painfully found out while writing cacheman).
Mercurial vs Git - Git's poor support for Windows was a deal breaker for me. Now I just need to find a Mercurial<->TFS bridge :-)

Running Ubuntu on Windows Server 2008 Hyper-V

2008-03-02T22:49:00.000-08:00

I'm hoping this post saves people the trouble I had with getting Ubuntu 7.10 to boot on Win2k8 Hyper-V.

If you're trying to install Ubuntu 7.10 (either server or desktop) on a VT-enabled machine, you'll be probably get stuck at the "Loading..." screen. If you're interested in the gory details, check out this bug - the issue seems to be around emulation of real mode instructions and the graphics instructions that ISOLinux uses to boot.

The fix is simple - get the patch from this thread and patch your ISO images (the patch makes a small change to your isolinux.cfg.

Ubuntu is now purring away happily on my Win2k8 box :-)

Cacheman update (0.0.2)

2008-03-02T00:57:00.000-08:00

I've just put out a new Cacheman release (0.0.2 for those of you keeping count). You can get the bits from the link at the bottom of http://www.sriramkrishnan.com/blog/2008/02/cacheman-fast-distributed-hashtable-for.html (which will always point to the latest release).

There are no new features in this release but a *ton* of bug fixes. I had fun stomping out several race conditions that had crept into the code. I've also tested it on a bunch of diverse environments (single-core and multi-core, x86 and x64, Windows XP/Vista and Windows Server 2003/2008) so you shouldn't see some of the OS specific issues that were present in the last release.

I would like to thank Ayende for taking the trouble to find and report a lot of the bugs in the last release. Thanks Ayende!

Cacheman - a fast distributed hashtable for Windows

2008-02-25T03:34:00.001-08:00

Cacheman is a fast, distributed hashtable for Windows, implemented purely in managed code. This is a personal side-project which I started and abandoned several months ago and picked up a few weekends ago. If you just want the binaries, scroll to the bottom of the post to get the raw, early bits (and I do mean raw and early!)

It all started with my fascination with memcached and how well it works for several heavy-traffic sites. I set off to create a bare bones hashtable with a set of requirements in mind

Really fast and really scalable. I didn't have a firm numerical target in place but wanted to atleast do a few thousand requests per second. I wound up doing a lot more than that ( details in the how-stuff-works section below).
Client agnostic i.e anyone should be able to take a look at the protocol and implement a client in their language/environment of choice. Though Cacheman can't work with current memcached clients, I've deliberately made the wire protocol similar so that I can make it compatible with some work
Ops-friendly. Put simply, it shouldn't be a pain in the rear to deploy and maintain on hundreds of production machines. James Hamilton has written extensively on this subject - and I would be overjoyed if I get around to doing half the things he talks about.
Windows and .NET friendly. I wanted something that felt 'native' to Windows. I don't have a good justification for this - just that I work on Windows and managed code a lot and happen to like them :)

Note that memcached meets a lot of the requirements above. My biggest reason for starting from scratch was to just see 'whether I could do it' :-).

I'll walk you through a little demonstration before digging into how it works and why certain design decisions were made.

A quick tour

If you grab the binaries below, you'll see 3 binaries - the server, a console/client and a library that you can link to your own apps.

Server:
Run CachemanServer.exe with the /? argument to see the various arguments that you can give. By default, it'll try to bind to the first IP address and listen on port 16180 and set a maximum memory size of 100mb for the items it holds. You need to run with administrative priveleges the first time as it installs a few perf counters on first launch.

Console:
Since it was a pain to write client code to test out the server, I wrote a bare-bones console. Run 'CachemanConsole.exe' and at the prompt, type 'connect <the-ip-address-the-server-is-listening-on>'. From there, you can do basic get/set/delete operations as well as run a few homegrown stress tests. Note that the times in the screenshot below are not representative of server performance - the first 5 ms set is more due to the client being run for the first time.

Client library:
The console is actually a thin layer over CachemanAPI.dll which is where the meat of the client lives. Frankly, I haven't had much time to work on the client API and it needs quite a bit of work and polish before it can be used in a production system.

Dare Obasanjo has written extensively on how typical code using memcached in .NET looks like and the same pattern is applicable here too. Here's some sample code using the client library. Ignore the IPEndPoint stuff - the next release will let you write this in a nice config file and never have to deal with IP addresses and port in code.

How stuff works (with a few design detours and some pretty perf graphs)

In terms of general architecture, Cacheman is similar to memcached as opposed to other distributed hash tables (for e.g, EHCache from the Java world is an in-memory cache with async replication to the server). The server listens on a socket, parses commands from the client and responds appropriately. The client itself is pretty dumb and is just a thin layer over some networking code. This means that Cacheman is neither a read-through nor a write-through cache at the moment.
When you do a GET/SET/DELETE operation for a specified key, the client first needs to figure out which Cacheman server instance to talk to. To do that, the client does a quick FNV hash of the key and then mods that with the number of servers to get the server to talk to.

The disadvantage of this approach is that when you add a new server node or remove a server node, the cache needs to get repopulated. The fix for this is a consistent hashing algorithm which I haven't gotten around to implementing (and from the little I know, there are not too many of these algorithms out there)

The other choice is to use a central 'master cache server' which stores a lookup table of server nodes at which the clients can constantly poll. I'm not sure whether I like this too much as it seems to be a lot of added complexity and brings its own set of problems.
The client talks to the server using a simple network protocol. My protocol looks similar to memcached's text protocol but is stripped down and simplified. I looked at a bunch of options and my choice was influenced by a few things

Binary protocols are a pain to debug - text protocols are simple and clean, especially if you are up at 3AM poring over a Wireshark trace, IMHO :)
The protocol I have is small and lightweight enough that it doesn't have much parsing or network overhead.
I wanted to make my server work with the huge set of memcached clients out there today.This is going to be a challenge since the internal implementation is vastly different.

The networking layer also takes care to not do small writes. This, along with the protocol design lets me get a perf win by setting TCP_NODELAY on my sockets.

The most interesting part of all this was the implementation of the server. Being new to scalable servers, I wrote a few naive implementations which just didn't scale. My first implementation had a model where the server had a thread dedicated per socket. This was quite poor scalability-wise and just lead to a lot of thrashing threads.

The model I have now is built around NT's IO Completion ports. Unlike the previous model, there is a M*N relationship between sockets and threads (rather than a 1:1 relationship).

When data is queued to the completion port (from the client), one of the waiting threads is woken up and it goes to work on the data. Once it is done processing (either handling the request or deciding that it needs more data), it goes back to waiting by calling GetQueuedCompletionStatus internally. This lack of thread affinity for client sockets lets me multiplex several clients to a few active threads.

On my home machine (2.4GHz Intel Core 2 with 2GB RAM), I can push the server to around 16000 requests per second (clients and server running on the same machine). I *believe* I can get it to around 25K requests on the same box with a bit of work but anything above that is going to mean some serious work.

Cacheman comes with a nice set of perf counters for you to get at these numbers anytime

However, my code isn't entirely async. When the server sends data back to the client, it blocks on the send. This was a deliberate choice - whenever I did perf tests with async sends, I saw a noticeable dip in speed (as measured in requests processed per second). My theory here is that the hit is due to the context switch between the 'request-processing' thread and the 'socket send' thread.
Internally, the cache items are stored in a giant dictionary. I plan on moving to a better model in the future as right now, I'm forced to take a lock over the entire store for any destructive action. Cache expiration is pretty naive at the moment - you have a choice of using either LRU or LFU to expire items from the cache (apart from the items which get thrown out due to living past their lifetime). I plan on adding more cache expiration algorithms in the future and looking into a generational model. But the current model is good enough to ensure that you don't run out of memory on your server :)

The Bits

There is a lot of work left to be done (a better client, wrapping the server into a NT service, making things more ops friendly, lock-free internal data structures,etc) but I wanted to try the 'release-early-release often' approach for once. Things are quite busy at work (some kick-a** Popfly features in the pipeline as usual) so only expect bug fixes over the weekend :)

Be warned - these are really early, really raw bits. Stuff will crash or not work. The next version will change everything. Demons will be pulled out of your nose. Use at your own risk! Have fun and send feedback through the comments or to mail@sriramkrishnan.com or sriramk@microsoft.com

Download - Cacheman_0_0_2.zip

Acknowledgements

I don't usually have an 'acknowledgements' section but I just had to have one this time. A shout out to my friends who saw very little of me these past weekends. :) And to Brad Fitzpatrick and the rest of the Memcached folks for their awesome work.

Updated 2nd March 2008 - Updated link to Cacheman_0_0_2.zip which has a ton of bug fixes

Goosebumps

2008-01-19T16:23:00.001-08:00

I was at a local theater yesterday to see Cloverfield and almost dropped the popcorn when I saw this.

Live action Popfly tutorial

2008-01-11T13:49:00.001-08:00

My team put this together a few weeks ago. I fell sick on the day of the shoot and couldn't make it into the video unfortunately :-(.

Amazon SimpleDB - Technical Overview

2007-12-14T00:28:00.001-08:00

Structured storage was one of the missing pieces in Amazon's cloud services jigsaw puzzle (the other has to be the ability to host a site completely on EC2 without using dynamic DNS hacks) and Amazon is plugging that hole today with the launch of SimpleDB.

There is an Information Week article and the official site is at http://aws.amazon.com/simpledb

Here are the highlights as I see it

Structured storage service in the same model as EC2 and S3.
Launching in a limited beta in a few weeks (I've signed up on the waiting list)
$0.14 per machine hour, $0.10 per GB of data transferred in. The data 'out' pricing is a bit interesting - $0.18 for the first 10 terabytes for the month, $0.16 for the next 40 TB and $0.13 for all data after that.
Apart from this, there is a cost for storing data -$1.50 per GB per month. Note that you can store data on S3 for cheaper rates and transfer it to SimpleDB for free
10 GB maximum per domain (think 'table') and a limit of 100 such domains. Amazon indicates that these restrictions will be loosened soon.
Data model is similar to that of a spreadsheet, except that each 'cell' can have multiple values. The screenshot below (stolen from their documentation) shows off the model very well. The individual worksheets are 'domains' in SimpleDB terminology, rows are 'items' and columns are 'attributes'. However, each item can have more than one value for an attribute (a car could be both 'blue' and 'black') and attributes are optional. In this respect, the model is similar to Google's BigTable. They are pretty dissimilar in other aspects - BigTable stores previous versions of all values and SimpleDB can do limited subset of SQL.
Both REST and SOAP apis available (the WSDL should be up soon at http://sdb.amazonaws.com/doc/2007-11-07/AmazonSimpleDB.wsdl)

There were a bunch of things which caught my eye

Pricing

The pricing for storing data on SimpleDB is much higher than the costs on S3. Storing 1 GB of data on S3 for a month is going to cost you $0.15 while the same on SimpleDB is going to set you back on by $1.50. This points to Amazon using pretty different hardware for the two services.

I'm also fascinated by the idea of 'box usage'. For every query, Amazon returns the amount of 'machine time' used to execute that query. Since these queries are almost surely getting distributed over a variety of nodes, I'm curious to know how this 'machine time' is calculated.

Data Model and APIs

I love the data model for SimpleDB. I've never been a fan of relational tables and SQLs and prefer data structures where everything's just one huge hashtable. Though SimpleDB's data model is not exactly a hashtable, it is pretty close. There are several things to like here if your programming loyalty lies with the dynamic side

Optional attributes
No types - you can stick any value into any attribute.
Multiple values per attribute (think lists or tuples)

All these things are possible using standard databases but would require quite a bit of work. And changing table schemas once you've piled up a decent amount of data is definitely not fun.

To program against the data, you have a choice of (some very clean!)REST APIs and SOAP APIs. Here's a sample REST request and response pair from the docs

Sample Request

https://sdb.amazonaws.com/
?Action=Query
&AWSAccessKeyId=[valid access key id]
&DomainName=MyDomain
&MaxNumberOfItems=3
&NextToken=[valid next token]
&QueryExpression=%5B%27Color%27%3D%27Blue%27%5D
&SignatureVersion=1
&Timestamp=2007-06-25T15%3A03%3A09-07%3A00
&Version=2007-11-07
&Signature=2wVXB1x0NSWWETwLylZPVP%2FtqXQ%3D

Sample Response

<QueryResponse xmlns="http://sdb.amazonaws.com/doc/2007-11-07">
<QueryResult>
<ItemName>eID001</ItemName>
<ItemName>eID002</ItemName>
<ItemName>eID003</ItemName>
</QueryResult>
<ResponseMetadata>
<RequestId>c74ef8c8-77ff-4d5e-b60b-097c77c1c266</RequestId>
<BoxUsage>0.0000219907</BoxUsage>
</ResponseMetadata>
</QueryResponse>

Eventual Consistency

This is going to surprise a lot of SimpleDB users (and probably cause a lot of hard bugs). Reading data from SimpleDB immediately after a write may not reflect the latest updates. SimpleDB relaxes the 'C' in ACID and doesn't promise that you'll instantly see your updates (due to it being propagated across all the copies of your data). Amazon may not have a choice here (see CAP Conjecture) but I don't think this is going to be popular with a lot of programmers.

Dare talks about this extensively and as someone writes code for a high traffic website with lots of data flowing around, I shudder at the prospect of not relying on data not being always up to date. For SimpleDB developers, this is going to mean some extensive coding to make their apps resistant to stale data - something programmers traditionally never had to worry about.

Another possibility is that frameworks could take away the pain of doing this checking - this is definitely going to be an interesting place to watch.

Query language
Unlike Google's BigTable which eschews any and all forms of querying (probably in favor of a map-reduce type paradigm), SimpleDB supports a simple set of query operators - =, !=, <, > <=, >=, STARTS-WITH, AND, OR, NOT, INTERSECTION AND UNION. Also, queries can only execute for a maximum of 5 seconds.

There are several interesting properties here

Since the data is split across several nodes, these queries must be getting palletized across several machines. There's been a ton of academic work on database query parallelism and you can be sure Amazon is making full use of it here .

The 'type-less' data model is going to lead to some peculiar query problems. For example, when comparing '13-5-2007' and '24-4-2006', do you compare it as a couple of dates or do you compare it as a couple of strings? The docs talk about this problem and asks developers to convert all dates to ISO 8601 format.

This is probably another example of Amazon choosing implementation simplicity even if it means that the developers have to do a bit more work. This is not necessarily a bad thing - I've seen products (non-Microsoft and Microsoft) which try and do the opposite and they're not necessarily always successful.

The ecosystem and the competition
Amazon has built a good ecosystem around their services. Their services all work together (the same AWS keys can be used, the same X.509 certificate system,etc). The only thing missing is the ability to statically host a site completely on Amazon. What's even more surprising to me is that these are the sort of services that you would expect Google to release, given their much talked about infrastructure. As far as Microsoft goes, the only current service I can think of that comes close is Astoria (something which I should definitely spend more time digging into).

If Amazon does as good a job with this as they did with S3 and EC2, startups are going to love this service. Instead of having to shell out a ton of money up front and having to worry about dedicated hosting and colos, you now have a pay-as-you-go database in the sky.

Update #1

This post says that SimpleDB is built on Erlang. Interesting!

Update #2

See the Techcrunch post and the Techmeme discussion here

The drama! The suspense!

2007-10-24T12:14:00.001-07:00

Someday, I hope they make a good movie out of the wild and crazy times we live in.

Popfly - open to everyone!

2007-10-18T09:16:00.001-07:00

I see that Soma has put up his post which means that this is now official -Popfly is no longer in alpha! This means that the old invite system is now gone - anyone can head on over to www.popfly.com and start using Popfly instantly. Woohoo!

It has been a fun and wacky ride to this point. Since May, we've added a ton of features and made a lot of changes across the board. From integration with Dapper and Facebook and a slew of other sites, to making the site a whole lot faster for everyone to a lot of other goodies. Instead of linking to just one mashup like everyone has, I urge you to check out this mashup of mashups (if you're at the Web 2.0 summit, you would have seen the kiosks running this).

For me, it has been a wild ride from the time I joined the team in May. I've worked on features as varied as Dapper and Facebook integration to the site-wide search and ranking system. I've learned a *lot* about building massive web 2.0 sites/services - for someone coming from a traditional desktop software background, it has been a pretty crazy experience.

And talking about crazy, here's a photo I took of the crazy team that builds Popfly (this was taken in my office on my birthday)

Update: Here's the Techmeme discussion.

Dave Cutler and the Bill Gates award

2007-09-10T01:58:00.001-07:00

Adam Barr beat me to a post that I wanted to write after the Microsoft company meeting on Thursday - Dave Cutler winning the Bill Gates award. Adam has a nice summary of what the award is and a couple of typical DaveC anecdotes.

What he doesn't describe is the atmosphere in the stadium when Bill Gates announced Cutler's name - you had thousands of employees give DaveC a standing ovation (and laugh at the joke DaveC cracked in his acceptance speech which I see Adam wisely left out :-) ). I remember telling someone sitting next to me "There goes any chance I had of ever winning this thing". When you start of an award series by handing one to Cutler, you're setting a standard which is almost impossible to live upto.

After the event, I sent him a heartfelt 'thank you' mail. I'm not sure whether he read it (I haven't gotten a reply yet) but I wanted to convey the gratitude that all of us felt - no accolade or award is too much for this man. Studying Cutler's life and his work in the past year or so have lead me to my current fascination for how to do good, sound 'engineering' (as opposed to just writing code and hacking together code).

I've talked to a lot of people who have worked with Cutler in the past. Almost everyone I spoke usually had two things to say

He was the greatest programmer/engineer/project leader they've ever seen.
He was the angriest and scariest person they've ever worked for

The latter is where all the fun anecdotes come from. Here are a couple of anecdotes I've heard from people who had worked with Cutler in the past. All these probably happened over a decade ago.

If you made a bad checkin, Cutler would take the offending piece of code and send a mail to Jim Allchin (who was the VP for Windows and a bunch of other stuff) with the additional line "Do you know that this person works for you?". The fun part was that you as the developer never got CCed so you were never really sure what was getting sent to Jim Allchin about you :-). Of course, no one ever got fired as a result of this but it always kept people on their toes. :-)
DaveC and members of his team were having a meeting in a conference room when a senior VP walked in with some bigshot she ¹ was meeting. There had obviously been some conference boom booking snafu. Clearly impatient, she snapped "I believe I had booked this room? Can you people please leave?"
Big mistake.
Cutler slowly stood up and said "I'm Dave Cutler. Who the f*** are you?". The VP quickly closed the door and beat a retreat without saying a word. Later, someone on Cutler's team looked up the conference booking information and found that the VP was right - she *had* booked the room and Cutler's team had mistakenly come to the wrong room. :-)

Seriously though, Cutler is probably the greatest programmer to work at Microsoft (a senior friend of mine called him a "contender for the programmer of the century of the century award"). If you're at Microsoft, I would strongly recommend reading through Cutler's code (you can do this if you are a student and have access to the Windows Research Kernel). I learnt a lot about how to write beautiful C code by looking at his scheduler implementation in the NT kernel. Above the level of coding style, it is pretty evident that he was able to make design choices back in the late 1980s that have let the NT kernel scale and adapt itself in a wide variety of ways.

Designing and writing large parts of an operating system that is used by almost a billion people - that's a pretty tough act to follow.

Notes

1. I think the person who told me this anecdote said it was a 'she' but I could be mistaken.

Apps of yesteryear (or) The one in which I feel old

2007-09-10T00:49:00.001-07:00

A conversation I had with Aarthi yesterday went like this

Me: I feel old.

Aarthi: Why?

Me: In the past, when I saw some good-looking girl on television and looked her up on the web, she would be much older than I am. Today, I looked up Deepika Padukone and to my shock and horror, she's 3 years younger! I feel middle-aged.

Seriously though, I've now spent enough time with computers to see a generation of software. I find that my usage of software has changed - I no longer use apps that I used to use all the time 4-5 years ago. Here are some of the apps I used to use a lot in the past but now no longer use at all.

Note - most of the apps below still have thriving user-bases so don't read this as a 'X or Y was used in the past but is now dead' post.

Winamp

For several years, Winamp was my music player of choice. I don't remember how exactly I came across it - I must have got it through one of those computer magazine CDs I used to collect back then. I remember collecting hundreds of skins for it and installing a zillion different plugins ( the most memorable were the DSP plugins which did magic to how it *sounded*).

This was back in the day when a new version of some software had new features (as opposed to a security fix :-) ). Whenever there was a new version distributed, I would install it and go straight to the About screen where there would be some funny release notes.

What I use now

Windows Media Player / iTunes

I'm not quite sure why I moved away from Winamp - I do remember spending a long time away from it after the disastrous Winamp 3 and then moving back to Winamp 5. I think over time, WMP caught up and fixed a lot of the annoyances that had stopped me from using it in the past.

To Winamp's credit, a lot of my friends back home in India still swear by it. This may just be a case of me being assimilated by the B0rg :-)

IrfanView

Before the likes of Picasa and Windows Live Photo Gallery, you were out of luck if you had an image in a non-mainstream format (I used to run into this a lot from my past in game map-making) . You can argue that you're still out of luck but I just don't deal with much of those images anymore. IrfanView was the best story in town for image viewing - it could deal with images from almost any format and was *very* light-weight.

From the website, it appears that IrfanView is still going strong. The author appears to be getting..umm... a lot of love from his users. Quoting from his site

...In the years since IrfanView hit the Net I've received more than 50,000 e-mail messages thanking me, congratulating me, and wishing me well. And at least a dozen women have wanted me to be the father of their children ;-)...

What I use now

The picture viewer built into Windows or whichever OS I happen to be using at the moment. I've dabbled with Picasa and Windows Live Photo Gallery but have found plain ol' files and folders good enough for most of my needs.

IrfanView is still a great app - there are lessons to be learnt from the community support it enjoys (check out the number of codecs, skins packs, etc).

Winzip

Winzip was the de-facto archiver pretty much wherever you went. I remember seeing it on all my school computers, all the computers at college and on every friend's PC. However, I always remember seeing the shareware version (which used to pop-up a nag screen) so I'm not sure whether any of the people I saw actually bought a copy of it.

I remember the day when I stopped using the Winzip 'wizard' and just used the plain interface. That was probably the first time I felt like a true 'power user' :-).

I hadn't followed Winzip news for a few years now - they seem to have been acquired by Corel and do have a new version (v11.1) out. They also have support for RAR and Bz2 now so I should probably go and give them a try.

What I use now

I moved to using WinRar and now 7-zip for some of the more esoteric compression formats. For good ol' zip files, I just use the 'Compressed Folders' support built into Windows.

Visual Basic 6.

What more can I say :-). This was the single biggest reason I took up programming full-time. Though I don't use VB day-to-day anymore, I still feel that my life revolves around the VB family. Apart from the fact that I work in the same building as the VB team, what we're doing with Popfly is in a lot of ways, VB 'for the web'.

It annoys me that VB6 doesn't get the credit it deserves at times. I would rank it as one of Microsoft's greatest applications ever, probably second only to Windows 95 and some of the Office releases.

Sometimes, I miss the days of installing VS 6 and then MSDN from the multiple CDs (ok- maybe not the multiple CDs bit). I should have guessed back then that I'll wind up working for Developer Division and be a part of making Visual Studio :-)

What I use now

Too many to list here :-)

I can think of a lot more apps that I've used in short bursts - Paintshop Pro, ACDSee, WinAce. Before anyone flames me, this list is Windows only because that's where I've done almost all of my computing. Also, a lot of the open source apps I use (Firefox/Thunderbird) are still apps that I use everyday.

I wonder whether someone will go through a similar exercise a few years later with social network sites of today :-). Does anyone have other examples of applications they used a lot in the past?

Social Graph = Blogging 2.0?

2007-09-09T16:16:00.001-07:00

Is it just me or are the people talking about the 'social graph' and speaking at conferences around the theme the same folks who were speaking at blogging conferences until they became passe ?

It's getting to the point where I just skip over any post in my feed reader which has the term in it.

P.S I don't include people like danah boyd or Dare Obasanjo in the list above as they actually *work* in that area.

A 'Hello World' Windows driver from scratch

2007-09-01T23:00:00.001-07:00

One of the advantages of having no social life is that you get to spend long weekends catching up on technical topics that you otherwise wouldn't be able to :-). For this weekend, I decided to take a break from the usual feverish Popfly hacking and catch up on some stuff I've always wanted to dig into but never really found the time for.

On the top of my list was to go and learn how Windows *really* works in kernel mode (a pretty long way away from my daily managed code/JS hackery). I ordered a copy of Microsoft Windows Internals by the incomparable Mark Russinovich and David Solomon and after some re-reading (the last time I read it was over a year ago), set out to write my first device driver.

This is not the first time I've tried to do so and I've often forgotten how to get started . This post is mostly a reference for me to come and look up later.

Note: This really isn't a tutorial. I would strongly suggest that you read all of James Brown's tutorials which do a much better job of *explaining* this stuff.

Pre-requisites (all free downloads)

The latest Windows Driver Kit installed. I pulled a copy from one of our internal shares but I just found out that it is now a free download as well (you could only get it shipped to you in the past).
Microsoft Virtual PC. This is to let me test my driver without having to get another machine and finding the right cable to go between them (I'm just lazy). Modifying this tutorial to work with a real machine is pretty trivial.
Debugging Tools for Windows. Visual Studio cannot do kernel-mode debugging and hence you need to use the WinDBG/NTSD/CDB family of debuggers.
Windows XP/2003/Vista installed on VPC. In my case, I have a Win2k3 VPC but depending on which operating system you have, pick the right build environment in the 'Build your driver' section below.

Code for your driver

1. Create a directory where your driver sources will lie. For the following example, all my source files will lie in C:\testdrv

2. Create a file called test.c with the following contents. This is as simple a driver as you can get (in fact, it doesn't even know how to unload itself). When started, it prints a 'Hello World' message which any attached kernel debugger or DbgView will see.

#include <ntddk.h>

NTSTATUS DriverEntry(PDRIVER_OBJECT DriverObject, PUNICODE_STRING RegistryPath)  
{

  DbgPrint("Hello World\n");
  return STATUS_SUCCESS;

3. Create a file called makefile.def with the following line it it. This pulls in the appropriate makefile for your build environment (more on that later).

!INCLUDE $(NTMAKEENV)\makefile.def

4. Create a file called sources (note that it doesn't have any extension) with the following contents. This specifies which source files you are compiling, that it will build a driver and the headers and lib files to use.

TARGETNAME = test
TARGETPATH = obj
TARGETTYPE = DRIVER

INCLUDES = %BUILD%\inc
LIBS = %BUILD%\lib

SOURCES = test.c

Building your driver

1. Click on Start->All Programs->Windows Driver Kits->WDK 6000->Build Environments->Windows Server 2003->Windows Server 2003 x86 Checked Build. I'm picking this since I have a 32-bit Win2k3 VPC - pick the one which matches the closest with your target machine. For example, if you have a physical Windows Vista x64 machine you want to try out your driver on, you would pick 'Windows Vista and Windows Server Longhorn x64 Checked Build Environment'.

This drops you into a command prompt with some nice environment variables (your build environment) already set for you.

2. Change to the directory which has your driver source and run 'build'. This invokes build.exe , the de-facto Microsoft build tool for almost two decades until MSBuild came on the scene (and even now, a lot of product groups still use build.exe). Written by Steve Wood over 15 years ago, this is one tool that has withstood the test of time :-) ¹

If you've followed along well, you should see a test.sys and a test.pdb (along with other files) in your object build directory.

Setting up your target machine for kernel debugging

1. To attach a kernel debugger to a running instance of Windows, you need to do some minor configuration. In the target VPC Win2k3 instance, go to My Computer->Properties->Advance->Settings under Startup and Recovery->Edit. You should see a line like

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise" /fastdetect /NoExecute=OptOut

Add "/debug /debugport=com1" to the line above (without the quotes).

If you're using Windows Vista as your target operating system, you can't use this technique - you'll have to use the inbuilt bcdedit.exe utility.

2. In the virtual machine's settings, redirect the COM port to a pipe called \\.\pipe\vpc . What we're doing here is telling Windows to redirect all debug spew to the COM1 port and then telling VPC to redirect COM1 to a named pipe.

3. Restart the virtual machine (the boot settings need a restart to kick in).

4. One final step - running the kernel debugger! Run windbg with the following command-line (if you already have symbols setup correctly, ignore the -y parameter).

windbg-y SRV*c:\symbols*http://msdl.microsoft.com/download/symbols -k com:pipe,port=\\.\pipe\vpc,resets=0,reconnect

This should open up windbg and connect it to your running instance of Win2k3 on VPC.

Registering and running your driver

Whew. Just a couple of steps more and we'll be home. The last 2 things for us to do are to register the driver and then to run it. Our driver currently cannot be unloaded (the ability to do so would just require 4 more lines of code though) so you'll have to restart the machine manually to see new changes reflected.

1. First, let's register the driver. James Brown's tutorial covers some of the ways you can register the driver but (like him), I'll recommend downloading the 'Driver Loader' utility from http://www.osronline.com which unfortunately requires a free registration. Run it on your target Win2k3 VPC

2. Copy your drivers' binaries over to the VPC (I typically do this using the shared folder feature). Point Driver Loader to your binary and register your driver. This needs to be done only once irrespective of how many times you rebuild your binaries.

If you've followed along so far, congratulations for you're finally home. Hit 'Start Service' in 'Driver Loader'. This should result in a call to your DriverEntry method and in your WinDBG running on the host computer, you should see this

To quote John McClane, Yippie-Ki-ay <insert non-family-friendly-term>!

From here, it is pretty trivial to start exploring. Here's a screenshot of a breakpoint in my DriverEntry routine (the WRK stuff is because I'm running a custom Windows kernel built from the publicly available Windows Research Kernel code).

Happy hacking!

Notes

1. If you're interested in the internals of build.exe, you should check out the code that ships with Rotor (online cached copy here). However, I haven't checked to see whether it is from the same code base as the real build.exe.

NYTimes on mashups

2007-09-01T19:49:00.001-07:00

The NYTimes is running a piece on mashups written by Anne Eisenberg which has a section on Popfly with quotes from John. I was involved with this piece, though quite tangentially.

A few weeks ago, I was going through my check-mail-first-thing-after-waking-up routine with my personal mail account. Instead of clearing out of my junk folder without checking it (as I usually do), I saw a mail there from Anne. My mental reaction in the first few seconds went from "This must be a hoax" to "Oh cr*p - someone from the NYTimes got my email id from my blog. Did I just blog something which I'm going to get fired over?"

Thankfully, neither was true :).

Anne - if you're still reading, thanks for the kind words about my writing skills :-)

The best complement that Popfly has ever received

2007-08-06T23:40:00.001-07:00

Lots of people have said lots of things lots of things about Popfly. One of my colleagues forwarded an article (you need to be a subscriber to view it) by Chris Pirillo in Computer Power User magazine where he says a bunch of nice things about Popfly (thanks Chris!). However, he wraps it up by saying something about Popfly which definitely has to be a first.

"..Even Paris Hilton could use Popfly.."

Forget the dog and the duck and all our current branding. I think we should use this as our tagline from now on. :-)

Open source and scratching itches in the cloud

2007-08-03T16:31:00.001-07:00

[Note : I happen to work at Microsoft but do read and understand the disclaimer at the bottom. This is just the geek in me talking. I'll get very annoyed if someone links to this with the heading "Microsoft employee says...". I don't even work in the Live org so don't even start thinking that this represents any future direction/future products from Microsoft. I work in Developer Division so if I ever say that we're going to bring back Algol-60, then you're onto something ;-)]

I read a couple of posts today which made me think about open source in web 2.0 - the first is Tim O'Reilly on Yahoo supporting Hadoop and the other is Dare on 'Free Data'. Tim has this interesting factoid about Nutch in his post

"...Some years ago, I was on the board of Doug's open source search engine effort, Nutch. Where the project foundered was in not having a large enough data set to really prove out the algorithms. Having more than a couple of hundred million pages in the index was too expensive for a non-profit open source project to manage. One of the important truths of Web 2.0 is that it ain't the personal computer era any more, Eben Moglen's arguments to the contrary notwithstanding. A lot of really important software can't even be exercised properly without very large networks of machines, very large data sets, and heavy performance demands..."

And from Dare

"...This leads to a tendency for the rich to get richer because since they have the most data they provide the most value for end users (e.g. Amazon). Another problem is that social software leads to lock-in. My buddy list on Windows Live Messenger and my list of friends in Facebook are useless to me outside the context of these applications. Although I can get all of my history and data out of these services, I lose the value I get from the fact that all my friends use these services as well...."

Based on the above (which I agree with), I'm going to posit the following.

Most of the value in Web 2.0 comes from data (generated by lots of users). E.g - del.ico.us, Digg, Facebook
It is beyond the financial means of any one person to run large scale server software E.g Gmail, Google Search, Live, Y! Search
Companies will fight hard to keep their users (and their data) in their system as that's what will keep them alive. You need the ad money from your users to pay your bills. At the least, moving to a competing service will not be painless. (pretty much any service which requires a log in)

Hacking the code

Now, let's take these three and juxtapose them with open source and in particular, where open source came from. For me, the powerful idea about open source has always been geeks scratching their itches. Find a bug in Emacs? Checkout the code from whichever CVS server it is on, fix the bug and then do the configure-make-make install dance. You could submit your change back as well (though YMMV on whether it gets accepted or not).

The same is not possible with any large cloud-based server. If you find something you dislike in Gmail or Windows Live Hotmail which can't be fixed in the browser through some GreaseMonkey magic, you're stuck. The same goes for Facebook, Digg, etc ¹. If you are have an 'itch' to scratch (and you are one of those people who wouldn't use Microsoft software if there was only PC left in the world and it ran Windows <grin/>), you probably have to contribute at one of these levels

1. Change things in Linux/*BSD which powers some of these sites

2. Change things in Apache/Lighttpd/Mongrel/memcached/MySQL

3. Change things in PHP/Ruby/Python

4. Change things using the API provided by the site.

There's a huge gap between #3 and #4 there - that's where the Facebook backend code lives or Google's backend code (or Windows Live code if you swap in the stuff that the Live guys use). This is code that you don't get to see or modify and you pretty much have no influence over.

Code might be data but I'd rather just have data

The above problem is solvable if Google and Facebook and Microsoft want to - they could go the Slashdot/Wikipedia way and actually put out backend code up on CodePlex or SF. But you'll probably soon find that the code doesn't have as much value as having the data to back it up.

Example - look at MySpace. It would be possible for a bunch of coders to take MySpace's backend and create a modified version of MySpace in a few days. However, it is useless without the data of the millions of users who make up MySpace. Obviously, this is not data that should fall into Joe RandomGeek's hands without the consent of the users themselves. You can see how this soon turns into a very hard problem to solve.

The only site I know which gives away both code and data is Wikipedia. It is theoretically possible to take MediaWiki and the WIkipedia dumps and have the equivalent of the CVS checkout I first talked about. You are free to hack on MediaWiki and since you have the data, you are free to do whatever you want with it. I'm curious as to why there has not been a lot of interesting work done on top of Wikipedia's data - the only notable work I know of is Aaron Swartz's when he ran for office inside Wikipedia.

Even if you have both code and data, you run into the third and arguably the toughest problem

It's all about the hardware, stupid

Your typical long-haired geek probably can't get his hands on the following from the comfort of his bedroom ²

Geo-distributed datacenters which have scary electrical,networking, legal, social and financial design issues to worry about.
Large scale blob storage (S3, Google File System)
Large scale structured storage ( Google's Bigtable)
Tools to run code across such infrastructure (MapReduce, Dryad).

Constructing one by yourself is out of the question (unless you are an expert on topics like air conditioning and setting up multiple diesel generators which are better than these). The other option is to shell out hard cash to get some racks on someone else's datacenter. And *that* costs a lot more than what your average college-going hacker (don't you love stereotypes?) can afford. Or even what I can afford :-)

Possible answers

I'm not really saying 'gloom and doom' here. I think we'll see consolidation and commodification of several of the above pieces.

"Hmm..so you want a few thousand machines. x64 boxes with oodles of RAM, you say? In Asia as well as Europe? Ok - one order coming right up. Would you like some chips with that, Sir" ?

As a geek, it is mouth watering to think of the possibilities when that eventually happens. In reality, instead of specifying machines, you'll probably just call some service and the magic (how many machines, which data center to serve the request from ) will happen transparently on the backend.

This needs to happen at multiple levels. At the datacenter/ geo-distribution level Amazon's S3 and EC2 provide great features at a great price point . However, you still need to run your servers...today. A few years down the road, you could theoretically lease a bunch of machines from Google, Yahoo, Microsoft or Amazon and you don't need to worry about machines going down, coming up, failovers across continents - it'll 'just work'.

In terms of code, Doug Cutting's projects and projects like memcached means that you already have a lot of backend code to play with. From the Microsoft side of things, you have everything from Windows Server to IIS to SQL Server which definitely do scale very well - Microsoft runs them *everywhere*.

The really tricky problem is data. No company wants to part with their data as it really is the 'crown jewels'. If an alternative social network sprung up which had all of Facebook's data and users, Facebook could go out of business very soon. This is where the 'Free Data' movement comes in (see Dare's post for more). I still don't see a good model of getting to everyone's data without trampling all over privacy and security. This may be a problem that never gets solved.

Whatever happens, there'll soon be a 'cloud software' equivalent to './configure, make, make install' and it is going to be fun to watch and see what it turns out to be. Until then, if you've got an itch, I'm afraid you can't scratch.

Notes:

1. LiveJournal might be an exception here

2. I'm not sure how far along Doug Cutting's open source equivalents have come along for these - so that might be an answer

Popfly videos

2007-07-26T19:12:00.000-07:00

There are a bunch of videos out there with demos of Popfly.

Popfly on On10 - this was put up there just a few hours ago. This was filmed before our new features last week (so you don't see the new home page, etc).
Popfly at ReMix Australia - Nick Hodge showing off Popfly in the land down under.
Channel 9 Popfly interview - this was done a few days before we went public in May. You see several members of the team in this one
The Overview video - this is a screencast on our site which walks you through the various features of Popfly

I've been doing a bunch of demos over the last few days and it is *always* scary when you do a public demo with Flickr or Twitter or any web site with user generated content. You get into these awkward situations in public where you wind up saying "So, I'm going to pull in photos from Flickr of beaches....and...ummm... here is a photo of a naked man on a beach."

We had trouble finding a 'safe' term search for by default. 'Cats' is probably the safest we've found so far :-)

Popfly Updates

2007-07-19T20:01:00.001-07:00

This afternoon, I typed in 'www.popfly.com' in my browser and closed my eyes and prayed. When I opened them and found that the page looked the way it was supposed to do, I shouted out loud and danced through the corridors (which I don't do at all due to my very limited dancing skills :-) ).

The occasion? We had just pushed out new bits to production and my code was live and out there. And it was running the way it supposed to. There's no high which comes close to this. :-) It was quite an exciting afternoon - we had a ton of emails fly back and forth as we tried out the new production site, touched and probed the various parts of Popfly to make sure everything works.

There are a ton of new and exciting features - you can find the entire list at our team blog. I'll try to expand on some of my favorites.

Aaron Brethorst did a great job with simplifying the navigation. We had different navigation schemes in different parts of the site - we now have one unified navigation bar.

We also put out some major, much-needed features related to projects. You can now add and see comments, tags and publish links to Facebook. Here's a link which our big boss just published to his Facebook profile.

Unlike the last deployment, I actually wrote some code this time <grin/>. I spent most of my time on the new home page. Until now, we just had a static page when you entered the site which had a couple of links. If you wanted to go back to the half-finished mashup you were working the previous night, you had to poke around through the list of projects. We wanted to change all that - so we put in a super-easy 'recent projects' list so that you are just one click away from your masterpiece.

The other big thing which you will see on your home page on the left side is a combination of news items from the team blog as well as info on what your friends are upto (are they sharing new projects? Are they making new friends? ). This was a whole lot of fun to code up as designing for such a large number of users is quite challenging. We frequently had design discussions where the conversation would go like this

" I think we should implement it using this technique"
"That would not hold up well once we have X users on the site...at the same time"
"Dammit!" (and go back to the drawing board)

Here's a screen shot of how it looks (with fake entries of course)

And if you're wondering what's the dog is doing at the top, well, we've played around with our logo a bit this time and tried to 'mash' things up. The evil purple duck is still present all through the site but you'll see variations of the Popfly logo at some places. Like 'Attention Dog' above. Yes, that's really what we call him. Yes, we're a bunch of nerds :-)

Have fun with the site, y'all. If you experience issues (happens sometimes when we deploy shiny new bits), leave a message on the forums or post a comment or send me an email.

Update - s/Alertness/Attention Dog. Thanks to the nitpicker who found it :-)

Popfly Files Part 1 - What is Popfly?

2007-06-23T00:54:00.001-07:00

Over the last one month, I've got a lot of queries on what Popfly is, what it can do and what it is going to do in the future. Instead of repeating myself over and over again, I thought I'll save myself some typing by compiling a few blog posts.

Unlike most of my friends who started off with C/C++ or Java or GW-Basic, I learnt programming with Visual Basic. I actually had tried to learn C and C++ but I just didn't find that interesting back then. Most tutorials focused on the syntax of the language, how to write simple math applications or how to draw circles or simple graphics. None of that stuff excited me - so I had made up my mind that programming and computers were just not my thing.

All that changed one sunny afternoon in mid-2000. I had just installed Windows 2000 on my first home computer and I fired up VB6 for the first time. I gingerly drag-dropped a button onto the main form and then hit 'Run'. What followed was pure bliss. I actually saw an application *I* had created running. Not Microsoft. Not Adobe or some professional. *Me* .It looked just like any other Windows application and the button was clickable ! This was a huge moment for me and probably the pivotal moment which changed me from just a computer user to someone who could create software.

Popfly tries to recreate this 'aha' moment for a new generation of computer users.

The Past

Two great technologies have done in the past what Popfly is trying to do now for a new generation.

The first is Visual Basic. VB let users create GUI applications without having to learn how Win32 worked. A lot of the features made popular by VB have stuck around. You see VB's influence in current day Visual Studio and even frameworks like Winforms which were built years after VB. I would credit VB with features like Intellisense, controls and being able to generate code for them by drag-drop, a WYSIWYG GUI tool with code generation and everything surrounding data access (ADO, the data designers, data binding, etc).

The second is the combination of HTTP+HTML+Javascript+the webbrowser. There are a couple of features from this world which stand out for me. The first is 'View Source'. Like I said in a previous post, this let everyone learn web development by just picking apart a webpage they liked. The second is the forgiving nature of the web browser world. If the smallest HTML mistake had just shown an empty page and a parsing error, a lot of web developers would have never become at what they do today.¹

What is Popfly?

Let me copy-paste from the official description page (which you should definitely read)

Popfly is the fun, easy way to build and share mashups, gadgets, Web pages, and applications. Popfly consists of two parts:
1. Popfly Creator is a set of online visual tools for building Web pages and mashups.
2. Popfly Space is an online community of creators where you can host, share, rate, comment and even remix creations from other Popfly users.

You can get a good overview of all the features here but let me try and point out the highlights.

Blocks

You can think of blocks in two ways. One way to think of them is as a component which provides some functionality. Blocks usually provide one or more of three types of functionality. Blocks that let you display stuff (like in a carousel, sphere or one of the mapping blocks), blocks that wrap around a web site and let you access data from that site (Flickr, Digg, Twitter, etc) and blocks which do some sort of transformation on their input (Sort, Combine, etc). Of course, there are a lot of blocks which don't fall into this neat classification (e.g. Calculator, the Block Inspector block) but they're (currently) few in number.

Another way to think of them is as arbitrary chunks of Javascript code. Since it is Javascript, blocks are free to (and often) use HTML and/or XAML for Silverlight. Every block on the site maps to a Javascript class². Blocks are represented throughout the site with a spinning red 'block' shown on the right. You can use any of the default blocks provided (written by the Popfly team) or over a hundred blocks (as of writing this) created by Popfly users. You can create your own - from scratch or by 'rip'- ing some other block's which lets you see how another block works and lets you build upon/modify it.

Blocks by themselves aren't that interesting - they're like the individual characters in Super Human Samurai³. It's when they join together that true magic happens!
Mashups

Like I said in the previous section, the blocks by themselves aren't that powerful - like the individual characters in an old TV show. A better analogy is components (COM/ActiveX/Corba/beans/pick-your-poison). When you put these blocks together, you create what we call 'mashups'. An example of a mashup would be joining the Soapbox block (which generates video output) and the video player block (which, well, lets you play videos) and voila! You instantly have your own mashup which plays Soapbox videos. Another popular type are ones involving maps. One slightly advanced mashup you can create using the Virtual Earth, Timer and GeoNames service lets you replicate Twittervision in seconds.

The primary tool used to create these mashups is the mashup design surface (shown on the right). Most people would agree that this is one of the most visually intense development environments created ⁴ . You can search for blocks and add and connect them. When you connect blocks, Popfly tries to match the right output to the right input. We try to do this using the 'type' of the input/output (more on that in a later post).You also have the ability to customize all the Javascript and HTML generated by the designer using the custom HTML view.

Currently, adding custom Javascript to mashups (as opposed to blocks) is slightly difficult - we're working on making this easier in future releases.
Web Page Creator

Popfly includes a WYSIWYG web page creator which was built off the Office Live Web Designer. This is very popular - users can easily create pages, change themes, styles and colors without writing any HTML. You also have the ability to embed Popfly mashups and share them externally just like mashup projects. When I initially came across this feature, I was surprised by the huge number of themes available - over 150!
Popfly Space

'Popfly Space' is the umbrella term we use to refer to all the social network aspects of the site. You can add friends, look at anyone's projects, rip anyone's projects, etc. There are a lot of interesting features that this lets us do. Every person gets a 'rating' which is an aggregation of how their projects have been rated - so you can see who the best creators on Popfly are, what the top projects are and so on. I personally use it as a sort of 'geek social network' :-)
Embedding and publishing

Popfly lets you embed Popfly mashups/webpage pretty much everywhere. Here are some of the places you can embed your mashup in

Any TypePad/MetaWebLog API blog
Live Spaces
As a gadget which you can embed in your Vista sidebar
As an iframe which you can embed..well..anywhere.

I've already seen bloggers embed Popfly mashups in their site. One frequently asked question is - do my blog readers also have to be Popfly users to view these embedded mashups? The answer is no - the embedded mashups are opened up for anonymous viewing. Since you're using an iframe to embed the mashup, you're also ensuring that there are no potential security issues. The mashup runs in its own iframe sandbox and hence can't do bad things to your site.

Popfly Explorer

This lets you upload and share projects built with Visual Studio Express. Note - you can share *any* type of project- Winforms/ConsoleApp/whatever. When you share it, your friends can access it and pull it down to their machines. The only limitation is that you have a 25 mb space limit on our servers currently. This is a great way to share projects without having to send zip files back and forth - a process I'm sure a lot of you are familiar with.

Resources

I plan on writing a series of blog posts to dig into various parts of Popfly. There's some great content available on the web about Popfly - I've linked to a few below.

Overview page and video
Tutorial videos (you need to be a Popfly member to view this). The 'Create a Mashup in 30 seconds' blew me away the first time I saw it.
The official Popfly FAQ
Popfly team blog. You can find links to the blogs of all the team members here. ⁵
Mike Ganotti's post on integrating Sharepoint and Popfly
Chris Craft's collection of Popfly links
Popfly Forums - for any questions, suggestions and bug reports. We monitor these forums very closely and you'll frequently see Popfly team members in the forum threads.

Notes

1. I've spoken about this a long time ago in 'Tyranny of the geeks'.

2. I know that Javascript doesn't have 'classes' per-se but now is not the time to quibble about prototype-based languages.

3. Does anyone else remember this show? I loved this as a kid!

4. Adam Nathan is the man!

5. I just noticed that my blog isn't up there. Need to go around and beat up some people into adding it (or sneak in the change myself ;-) )

Firefox and IE deal with no-cache differently

2007-06-14T23:08:00.001-07:00

I just spent a few hours debugging something that I finally tracked down to IE and Firefox dealing with HTTP caching differently. To make a very *long* story short (and believe me, this was a story of hours in Firebug and Fiddler and VS), IE does *not* cache HTTP pages if you send down the no-cache directive. Firefox does not cache HTTPS pages but *caches* HTTP responses even if you send down a no-cache. The fix is simple - send down a no-store directive instead. So instead of

Cache-Control	no-cache

you send down

Cache-Control   no-store

I eventually found a Bugzilla bug which deals with this exact scenario. It seems to be a contentious issue with different opinions on what the right behaviour should be. I'm just happy that I get to fix it with just a one-line code change :-)

Response.Cache.SetNoStore();

I'm putting it out there so that some poor soul in the future can save some time. :-)

Return of the duck - new Popfly bits

2007-06-14T15:47:00.000-07:00

We just pushed out a major update to our site. You can get the entire list here but the biggest changes for me are the tremendous perf improvements and the fixes for embedding your mashups.

My actual contribution to this release is quite minimal but the entire team has put in some real hard work to get this out the door.

For those of you already inside Popfly, play with the new release. Everything should just work and if it doesn't, it is a bug <grin/>. Let us know if you find an issue (mail me or puff@microsoft.com or post on the forums).

For those of you not inside Popfly, sign up for an invite. We are sending out thousands and thousands and you should get yours real quick. Of if you have a friend inside Popfly and get them to send you one.

Have fun playing with the new stuff. Don't be gentle. :-)

Random comment from our team meeting yesterday - "Let's not get stuck with the evil duck"

Making music

2007-06-05T22:26:00.000-07:00

In the middle of a thread on whether Zooomr stole code from Flickr or not, Stewart Butterfield from Flickr has one of the best comments I've seen on our industry in a long time.

... My favorite analogy to software development and product design is music: it's possible to be entirely original in a context where there pretty much every chord progression, every rhythm, every melody and every harmony has been done before, even if it is combining influences and whole elements from those who influence you....

That's truly what our industry is about - making music. Easy to forget in the blur of Google vs Microsoft vs Yahoo or the latest Techcrunch startup.

P.S On a tangential (and jarring) note, the way Zooomr operates perplexes me. You have a CEO who says he isn't familiar with coding ethics and legality and a sole developer who hand waves away suspiciously similar-looking code as just naming conventions. What gives?

P.P.S As always, do read the disclaimer at the bottom of the page (this is my own opinion, yada, yada)