Pulvis Et Umbra

A Quick Guide To Twitter

2009-06-18T15:17:00.000-07:00

It seems like I have to explain Twitter every few days. While I think it's a wonderful service, I think everyone can agree that it's a bit obtuse. In order to save myself some trouble, I'm writing this post so that I can refer people to it. Call me lazy.

What Is Twitter?

This is a very good place to start. The easiest way to describe Twitter is:

A little bird that tweets in all of your friends' ears.

Basically, Twitter is a service to make communication easier. This is important to keep in mind. It is VERY easy to publicly humiliate yourself on Twitter. One wrong character can make a private Tweet very, very public.

The Terminology of Twitter

You've already probably noticed that I call a message sent via Twitter a "tweet". Here's the exhaustive list of terms you'll find useful:

twitter: (n) a messaging service.
tweet: (n) a message sent via the Twitter messaging service.
(to) tweet: (v) the act of sending a message via Twitter
twitter client: (n) a program used to Tweet on Twitter
follow: (v) indicating that you are interested in a person's Tweets (often used to refer to your group of followers or the people that you follow)
at-reply: (n) a PUBLIC reply on the Twitter service, denoted by beginning with an @username
at-mentions: (n) a PUBLIC mention of someone on Twitter, denoted by having @username somewhere other than the beginning of a message
direct-reply: (n) a PRIVATE reply to someone on Twitter, sent by starting a message with "d username"
tweeple, tweeps: (n,pl) the group of people who follow you on Twitter
retweet: (v) the process of posting an interesting Tweet that you received from someone

Strange Limitations

Twitter, like MySpace before it, is a service that really "happened" more than it was "planned". Because of this, you may notice that it can be a bit rough around the edges. The factors that shaped it have given it some particular warts that I'll try to mention.

Message Length: Messages on Twitter can only be 140 characters. Longer messages are cut off.

Cryptic Commands: Private messages, at-replies, and at-mentions focus a rather crusty mix of single letters and symbols.

Slowness: At times, Twitter can be behind by hours.

Link Mangling: Web-links in messages are rewritten in a way that hides where they point.

It turns out that every one of these limitations comes from one of Twitter's biggest strengths--SMS support. Twitter, from the beginning, has been made to be used on a simple, unadorned cellphone. While this undoubtedly has made it a success, we have to live with the above problems.

Cell phone messages (so-called SMS messages) are limited to 160 characters. This means that messages from Twitter have to be shortened. This is where the length limitation comes from. It's worth noting that messages longer than 160 characters are supported by some phones, but the other phones hold us back. Some people feel that the brevity forced by this limitation is what makes Twitter enjoyable--it forces people to be succinct.

The cryptic commands and link mangling are both ways to cope with the limited space. The commands have to be short, and the link mangling is done to make the web-link shorter.

The slowness is just the nature of the beast. It turns out, that some people have lots of followers. When these people Tweet, every one of their followers must be notified. Today, those two users alone are responsible for over 15 million messages that must be sent. To put that in perspective, if they started at midnight (which they didn't), that would be 264 messages per second!

Suddenly it makes sense why Twitter falls behind!

How To Use Twitter On The Phone

There's a list of commands here.

To send a public message to all of your tweeps, just message the Twitter number (40404 in the US).

To reply to a user named kagato, send a message like "@kagato I liked your omelet recipe.". This message will appear on your public profile as your Latest Tweet, appear in your public timeline, and generally be messaged to that user.

To mention a user named kagato, send a message like "Omelets are AWESOME! Right @kagato?". In this case, the message will appear as your Latest Tweet, be messaged to all of your Tweeple, and generally be messaged to that user as well.

You'll often see mentions like "Hanging at out an awesome Omelet Bar in Vegas with @kagato and @freeformz".

To send a private reply to a user named kagato, send a message like "d kagato Pssst! Omelet's make me gassy.". Be very careful when sending private messages. It is very easy to accidentally type "dkagato ...". In that case, your message would go to all of your tweeple!

To mark a user's last tweet as one of your favorites, send a message like "fav kagato". Note that this will sometimes mark a later tweet than the one that you intended when Twitter gets behind. You can also just send "fav" to mark the last tweet you received (from whoever) as a favorite. Again, this can suffer when Twitter gets behind.

To follow a new user, send a message like "follow kagato".

To nudge a user (silently indicating to them that they aren't talkative enough), just send a message like "nudge kagato". Not many people use this one.

To disable all phone messages, send "off". To enable them, send "on". You can also put a user's name after them to just muzzle a specific person. That can be helpful when people are drunk-tweeting. Just don't forget to unmuzzle them when they sober up.

To find out information about a user, send "whois kagato".

To retrieve the latest tweet from someone, send "get kagato".

To be vain and get information about yourself, send "stats".

To invite someone to twitter by text-message, send "invite 415 555 1212" (substituting whatever their number is).

How To Use Twitter on the Web

Just play around at twitter.com.

Ironically, one you log in, just type a message like you would on your phone. There are buttons for most of the features (following people for instance), but messages, @replies, @mentions, and direct messages all work in the little box.

The search functions here can be nice. The public timeline can be nice. Otherwise, it's about like your phone (at least in terms of functionality).

The most useful feature that is on the website is the ability to control whether you get "mobile updates" for a user. This lets you get realtime updates for some people, while other people only appear on the web (or in your Twitter Client, if you use one).

I mostly have close friends and coworkers enabled for mobile updates. This lets me catch messages like "Anybody up for dinner?". Setting dinner plans over Twitter can be a very cool pick-me-up at the end of the workday.

On the other hand, I have many other people I follow. I occasionally (maybe once every other day) check my Twitter client. This lets me pull in the full volume of Twitter on my own time (instead of over my entire day).

Other Neat Stuff

There are other services available over Twitter.

You can use twitpic.com to put links to pictures in your Twitter messages.

People often put "tags" in their messages for people to search for. Whenever you see a message like "I love Chicken Tikka Masala #food", the #food part is a tag available for searching.

Sometimes you'll see a message like "Just bought some $AAPL because $MSFT is down". These are used to indicate that a word is a stock symbol.

Some people even put links to their location on a map. The possibilities are really endless.

Twitter Clients

If you become a "hardcore" Twitter user, you may download a special program just to access Twitter. If you're really into it, you might even get a special one for your iPhone or BlackBerry.

Twitter Clients are where things get pretty crazy. The functionality like stock symbols, location links, @replies, TwitPics are all integrated into various Twitter clients. From my phone, I can take a picture of something and Tweet about it with a link to the location on a map. It's surreal.

Other Twitter Quirks

The one other thing that always seems to trip people up is retweets. Let's say that you hear about some new thing you like. For example, you might receive a message like "freeformz: A bank car overturned at 4th and King, free money!". You, understandably, might want to send this to all of your friends so they can become wealthy with you.

However, knowing that freeformz will soon be a rich man, you don't want to offend him by implicitly claiming that you are the source of this blessed information. This seemingly simple scenario can be confusing.

Many people re-tweet with a message like "RT @freeformz: A bank car overturned at 4th and King, free money!". The RT indicates that it is a retweet. Some people will do it like this: "A bank car overturned at 4th and King, free money! (via @freeformz)". In either case, it does the same thing.

To complicate things, sometimes people want to add some more to the message. This makes it really confusing. For example, you might see this "RT @freeformz: A bank car overturned at 4th and King, free money!-- I'm RICH!". In this case, it's easy to miss that @freeformz was the one with the initial message and that the re-tweeter added "I'm RICH!".

In other words, read carefully, and watch for signs that someone added a message to the end of a re-tweet.

Conclusion

Twitter is a powerful and useful tool for forcing people to effectively communicate through cryptic abbreviations and haikus. Hopefully it can be as transformative for your time-wasting as it has been for mine.

Good luck, and have a lot of fun!

Interpreting The Evidence on Human Reasoning

2009-06-02T17:47:00.000-07:00

I recently read two largely unrelated articles. Much like french fries and milkshakes, they go together well even though you might not expect it.

If you have a technical bent, read this whole article. It's about a guy building electronic brains, and it's fascinating. If you just want the salty-sweet juxtaposition, just read the first question on the third page here.

Now read this article on the recent friction between science and politics.

What's interesting is the fact that the observations of the first article cast the second article in a disturbing but understandable light. As thinking creatures in what some assume is an age of reason, it's comforting to think of reason as being somehow inherent in ourselves. In a democracy, it's most depressing to conceive that your voting peers might lack that spark of reason that we've hoped drives a democracy towards informed legislation.

The sad truth appears to be that the brain is really just a powerful, dynamic pattern-matching system. While this makes it very generally applicable, it also means that it's possible to train it to match in a way that is wholly incompatible with logic and reason. I guess nurture wins out of nature on this one.

What I find to be most interesting is that this is a clue that many AI researchers are focusing in the wrong places. It's pretty clear that achieving intelligence in a human sense will have to be highly statistical in implementation. While I do sincerely hope that we can someday build a logic processor that is based entirely on semantically-infused logic, I wonder at how such a machine would contrast with humanity.

What Java Should Learn From Ruby and Rails

2009-04-27T11:59:00.000-07:00

I hate to say this, but Java has gotten a bad rap among some programmers. It's deserved at times, but it's worth noting that it's often more about the people behind Java than the technology itself.

Here's an example. Clearly, this is horribly wrong. Sun refuses to fix a bug because people have worked around it, thus it must be maintained--to the detriment of all programmers to follow. This gradual growth of cruft is one thing that has driven people away from Java.

Why is this important to me? Well, it's mostly important because I think that it prevents good comparisons of technologies. It sends the wrong message when people move to Rails because Java is riddled with unnecessary complexity and buggy libraries. While it is some compliment that it says that Ruby is easy to use compared to Java, it doesn't emphasize what makes Ruby such a different way of doing things.

Real improvements like reduction in code through expressiveness, flexible extension of types, the block syntax, et cetera get lost in the shuffle. What's worse, it doesn't send the message that Ruby can be used to solve industrial problems. The eternal mantra of the Ruby naysayers--that it doesn't scale--is at least partially reinforced when people flock to it for simple projects.

For those who have built scalable applications on top of Ruby, this is laughable. Yet the point remains that dynamic languages in general and Ruby in particular have a great deal of value to deliver to real, serious software development. Even though it's fun to play with, it's not just a toy.

Similarly, when people legitimately work on projects like JRuby that work gets ignored by people who just don't want to deal with the cruft. While that's a great reason to avoid a class-library, it's a horrible way to allocate our resources as an industry. In a world where languages live or die by their "standard libraries", this just makes people's lives harder and guarantees that we spend more energy to move the same distance.

I can only hope that Oracle's recent acquisition of Sun might change this attitude and position Java as a more useful technology. It's a shame to see all of the hard work people have put into the JVM be squandered because some people are afraid to break an API. If anything, this could have been turned into an opportunity for Java to offer versioned APIs, but instead it was met with duplication of functionality and degradation of the clarity of their class library.

I feel that this puts work on Rails 3 in an important light. The Merb project was about a lot of things, but one of its most important aspects was that the people behind it continued the march towards a better product. The Rails core team did a great thing by welcoming the best of Merb into Rails. It's letting us build a better future, and it's showing that we can maintain compatibility and still move forward.

We just need to always be mindful that the day that we stop improving Rails is the day that someone starts writing its replacement. More than anything, we need to insist that there is no excuse for not fixing a bug.

Random MacOSX TCP Behavior

2009-04-25T19:41:00.000-07:00

TCP is an interesting protocol. It's interesting mostly because it is less a specification of bytes and more a specification of behavior. Most TCP implementations have developed from the sort of arcane knowledge that you can only amass after trying to implement such a basic protocol over a long period of time.

By the time TCP (and indeed the whole IP stack) has made it to my desktop, it had been on a long journey. MacOSX's network has a storied pedigree that goes back deep into the iterations of the original BSD Unix. There is, as they say, heavy voodoo.

Today I hit a corner case that only made sense after some pretty serious debugging. I'm sharing it here with the hopes that it may save you the headache.

The backbone of the Internet is designed to just have these routes disappear for a while. You get a few ICMP messages back if you're very lucky. Otherwise, your packets might just disappear.

One of the nice features of TCP is that it's incredibly resilient to network links just disappearing. This is no problem for it. In fact, if you're not sending any traffic, you may not even notice that you're down. Failures being invisible is a nice feature when you're not doing anything.

That said, there are protocols that really want to know when they're down. XMPP is one of them. For protocols like XMPP, there is a pretty standard procedure of having some sort of "keep-alive" data that you occasionally send. Since XMPP data streams are just XML documents, most XMPP implementations just send a few whitespace bytes in between stanzas when idle.

Today, I was debugging an XMPP connection over a 3G modem. This manifests itself under a ppp0 link in MacOSX. While I wasn't thinking about it, I walked around with my laptop. One spot caused the phone connection to lose signal and it failed. When I noticed, I reconnected the modem. This provoked some very interesting behavior (or rather lack of behavior) from the BSD IP stack.

Normally, when some fundamental aspect of network changes, there is some device that will interrupt your connection. For example, if my XMPP server had lost power, when it recovered the keepalive packets would have triggered a TCP reset, which breaks the connection. Similarly, if I remove an IP address from a Linux machine, connections on that IP are interrupted. It just so happens that in this case, the IP stack did NOT break the connections.

In fact, it just silently ate any data that the connections attempted to send. So the keepalives completely failed to kill the dead connection. It took almost fifteen minutes until some sort of behavior that caused the IP stack to notice that the connection should be killed.

It took a while to track down what was happening, but apparently the connections were maintained (so says netstat) and the sent packets just disappeared without any sort of sending error! Very weird behavior triggered by an odd corner case. I've also discovered that this appears to also happen when you close your laptop and go to another wireless access point. This is ugly for my use-case, as I want the agent on the laptop to reconnect when it has a new address. If anyone has a good way to detect this in a portable way (i.e. not plugging into Apple's NetKit watchers), please let me know.

While I find this mildly annoying, I have to admit that I can't fault TCP. If the packets are just disappearing, the best behavior is to just resend and keep waiting for the connection to come back up for a reasonably long time out. This is exactly what happened. Instead, I hope that Apple will eventually do what Linux does and push an error into the socket when it tries to send from an address that isn't valid for that machine anymore.

One Year Update

2009-04-23T08:40:00.000-07:00

Who gave this guy permission to grow? I sure didn't! I guess it's okay, though... :)

Here are some good pics grandma Vantuyl got of the birthday boy:

One down, seventeen to go. ;P

On Tail Recursion Elimination

2009-04-23T06:36:00.000-07:00

There was a bit of a controversial post on Guido van Rossum's blog that I thought deserved a little comment.

To sum up Guido's argument, he doesn't feel like implementing Tail Recursion Elimination (henceforth referred to as TRE) in Python because:

Stack traces help debug, TRE makes them useless
TRE Is Not An Optimization (it creates a class of code that explodes without it)
Guido does not subscribe to the "Recursion is the basis of all programming" idea
Due to Python's highly dynamic namespaces, it's very nontrivial to know if a call is a recursion.

The funny thing is that, even though I am a big supporter of TRE, I actually agree with all of these points. Taking them in turn:

Stack traces are critical to debugging code. Ruby and Erlang both started out with uglier stack traces than they have today. They were adjusted because they are important. Similarly, the most difficult Python frameworks have always been that way due to their affect on stack traces. Zope and Twisted--I mean you. Zope would create absolutely titanic stack traces. Twisted, on the other hand, creates virtually no useful stack traces (as a consequence of the deferred model).

While I find this to be telling, I would caution against considering this in the context of TRE. Why? Simple. Some styles of programming don't generate good stack traces. As an example, consider the construct of a loop. Loops don't generate stack traces. When code explodes in a loop, you have no idea what state led-up to that explosion. Does that mean that loops are "bad"? No. It simply means that what we want is not a stack-trace, but rather information about what led up to the failure.

For some classes of code, stack-traces provide that fairly economically. The catch is that other types of code don't benefit from them at all. Guido's statement that you can trivially rewrite any TRE function as a loop is very practical, but it actually detracts from his point. It's pretty clear that the type of code that benefits from tail recursion doesn't benefit from stack-traces. This means that effectively he's deciding that Python just doesn't tailor to a certain type of code. I'm mildly opposed to this and I think there's a good case for a type of TRE in Python.

Taking the second point, Guido has this right. The fact that there is code that would explode without TRE is a good reason not to just add it to the language. That said, it doesn't mean that we shouldn't do it. In fact, it is largely an argument to do so in the main Python runtime. If anything, fragmenting the Python codebase is negative. However, TRE tailors to a certain type of code, and the fragmenting effect of that reality means that he is really choosing between fragmenting the Python codebase and driving people to other languages that handle the use-case better. He might be okay with that, but I don't really believe it to be necessary.

Taking the third point, talking about the "basis of all programming" is a little tough. From a very high level, design a language without recursion. What's left that defines programming? In all seriousness, I think that the statement that "recursion" is the basis for all programming has a strong case assuming that you mean "recursion" in terms of the function call (or subroutine, or whatever you call it).

Without functions, you don't really have anything but a long list of instructions. All control statements are pretty much functions or branches. Branches without the concept of subroutine are effectively goto statements, which have always been as controversial as they can be useful. The point being that TRE allows you to turn one of the most fundamental constructs of programming and use it in a way that increases its utility. If recursion isn't the basis of all programming, it's still pretty fundamental.

Addressing the final point, I recall part of the Zen of Python, "Explicit is better than implicit." The statement that Python has difficulty (at compile time) determining whether a call is recursion is a good one. There's a really simple solution to this. Add a keyword to make tail-recursion explicit. Perhaps we could propose abusing the pass keyword. Basically, where pass is used now, consider it to mean "pass None". The idea is that it is either passed a function call or None to indicate tail recursion. The nice bit of magic here is that it reuses an existing keyword compatibly, clearly indicates what's happening, easily translates to bytecode, and works really well for TRE's most powerful use-case (coroutines).

So, as an example, consider the following code:

class cli:
  def start(self):
   print "Welcome to the CLI"
   pass self.loop()

  def main(self):
   print_prompt()
   cmd,args = parse_line()
   cmd = getattr(self,'cmd_' + cmd,None)
   if cmd is not None and callable(cmd):
   pass cmd(*args)
   else:
   print "Command not found"
   pass self.start()

  def cmd_help(self):
   ...
   pass self.main()

...

The above code may seem slightly convoluted, but it's actually interesting because it tackles one use-case that Python is bad at. Admittedly, the above could have been factored as a loop, but the more steps you insert into the process, the uglier it gets. What is the purpose of continually having to implement some sort of mini-scheduler in a loop? Why have all this convoluted logic that has to handle a random series of states and detect an "end-state". This is just simpler, and more importantly "Beautiful is better than ugly."

Critics might say that loops give you an easier understanding of the top-level flow of the program. I would contend that any problem that needs the above technique would instead result in a loop so complex that it would generally serve to reduce the understanding of the problem--replete with various ifs, returns, whiles, and breaks.

In assembly, C, or any number of other systems, this would have been implemented with goto statements. The problem with goto statements is that they very easily obscure the boundaries of single bits of code and can poorly document the intent of the goto. This is a nice abstraction that allows the use of goto-like functionality without losing information about what's supposed to be going on. There comes a point where the above would get factored into a general state machine implementation, but it's not really necessary.

Another thing to notice about the above is that it gives you a really good point for debugging. Any Python implementation of the above type of problem would not have stack-traces to debug. However, the pass statement gives you a place to track behavior. In a debugger it's pretty clear what's going on. At the beginning of each iteration, the function arguments clearly define the state of execution.

This is something that is much more useful that implementing this with a loop. With a loop, the state is invisibly contained in the current binding namespace. Temporary values easily leak. There's not a clear point where the state of the iteration is distilled. This is why people want TRE.

There are just a lot of situations where people would benefit from greater control over the constructs of flow control. Right now we've already abrogated the model of nested-call-return with generators. Most control structures that are difficult in Python could benefit from further disentangling the assumptions of function calls.

Someday, I would love to see the work that was started with generators end with easy coroutines, promises / lazy evaluation, message passing, and smarter code replacement. Every language has idioms for what a recursion or evaluation means. We have the opportunity to make this handling explicit, and I suspect that we would benefit from it.

Making Sense of Erlang's Event Tracer

2009-04-18T06:01:00.000-07:00

Let's face it people, while Erlang is wonderful in a number of ways documentation is not one of them. I mean, it's there, and it's better than some other projects. However, there are incredibly useful things staring you in the face that are just impenetrably difficult. The et module is one of those.

Fear not, gentle reader. After more time than I'd care to admit, I've managed to figure out roughly how the pieces fit together. As usual, they exhibit the combination of weirdness and inspiration that have driven us to embrace Erlang. Without futher ado, let me take a stab at laying out exactly how to actually use it.

Four Modules

The event tracer framework is made up of four modules:

et
et_collector
et_viewer
et_selector

In addition, you'll probably want to familiarize yourself with the dbg module and possibly seq_trace module as well.

The Event Tracer Interface

This is perhaps the most confusing module. It contains a function that you call to report arbitrary events to the tracing interface. The method is named report_event. There is also a humorous alias named phone_home. Get it? et:phone_home? Yeah, it makes me sad, too.

Ostensibly, they're supposed to be called with options something like:

report_event(85,from,to,message,extra_stuff)

The number (in this case 85) is an integer from 1 to 100 that specifies the "detail level" of the message. The higher the number, the more important it is. This provides a crude form of priority filtering. Avoid using 100, since it seems to disagree with being displayed in the viewer.

The from, to, and message variables are exactly what they sound like. From and To are visualized as "lifelines", with the message passing from one to the other. If from and to are the same value, then it is displayed next to the lifeline as an "action". The extra_stuff value is simply data that you can attach that will be displayed with someone actually clicks on the action or message.

In a perfect world, this would be enough to start to get your hands direty. We do not live in that perfect world. Oddly enough, these functions do absolutely nothing. Let me repeat that. They do nothing, except return an ominous atom hopefully_traced. This confused me at first, and let me down the long rabbit hole that is the event tracer.

Their purpose is to be traced. Rather than having them do something, they're just there so that the system can notice when you call them (via tracing) and use the data accordingly. While this is a great idea it has three faults. The first is that it makes the process rather magical. The second is that Erlang tracing is a seething pile of pain that involves reasonably complex knowledge of clever ports, tracing return formats, and specialized tracing MatchSpecs (which are really their own special kind of hell). The third problem is that this conspires to make an incredibly useful and reasonably flashy feature inaccessible to users.

The Collector and Viewer

These two pieces work in concert. Basically, the collector receives trace events and processes them. The viewer interrogates the collector and displays a relatively nice, interactive representation of them. As usual, it's in TK, which is a shame.

You might wonder why these aren't just one module. It turns out that, in typical Erlang style, the collector is a generic full-fledged framework that allows processes to "subscribe" to the events that it collects. This would probably be really useful if there were any other subscribers. Someday I may write one to help debug some code, but for now just trust that it really is a pretty interesting architecture.

You also have the advantage that the viewer creates a collector for you. With a few options and some debugging settings you can start collecting events.

The Selector

This is perhaps the most useful and frustrating module in the entirety of the et suite. This is mostly because it isn't really mentioned in the User's Guide. It turns out that the collector needs "filters" to convert the raw trace data into "events" that it can display. The et_selector module provides the default filter and some API calls to manage the filter pattern. Due to the architecture of the collector, this module is quite a bit of a bear.

Effectively, it's a mishmash of functions that achieve the following:

Convert Any Trace Message Into An Appropriate Event
Magically Notice Traces of the et Module and Make Appropriate Events
Carefully Prevent Translating A Message Twice
Manage A "Trace Pattern"

It turns out the there are a few strange interactions. First of all, you still have to tell the system to trace the right things. Secondly, without the correct incantation you will get infuriatingly strange "trace" events instead of the nice events you're reporting with et:report_event/5. Finally, the documentation doesn't do a decent job of telling you what the trace pattern even is.

How To Put It Together

It turns out that the collector automatically registers itself to listen for debugging events, so all you have to do is enable them.

For those people who want to do general tracing, consult the dbg module on how to trace whatever you're interested in and let it work its magic. In my case, I just wanted et:report_event/5 to work, so that's what I'll illustrate here. I did the following:

Create A Collector
Create A Viewer (this can do step #1 for you)
Turn On and Pare Down Debugging

The following module achieves this.

-module(trace_test).

-export([test/0]).

test() ->
  et_viewer:start([
   {title,"Coffee Order"},
   {trace_global,true},
   {trace_pattern,{et,max}},
   {max_actors,10}
  ]),
  dbg:p(all,call),
  dbg:tpl(et, report_event, 5, []),
  Drink = {drink,iced_chai_latte},
  Size = {size,grande},
  Milk = {milk,whole},
  Flavor = {flavor,vanilla},
  et:report_event(99,customer,barrista1,place_order,[Drink,Size,Milk,Flavor]),
  et:report_event(80,barrista1,register,enter_order,[Drink,Size,Flavor]),
  et:report_event(80,register,barrista1,give_total,"$5"),
  et:report_event(80,barrista1,barrista1,get_cup,[Drink,Size]),
  et:report_event(80,barrista1,barrista2,give_cup,[]),
  et:report_event(90,barrista1,customer,request_money,"$5"),
  et:report_event(90,customer,barrista1,pay_money,"$5"),
  et:report_event(80,barrista2,barrista2,get_chai_mix,[]),
  et:report_event(80,barrista2,barrista2,add_flavor,[Flavor]),
  et:report_event(80,barrista2,barrista2,add_milk,[Milk]),
  et:report_event(80,barrista2,barrista2,add_ice,[]),
  et:report_event(80,barrista2,barrista2,swirl,[]),
  et:report_event(80,barrista2,customer,give_tasty_beverage,[Drink,Size]),
  ok.

Running through the above, the most important points are:

Turn On Global Tracing (it doesn't work for me if I don't)
Set a Trace Pattern
Tell The Debugger to Trace Function Calls
Tell It Specifically To Trace The et:report_event/5 Function

The Aftermath

The most vexing part of figuring this out was the trace pattern of {et,max}. The trace pattern is basically a tuple of a module and a detail level (either an integer or the atom max for full detail).

The specified module flows from your instantiation of the viewer, to the collector that it automatically creates, gets stashed in as the trace pattern, and eventually goes down into the bowels of the selector. Tracking this down sucked.

This fact is documented basically nowhere. It also doesn't make a lot of sense, either. The module that you specify gets passed down (eventually) into et_selector's default filter. The format of the report_event/5 function call is hardcoded in that filter. That makes it very hard for me to imagine why you would ever specify another module. I suppose you could replace the default filter, but it turns out that doing so is pretty obtuse to do even if you wanted to. At any rate, just pass it through and it works.

That said, I think it was worth it. If you compile the above example code and execute trace_test:test(), you'll see something like the following screenshot. Source code here, screenshot here.

Beautiful. I think I'm going to get a cup of coffee now.

Reigning In LVM pvmove Memory Leakage

2009-03-10T02:36:00.000-07:00

If you use the Linux Logical Volume Manager, pvmove can be a wonderful utility. It lets you move a logical volume (or many, in parallel) from one physical volume to another. Engine Yard was recently attacked by raptors. To help deal with this, we needed to move off of the disks before they exploded. pvmove to the rescue.

It turns out that we hit a few bumps. The biggest was that the pvmove command would slowly consume massive quantities of memory until a node would run out. This was both vexing and confusing, as we were pretty sure that the pvmove command wasn't really doing any of the actual moving work.

Some digging in the source code indicated that the pvmove command utilized some sort of generic polling daemon implemented within LVM. A little more inspection showed that this daemon would scan the volume groups for an in-progress move, then would check on the progress of the move. This was done on some interval (15 seconds by default).

It turns out that the functions that scan the volume groups don't really release memory properly (at least not the way they are currently used). In most LVM invocations, this is fine. The data is only loaded once each invocation. For pvmove, this caused a leak of memory every interval.

While some deep work on the source code could eventually fix this memory leak, we took the easy out. We realized that you can just crank up the polling interval and it cuts the rate of leak to an acceptable level. On our test server, we show the leak around of 120MB per hour at the default 15-second polling interval. Increasing the polling interval to five minutes reduces the leak to less than 10MB per hour, at the expense of an average of 2.5 minutes of "wasted" time between pvmove invocations. Not too shabby.

Changing "pvmove /dev/a /dev/b" to "pvmove -i 300 /dev/a /dev/b" makes a big difference.

Hopefully this can help out some other people. Enjoy.

Maildirs, Timestamps, and Metadata Indexes

2009-02-08T19:13:00.001-08:00

Every once in a while, you have a problem that has an approachable solution, but you just don't want to tackle. This weekend, I took a brief break from a more significant problem to attack one that I've had since February 20, 2008 at about 12:35pm.

You might ask, how do I know the date so precisely. Well, it's been staring at me from my e-mail Inbox. Essentially, around that time I was getting ready to shut down my e-mail server I had run out of my cybercafe for a while. In doing so, I had to choose to move somewhere. I tried GMail. While GMail is a great webmail client, it's IMAP implementation leaves something to be desired, so I quickly decided to move somewhere else. Being short on time and not having good luck with the three other people I checked out, I settled on running my own on a small VPS at a decent provider.

In order to get the mail back out of GMail, I used imapsync, a script that will dump an IMAP mailbox into a Maildir. At the same time, I switched to the Dovecot IMAP server. I had previously used it, but had never wanted to fight with Cyrus IMAP, which I had been using before GMail. I figured, what could possibly go wrong?

Well, what went wrong was fairly understandable. Dovecot decided that it would use the filesystem timestamp on the files instead of their Date header. This created the interesting situation where all of my imported mail showed the time that it was downloaded from GMail, but paradoxically showed a different Date header.

This has been more than a minor nuisance. Essentially, when I have wanted to read an old conversation thread, it's all out of order. Messages downloaded in a strange order that didn't exactly match the receive order, so threads were often a challenge to read. Additionally, any really old mail couldn't be found by sorting by Date, so I was also challenged when finding messages that I can't remember their exact content, but I can remember when they were received.

This was an entirely ignorable problem, with a reasonably tractable solution. I just never really wanted to tackle it.

While my parents were here for my birthday, Mom (who also hosts her e-mail with me) remembered that she was having a similar problem. Since I got stuck on a different problem, decided to tackle this.

If anybody ever has a problem where the IMAP server dates their e-mail by stat (i.e. mtime) instead of RFC822-style date, here's a script for you. Should work in Python 2.5.

fixdate.py

UPDATE: Added fix so that Ctrl-C actually stops the script. Thanks to Noah Kantrowitz for that one.

What If There Were No CDs?

2008-10-26T22:18:00.000-07:00

I was just retagging some of my music. I ran across an interesting set of unexpected issues with Sgt. Pepper's Lonely Hearts Club Band (by the Beatles, duh!). The end was cut off of some of the song names, so I wanted to fix it up. I looked up the album on the Internet and began fixing song names. Immediately, I noticed that the number of tracks in my list didn't match the number of songs on the record.

After some headscratching I realized that the songs were listed by side of the record. After smacking my forehead, I started thinking about how iTunes indexes songs. It has metadata for multiple discs, but not different sides. My immediate thought was, "What if there had never been CDs? Would there be extra data for which side of the disk the song was on?"

After pondering this deeply philosophical alternate history, I noticed on the above Wikipedia article that the original release had some interesting quirks. Specifically, after the last song was a 15 kilohertz tone designed by John Lennon to annoy people's dogs, followed by some jibberish studio noise put into the gutter track. So, the original record would end by annoying your dog and then playing gibberish endlessly.

Apparently the CD re-release only repeats the noise eight or nine times and then fades out, since CDs can't really simulate an infinite loop like you get out of the gutter track. Similarly, I'd imagine that the 15 kilohertz tone might not survive the MP3 encoding process. Even though the tone and gutter track magic are really just tricks of the vinyl, it's interesting to think that these aspects of that album are effectively forever lost on future generations of Beatles fans.

Oh, well. Endlessly repeating gibberish probably doesn't have much of a place in the world of iPod playlists. Still, I can't help but feel that a little something is missing...

Erlang Error: {open_error, -12}

2008-10-14T20:10:00.000-07:00

This error happens when you have old or missing .so libraries for your Erlang app. It is rarely mentioned on the Internet and can consume hours of wasted debugging time. Now you know. :(

No to Bail-Out; Yes to Buy-Out

2008-10-04T00:26:00.000-07:00

Leave it up to an opinion column in Bloomberg to perfectly articulate what I've had brewing in my head.

I kept thinking, "How does putting money into the banks fix the problem that these assets are overvalued?" Unless we are buying these securities, and then writing them down to a corrected value, nothing is fixed. It's quintessentially anti-market. The properties aren't allowed to go down in value.

Ron Paul hit it on the head, it's price-fixing. As with most of Ron Paul's theories, I don't completely agree with his solution (i.e. the Federal Reserve and paper currency is the root of all evil). I think that there's a balance to be struck between letting banks set prudent interest rates and preventing a predatory banker's trust to form again. Pure-free-market types often forget that freedom, liberty, and justice are rarely selected by the market.

At any rate, it appears that this has been solved before by buying an interest in the bad securities, rather than just extending the failing banks more credit (i.e. letting them dig the hole deeper). Despite a deep understanding of all of the issues, this solution already is a favorite on Main Street.

The right solution seems to have worked for other people before, where the wrong solution didn't work so well. Of course, they eventually figured it out and appear to have learned something in the process.

iCal and Automator

2008-10-02T02:28:00.000-07:00

Recently discovered something kind of cool. Apparently, you can save an automator workflow as a "iCal Alarm Plug-In". Then you can select it as the "alarm" for a message. Depending on your bent, this makes either a very elaborate alarm or a poor man's crond.

Rather skimpy details here.

Sidestep The Little Bits of History Repeating

2008-09-28T04:57:00.000-07:00

Is it just me, or does this make anyone else nervous.

Read this.

Then read this.

Deja vu? Is it just me, or is this exactly history repeating itself. The most disturbing confluence is:

Cheap Credit
Bad Loans
Reinvestment in Real Estate

Basically, credit gets cheap. Banks compete too much because of the cheap credit. Bad loans are made because of the competition. Given the cheap credit, people reinvest in the real-estate, essentially refinancing themselves so it is never paid off.

In that environment, it was just a matter of time until bad loans creeped unacceptably high. Even worse, naive interpretation of loan failure statistics gave a false sense of security. When times are good, the bad loans look okay because they aren't failing. As soon as the economy contracts, everything goes down like a house of cards.

The crazy bit is that the issuing of bad loans is incentivized. It's a Loan Officer's job to make loans, under-producers are fired, banks compete, and the process of elimination wipes out any Loan Officers with restraint. As the banks each try to get an edge, they find ways to creatively relax loan requirements. When all is done, whoever loosened the loan requirements is at fault, leaving the investors in the bank, possibly the depositors, and the loan-holder picking up the pieces.

While it would be nice to point the finger at the Loan Officers, company boards, or some middle management, I bet that most of the time it wasn't even a single person, but a group of people working uncoordinated. Sometimes it would have been a large investor who was overly pushy trying to squeeze out some more stock value. Other times it would be a loan officer struggling to cram in a deal that was already borderline. Maybe it was a bank manager trying not to come up on the bottom of the list.

As usual, the wisdom of the market can make major things happen, but it doesn't prevent abuse. It's a raging inferno that's just as content to power our economic engine as it is to blow the engine apart.

At some point all of Wall Street was silently overrun with people who's predictions, worse-cases, and guarantees weren't based in reality. They should have been the last line of defense. Some auditors or fund managers should have raised a flag--and their management should have taken them seriously. I have a hunch that nothing currently being considered can really address that--the human tendency to avoid bad news runs deep.

The Nikon D700 Rocks!

2008-09-25T04:04:00.000-07:00

So, I picked up a Nikon (a D700 specifically). It was pricey, but I figured that David would only be tiny once, so I should get quality pictures. After a deep breath, I signed the check and went home with my equipment.

I don't regret it. See my photostream on Flickr for the goods.

Useful Mac Script

2008-07-13T16:34:00.000-07:00

When I used to use Windows more regularly (i.e. Windows NT), one of the most useful commands was the "start" command. It would behave much like the "Run" dialog. The number one thing I used it for was to open the current directory (or another directory) in the Explorer.

When I moved to Linux, this was still relatively possible (using konqueror or gmc from the terminal).

Upon moving to the Mac, I've never found anything that does it very well. I googled around for something today, and still found nothing. So I thought a little bit and wrote one.

You can download it here. I called it "reveal". It does some very simple (by no means comprehensive) URL escaping on the specified path (or current directory if none is given) to create a file URL. Then it uses osascript to tell Finder to open that location. Simple.

UPDATE: I added a line to bring Finder into the foreground.

UPDATE++: Apparently there's already a command to do this (open). Appeared in NextStep. Curse you Google!

Preying On Youth?

2008-07-05T21:52:00.000-07:00

Being fairly unconventional in my youth, I have some exciting stories. One time I spent my birthday money on "Subliminal Tapes". The theory was that they had hidden recordings that played directly into your subconscious.

That sounded like it made sense to me (being about 12), so I purchased a few. Two that I recall were one to improve my bowling game (I was in a kid's league at the time), and one titled "Do It Now!". Even at 12, I knew I was a procrastinator. It was a harmless purchase, even if a naive one.

At that point I clearly had the idea that I was spending my money, and what it meant. A few years earlier, not so much. I had previously fallen victim to a much more insidious scheme. In the later case, "Subliminal Tapes" were intended to play upon the naivety or curiosity of adults--people who should know better. People who would know to read the fine print.

Here's the scenario. Being a 10-year-old, I heard a commercial on the radio for a phone number that would get me a message from Santa Claus. How could I resist? I mean, I wasn't so certain about his existence at the time, but I thought it sounded like it was worth calling. Of course, there was some boilerplate about having your parents' permission, but it obviously didn't (and wouldn't) have a huge impact on a 10-year-old. So I called this number. I'll give you a hint, it started with 900.

That's right, instead of a 900 number being used as God intended, for audio porn and hookup lines, it was instead being used to drain the wallets of unsuspecting parents. So, after about ten calls to a repeating automated message, my parents figured it out, talked to me about it, and I learned that I had been a party to their fleecing; and that Santa Claus doesn't have a phone.

Not cool. In fact, later, these people were pretty much categorically sued into the ground by various States' Attorneys General. It stands under the some very old, and very sound, legal reasoning--i.e. if you put a cookie in a bear trap in your front lawn, you are liable when a child gets caught in it.

These days, I'm a recent parent. I think it's important to supervise my son and my step-daughter. A kid can't raise itself. Television is not a substitute for supervision. That said, I don't need some greedy schmuck making parenting more difficult either. As an analogy, just because I don't let my kids smoke doesn't mean that I'd be thrilled about Mickey-Mouse-themed cigarettes (note, Mickey Mouse is wholly owned by the Disney Military-Industrial Complex).

You can imagine my dismay when I saw a certain commercial on television. Was what I saw an attractive nuisance? No. Was it designed to prey upon parents who raise their children with a television? Yes, it was. In fact, being shown on the "Teen Network" was direct, if brazen, targeting.

What was it, you ask? It was so simple, and yet unoriginal, that it must be the work of some sort of greedy idiot savant. Imagine the Magic 8-Ball accessed via a cellphone text message. That's right, you text your "question" to "Ask The Phone", and you get cryptic, generic responses back.

If you read the fine print, it says you must be 18+, or 13+ with the "wireless subscriber's permission". Oh, and it's $9 per month for 5 questions, plus $2 each additional question (with varying differences for different carriers). Wow.

Should this be outlawed? No. Should these people be shut down? Not directly. Should it be possible for a small, determined group of parents to make these people suffer in court until they closed down? Very probably.

Genius Speaks For Itself

2008-06-27T21:16:00.000-07:00

As I'm sure everyone has heard already, George Carlin died this week (picture kindly borrowed from Wikipedia).

We all choose to remember the people that influence us in different ways. George is easy to memorialize because so much of his work is so faithfully recorded.

I was able to find my favorite one of his routines on YouTube. I've included it here for your viewing.

Make sure you listen all the way through. He insults a lot of people early on, but if you listen to his message, you'll see that his message, while dark, actually transcends most of the rhetoric on the subject.

Crash!

2008-06-16T14:15:00.000-07:00

Saturday was eventful. A MUNI train rear-ended another one in front of the apartment. It was before a game, so there were tons of people around and it was a bit crazy. There were 6 ambulances at one point. The track was running later that day.

It's hard to appreciate the damage from the pictures. Keep in mind that these trains have a giant steel connector that sticks out of each end about a foot or two. For these trains to be that close, it means that two of these assemblies have met and crumpled enough to shorten the whole thing by four or five feet. Two giant steel assemblies that are each hefty enough to hold multiple cars together--CRUNCH!

Random Erlang Article

2008-06-02T00:54:00.000-07:00

We really should have tried harder to hire Yariv. He's an awesome guy. I recently found this on his blog.

It's an excellent comparison of Erlang and Scala. However, it's even better as a very concise, succinct explanation of how the pieces of Erlang fit together to make it so useful.

A Sad Goodbye

2008-05-03T15:19:00.000-07:00

For those of you who haven't heard, Daniel Joseph Dougherty died Friday from complications from his cancer. The specifics probably don't matter. For those who want to know, his heart was stopped by a blood clot, possibly from bleeding within his heart. The story I got was muddled.

Ironically, I was actually looking at plane tickets to go see him shortly before he died. He had requested that we make it out so that he could see us and his namesake, David Joseph, before things got any worse. As you might imagine, this has hit Alicia and I pretty hard.

Before we moved out here, he actually intended to come out with us. It seems like only a month ago--actually it was only a month ago--that he was dreamily thinking of starting an exciting new chapter of his life in San Francisco.

He actually sent out his bicycles and his most prized possessions with us when we came out. Being Joey, it was mostly antiquated but somehow unique old computer hardware.

He will be missed.

I am currently unclear as to exactly where the funeral and visitation will be held, as I've alternately heard it would be at (I think) the First Christian Church or at Waters Funeral Home, in Vandalia, Missouri. The visitation will be from 5pm to 8pm on Wednesday, May 7th, 2008. The funeral will be the following morning at 10am on Thursday, May 8th, 2008. I am currently assuming that the visitation will be at Waters and the funeral will be at the Church.

Funeral home is at 500 Main St in Vandalia. Here's a (hopefully accurate) map:

View Larger Map

The church (assuming I have the right one) is at 205 W Park Street, also in Vandalia, Missouri. Here's another (also hopefully accurate) map:

View Larger Map

I'll edit this post with updates as I get more information.

A Warm Welcome

2008-04-16T21:21:00.000-07:00

I'd like to introduce you to David Joseph Vantuyl. He was born April 15, 2008 at 8:39 PM PST. It was a warm spring day in San Francisco, California. He weighs in at 6 lbs 5 oz and is 19 inches long. He is also disarmingly cute, as can be seen in the pictures here.

He's healthy and is generally happy (unless tickled).

First Post, Again

2008-04-02T20:42:00.000-07:00

Well, my old blog is effectively gone. I could recover it, but I don't have the time. Wasn't that much on it anyways.

Let's see how well blogger handles remote blogs. I'm kind of curious actually.