a life of coding

Improving Common Tag : Worse is Better

2009-06-12T10:54:00.005-04:00

Common Tag is an "open tagging format developed to make content more connected, discoverable and engaging" [commontag.org]. It mixes RDFa into XHTML to add metadata to specify metadata for a content block, most importantly a link to a common database entry that iconifies the topic like the one word tags used in many places. This is an improvement over word tags, which can be non-descript or ambiguous: does "apple" refer a fruit, a computer company, a record label, or someone's name? Tags that are acryonyms may have no meaning to the user. RDF is commonly referred to as the Semantic Web, because it helps computers link concepts together. Everyone wants the Semantic Web, but somehow it never happens... maybe its because RDFa looks like this:


<body xmlns:ctag="http://commontag.org/ns#" rel="ctag:tagged">
    <span typeof="ctag:Tag" rel="ctag:means" 
         resource="http://rdf.freebase.com/ns/en.u2"/>
</body>

This is a very explicit piece of data. Much of its content is XML support structure. The semantic knowledge contained in there is:

the tag for this span is "en.u2" in freebase

The structure contained in there (removing any content) is:

xmlns:ctag="http://commontag.org/ns#"
rel="ctag:tagged"
typeof="ctag:Tag"
rel="ctag:means"
resource="http://rdf..com/ns/"

There are a lot of things that can break without actually removing any semantic meaning. If there were a typo anywhere in the structure above, your tag would be hopelessly borked - all that work (and bandwidth) for naught. More importantly, this format says that if I want people to understand my tags I have to embrace XML, and there are few things that I dislike as much as XML. Look at all the structure required because of XML, and the complicated tools that are required to manipulate XML!

Lets take that content and put in something sexier: simple HTML. Over at Hacker News, someone suggested

<p ctag="wikipedia/The_Beatles">We're talking about The Beatles here</p>

I like this direction, but its lacking in a gruesome way: ctag is not a valid HTML attribute. Browsers may not like it, and they certainly won't be able to read it, so it isn't as clean as the RDFa. How about this:

<p class="-ctag-wikipedia-The_Beatles">We're talking about The Beatles here</p>

This is valid HTML, browsers can operate on this content, and you can even style it! This is much better than the previous suggestion (which was quite good, and spurred me to write this article), but is it better than RDFa?

Content:

the tag for this span is "The_Beatles" at Wikipedia

Structure:

class="-ctag-"

It is by far more concise than RDFa, but it has limitations - the tag content has to be valid inside a CSS class, which means alphanumeric, dash, and underline. There is additional flexibility if you use backslash, but this is uncommon in CSS classes and may not play nice everywhere. Most significantly, it doesn't include a link to wikipedia, only the name, and Semanic Web people really dislike that. I suspect that most people will link to Wikipedia, and if not, a search engine can figure out the most likely host. I mean, how "smart" are your tools when they can't deduce the meaning of wikipedia? If you're using an internal host or very specific database, you can always fall back to RDFa.

Worse is better. The CSS tag format I propose is not as specific as RDFa, but it is easier to implement, harder to mess up, works with non-XHTML, and easy for humans to verify. These generally overlooked and undervalued qualities make adoption easier for people, which is in the end all that really matters.

The NYTimes Doesn't Understand Social Networks

2009-05-04T11:53:00.003-04:00

In "Tinker Away, Facebook Says", the Times summarizes Facebook's recent API expansion announcement with a comparison to a grocery store that "props open the front door and invites everyone to come in, take the merchandise free of charge, and then give it away themselves". They call this "the counterintuitive business wisdom infecting Silicon Valley these days", and proceed to imply that Facebook is doing this from the kindness of their heart.

Let me suggest that Facebook is not a charity but a profit driven company, seeing the consumer primarily as an unavoidable obligation. An API expansion attracts additional users and (more importantly) encourages current users to not leave. A permissive developer API allows more people to participate in innovating their platform for little expense to Facebook. These companies not affiliated with Facebook take risks in developing features that users might want. If users like these features Facebook can incorporate them into the product, otherwise they will wither and die without tarnishing the Facebook brand.

Developers see this as a way to capitalize (however meagerly) on the success of a powerful brand. Consumers see it as a new channel to cater to their needs. What would be better for both developers and consumers is a federated system with non-centralized intellectual property ownership, and thats exactly what Facebook intends to prevent. The most important thing that you should know concerning any Facebook API is that one thing remains the same: Facebook owns its customer's data, and no one, not even the customer, is allowed to export that data from them.

To an extent, this bothers customers. Facebook recently changed their terms of service to say that Facebook owned the property rights all user entered content. Since this would have been catastrophic to artists and writers, there was a substantial backlash that resulted in Facebook rescinding this clause. They additionally apologized and claimed that this was never the intent. It seems unlikely that they would attempt to own all user content in this way, and I believe that they were honest in saying that this was a mistake. What the terms should have said is that Facebook claims ownership of the format and
context of data as it appears in Facebook. You own your posts, but the only way to use them outside of Facebook is to retype them by hand. You own your photographs, but maybe only if you have the originals. And under no circumstances can you export Facebook's bread and butter: the relationships that you have established.

And so Facebook's goal is to make you just happy enough that you won't jump ship. They will claim as much ownership of content that they can for the sole purpose of making it hard to leave. They will make it difficult or impossible to remove content. They will sell as much of your identity as advertisers will buy without the government getting involved. These are what the investors of social networks talk about, and the only way that they can survive as free services.

I am sad to see the Times miss what is blatantly obvious to me. Is it not the age old motto of investigative journalism to follow the money? It may be that our journalists have lost their curiosity. Or possibly that people have generally become consumers, and consumers see themselves not as powerful individuals but as reflected in their relationship to companies. With Social Networks, the money is in owning customer data as much as possible, and selling that data (via keyword advertising, or outright) to advertisers.

If I impress anything on you, let it be that Facebook needs you more than you need Facebook. They need you to buy products so that they can sell advertising, because Facebook is not really a free service. Years from now, the person who owns your past (photographs, notes, messages with friends) will be the one paying for it now, and if you are a Facebook user, that person is not you. The best thing that you can do for your future self is to get involved and start owning your present identity. Data in a product like Wordpress, Blogger (but not Blogspot), or simple HTML on a web server has been paid in full. There is no question about who owns the rights to your past, or what you are allowed to do with it - it is simply yours. Certainly, Facebook has extra features that you cannot get from these truly free products, but comparable features won't exist in other products (free or consumer paid) until there is a demand for them.

So get out there and encourage people to think about the past of the future, which might be tomorrow. Maybe even use Facebook to spread the word - but don't be surprised if they shut you down. After all, they are the ones footing the bill.

Firefox: An Acceptable Cross Platform GUI Toolkit

2009-01-07T13:47:00.002-05:00

A recent Hacker News submission asked what an "acceptable" cross-platform GUI toolkit would be. It discusses common complaints about the existing toolkits GTK+, Qt, Tk, and wxWidgets. It suggests creating a new toolkit with the qualities of being written in C, keeping it simple, LGPL licensing, easily skinnable for different OS's, binding for scripting languages, and be "simple and easy to use".

Good news! I know of an existing GUI toolkit that fits this bill. Its written and extensible in C/C++, has liberal licensing, is very skinnable, has bindings for JavaScript, Python, Java, Perl, and Ruby, and lots of existing open source code that you can learn from. It even has a large, mixed platform install base, so you know that the bugs are minimal. Its based on an object brokering system called XPCOM, and an application framework called Mozilla, but often goes by the name Firefox. Given the popularity of web applications in the startup community, Firefox seems like an obvious option for desktop apps. Yet it hasn't caught on, and I'm not sure why.

I recently worked on a desktop application for my day job. Given their limited interface goals and a game-oriented 3d engine, they had decided to write the interface directly in OpenGL. It was difficult to design interfaces like that, so I did a prototype integration with wxPython. That worked well enough to start real product development, during which I realized that wxPython had many of the "cross-platform toolkit" woes like needless internal complexity, difficulty in writing new components, and general ugliness.

I spent some time contemplating how people wrote successful cross platform desktop applications, and the two things that struck me were (a) there aren't many, (b) except Firefox, (c) and this might underly the popularity of web applications. So I did a new prototype using Firefox with a XUL user interface. What I found was that Firefox worked out great as a standalone application with modular code design, but it was difficult to write good interfaces in XUL. XUL seems to be well tested only as far as the functionality in Firefox is concerned. We switched to Ext-JS, started pretending to be web app devs, and got down to implementing features.

Writing a desktop app based on XULRunner has some interesting side-effects. For one, it makes retargeting your code as a web application or Firefox plugin really easy. It also lets you develop using a variety of hacker friendly languages. Like programming in lisp, it changes your perspective on how applications should be developed, and makes questions like "which UI toolkit is the best?" seem fundamentally flawed. In an age when Apple is rewriting their apps to run in a browser (and still look and feel like desktop apps), isn't writing a code against a desktop UI toolkit fighting the tide?

JavaScript Performance

2008-12-10T13:48:00.007-05:00

There are a lot of JavaScript performance benchmarks flying around out there. Some claim that Chrome shames the competition, others say that Firefox 3.1 is neck and neck with Chrome, others crown WebKit as the fastest of them all.

But what they never seem to do is compare them against other languages. One website, the Computer Language Shootout does, but it never seems to rate JavaScript particularly well. There are a few reasons for this. JavaScript has gotten a lot of attention recently and has thus been rapidly improving, and the CLS doesn't update very often. Tests are also written by different people, and folks who write JavaScript have never been the performance oriented crowd. Finally, the command line harness they use to execute JavaScript isn't representative of how users execute JavaScript.

So, what happens if we try to mitigate some of these? I picked a single test, rewrote it to run as a web page, and ran it using the latest version of browsers that can be considered stable. The results might surprise you.

n = 10

gcc	0.5s	1.0x
java6	0.7s	1.4x
java5	1.1s	2.2x
webkit	1.8s	3.6x
jsc	4.7s	9.4x
minefield	6.4s	12.8x
firefox	19.4s	38.8x
rhino	22.1s	44.2x
python	33.1s	66.2x
rhino	47.8s	95.6x
ruby	58.3s	116.6x
webkit [parallels/xp]	1.0s
chrome [parallels/xp]	2.0s

n = 11

gcc	5.1s	1.0x
java6	6.5s	1.3x
java5	12.6s	2.5x
webkit	23.4s	4.6x
jsc	57.5s	11.3x
minefield	81.2s	15.9x
webkit[parallels/xp]	15.0s
chrome[parallels/xp]	29.0s

Timings done on a Mac Book Pro 2.4 Gz Core 2 Duo / 4 GB RAM. Parallels/XP means running on Windows XP SP3 inside Parallels on the same computer.
WebKit is nightly build 39090.
WebKit [parallels/xp] is nightly build 39088.
jsc is built from svn rev 39090.
python is 2.5.1.
ruby is 1.8.6.

So, interesting things:

The fastest JavaScript implementations are close to the speed of Java.

The coming JavaScript implementations are substantially faster than Python or Ruby.

WebKit is about twice as fast as jsc (the command line interpreter), so the Computer Language Shootout numbers will be inflated.

Java 6 is substantially faster than Java 5.

So, I expect to see more web services written in JavaScript. Why? The argument for Python or Ruby has been that they are much more productive than Java/C/C++/C# that the performance of these languages isn't important. I certainly agree with this. However, JavaScript has about the same level of language productivity, and now has an implementation thats 18 times as fast as python and 32 times as fast as ruby. And you can use the same language across the board for web apps.

I also expect to start seeing desktop apps written in JavaScript. Why use a hacky python or ruby desktop app wrapper when you could use the best cross platform GUI kit there is? And, why bother with a local rails or django instance when you could do everything in a full MVC AJAX kit like Sproutcore or Objective-J?

At the risk of giving away the secrets to my sauce, desktop JavaScript is going to change everything. There are already a couple of frameworks out there (Jaxer, Titanium), but I think that there will be many more to come.

(PS: Lisp FTW!)

Java Static Class Object Fields and Weak References

2008-08-04T16:01:00.005-04:00

In browsing the Mozilla Rhino code, I stumbled upon a garbage collection problem that is truly bizarre. Rhino uses thread local storage for the current Context object. TLS uses a weak dictionary, so it is important that there is no strong circular reference to the key. Years ago (CVS r 1.234), the rhino class Context used to have a static reference to the thread local storage object. This was a problem because even when an array is empty, it has a reference to the class of objects it can hold. Thus, the TLS key (the ThreadLocal object itself) is referenced by the Context class, which is referenced by the (empty) Context[].class, which is a value in the TLS map! This circular dependency caused rhino to leak memory whenever a thread that used JavaScript completed. The solution to this was to change the value to an Object[]. When there are Context objects in the array, the reference still exists, however once the array is empty, there is no longer a reference and the TLS weakref dictionary will cut that key/value pair loose.

The comment, original, and fix.

Wow, thats a lot of detail. Here's the summary: if you have static class fields, make sure they don't eventually point to the key of a weak dictionary. If find this unavoidable, you can break the loop by storing a value as an Object and casting any usage of it. You will still have to set this variable to null in order for GC to happen.

Firefox 3 is the Ugliest Browser Yet

2008-06-17T16:00:00.003-04:00

It's a bit trite for a blog post, but seriously, Firefox 2 looked way better than 3 on Windows. If you customize the toolbar (via right click) to use "small icons", its marginally better, but people actually spent time working on the design of this UI, so I shouldn't have to fix it for them. Everyone would have been much better off with the third party Firefox 1.x theme Charamel.

I haven't used the Mac version yet, but it appears to have the same heinous icon problems. Look at Safari and try again please.

XPCOM nsISupports Proxy Crashes

2008-05-29T17:39:00.003-04:00

In building an application based on XULRunner, we've uncovered a number of bugs in the Mozilla codebase. One that really bothers me is the implementation of asynchronous object proxies. First, here's a brief overview of nsISupports proxies.

XPCOM

XPCOM is a generalized system for calling methods on objects. These objects can be implemented in C++, JavaScript, or Python (using a 3rd party library called PyXPCOM). XPCOM is the mechanism for defining Firefox extensions and writing applications that use XULRunner.

Working With Threads

One handy feature of XPCOM is the ability to have your method call execute in another thread. One example of this is a background thread that wants to do something to the UI, such as update a progress bar. Similarly to Java, Cocoa, and some Windows programming, UI access in Mozilla has to happen on the main thread. Another common situation is sending commands to a worker thread from the main thread. In both cases, we have a simple command to issue, but a direct method call is the wrong way to accomplish this.

So XPCOM has a mechanism for defining proxy objects that look identical to the original object, but executes your request in another thread. There are two flavors of this, synchronous (PROXY_SYNC) which will block your thread, and asynchronous (PROXY_ASYNC) which queues a request in the other thread and continues on. One implication of asynchronous dispatch is that there will be no return value, since you don't wait for the method to finish.

The Async Warning

The page describing proxies contains a very specific warning: if you pass variables by reference and those variables are on the stack, bad things will happen. It turns out that this is a bit understated, and doesn't convey some of the subtleties, so I will try to clarify.

Asynchronous XPCOM Proxies Are Fundamentally Broken

Using them will likely cause you nothing but pain. This pain will be in the form of totally random crashes, with meaningless (but seemingly useful) stack traces. You will ask people for help, and they will have no idea where to start. Unless you are intimately familiar with the XPCOM message dispatch and QueryInterface source code, PROXY_ASYNC should be a keyword which in your mind expands to Bad Bad Evil Crashing.

Why? Well, it turns out that return values are implemented as "out" parameters, which is a fancy way of saying references on the stack, which you may have recently learned is something that you should never mix with PROXY_ASYNC. So, if you were to call any method on an asynchronous proxy that could return a value, it will corrupt memory. There's actually no uncertainty here, XPCOM will write over some random memory address which is probably on some poor thread's stack.

The Workaround

"Ah!" you say, I can use PROXY_ASYNC and still sleep at night as long as I only call methods that return "void". Well, you might sleep a night or two, but soon you'll realize that bad things are still happening. The reality is, there is no workaround (sorry!). Even if you avoid methods with "out" or "in out" parameters, and ones that return values, Mozilla will still punish you by calling exactly these sorts of methods internally. If you're using a scripting language (generally JavaScript), things are even worse for you, because it makes a bunch more of these calls to set up your script object.

The Documentation

This bug is currently documented in buzilla. Unfortunately, there seems to be no workable solution to the situation. On top of that, the people responsible for the code base don't know why anyone would even want to use asynchronous dispatch. So, that doesn't sound good. The only real fix is to find a way to remove any mention of PROXY_ASYNC from your code. Depending on your application, this may be an easy fix. So far for us, this has not been the case. (Mozilla was never intended to do what we're doing)

FYI: JavaScript is Lisp

2008-01-04T12:44:00.000-05:00

Just a heads up in case anyone hasn't figured this out yet.

Some Things Never Change

2007-11-17T17:05:00.000-05:00

I just finished watching a half hour video on computer graphics. It covers the basics of 2D and 3D graphics and basic animation in a way that the average person can understand, but with enough detail to provide basic understanding.

The information itself isn't particularly blog-worthy; there are plenty of resources for learning about computer graphics. There are two reasons that this is interesting to me. First, television seems to have a dwindling amount of educational content. I remember watching 3-2-1 Contact, Mr. Wizard, and NOVA as a kid. The programs currently on PBS don't seem to show the same interest in science, substituting fantasy, social interaction, and ironically, reading. This may be related to my perception that they are also oriented to a younger audience, but I can't help but feel that kids should be learning these things instead of watching TV, not by watching it.

Second, this video is from a program called For All Practical Purposes and was filmed in the late '80s. The computer systems used are from Symbolics, a company that made their own hardware and operating system written entirely in lisp. This idea is similar to the current Squeak project, which is written in smalltalk instead of lisp. The video is old, the computers are old, the technology is old, and yet, it isn't particularly different from the basic concepts that I learned at Georgia Tech a few years ago. The biggest changes have been in the hardware that allows us to animate using physical simulation, and have realtime rendering.

Modern computing doesn't seem that different from 80's computing, except that everything is faster, larger, less efficient, prettier, more connected, and cheaper (much, MUCH cheaper). Just today I learned of a London startup that is writing games that would have run on a 1.2 MHZ Atari 2600 in Macromedia Flash - which uses most of my 1,200 MHZ CPU. But ultimately, its all the same. In my eyes, this makes a stronger case for the idea of Computer Science, because if you ever really learned how computers work, you might understand them forever.

parent and parent and proto, Oh My!

2007-11-16T15:40:00.000-05:00

Sometimes Javascript development can be rather scary. Unlike C or Java, which have well known standards, Javascript (like HTML) is often described by what works on one or more of the current implementations. On top of that, few people see Javascript as a legitimate language, and a vast number of novices are out there writing code that would make your skin crawl.

In short, if you are a knowledgeable programmer, quickly picking up Javascript is much harder than it should be. The syntax is C-like (generally regarded as a plus), the functions are first class (almost universally positive), and the object system is prototype based (at least smalltalk people like this). Soon there will be a common just-in-time compiler, and some people have suggested that it might displace currently popular languages. Personally, I wish that it ditched the prototype object system for a Python-like one, and had lisp-like macros, and just-in-time compilation can't come soon enough.

When a question about a Javascript feature like "x.__proto__" arises, it can be difficult to answer. In my experience so far, the best places to look are MDC and the IRC channel listed on the page.

What is __proto__?

When you make a Javascript "class" (ie constructing function), you can specify a prototype:

function Example() {
  this.foo = 1;
}

Example.prototype = {
  sample : function memberFunction() {},
}

This attribute is copied onto constructed objects as __proto__, like so:

ex = new Example();
if (ex.__proto__ == Example.prototype) print("same!");

__proto__ is a special attribute that is searched if an attribute is not found on the object, it is analogous to a parent class.

parent and __parent__ are substantially different. parent comes from the DOM, and is the tag that encloses the current tag. __parent__ is a special variable that is a reference to the scope that created the object. Like __proto__ is checked when a variable lookup fails. Where __proto__ is checked when the failure looks like this.missing_attribute, __parent__ is checked when the failure is in the form missing_global. According to Mozilla, __parent__ and __proto__ have been deprecated, which is unfortunate since they can be quite useful.

Debugging Python

2007-11-05T23:41:00.000-05:00

Above all else, my greatest annoyance with python is the lack of good documentation and defaults. I bet that there is a group somewhere that knows everything there is to know about python... I beg of them, write that knowledge down!

Here's an example. I want to debug a faulty program:

x = [1,2,3]

def foo():
sum = 0
for i in range(4):
 sum += x[i]

import pdb
pdb.run("foo()")

I'm then prompted by pdb: (> replaced with ])

] [string](1)?()
(Pdb)

Good, now I'm in the debugger. To get things started, I type "c", return, and get:

Traceback (most recent call last):
File "[stdin]", line 1, in ?
File "c:\Python24\lib\pdb.py", line 996, in run
Pdb().run(statement, globals, locals)
File "c:\Python24\lib\bdb.py", line 366, in run
exec cmd in globals, locals
File "[string]", line 1, in ?
File "[stdin]", line 4, in foo
IndexError: list index out of range
]]]

You should notice that the last line is the standard python prompt (with the aforementioned character replacement to make blogger behave), not the debugger. I have finished my debugging session due to an error. Gee, it sure would have been nice to debug that, since my state is now lost! I was already using the debugger to run this code, why did it not catch this exception?

It turns out that exception handling is done by sys.excepthook, and pdb.run doesn't set the excepthook. Some searching turned up two options. The first is simple but crude - add the following to site-packages/sitecustomize.py:

Thomas Heller

import pdb, sys, traceback
def info(type, value, tb):
    traceback.print_exception(type, value, tb)
    pdb.pm()
sys.excepthook = info

The second is more sophisticated, and checks for interactive mode:

ActiveState Python Cookbook.

# code snippet, to be included in 'sitecustomize.py'
import sys

def info(type, value, tb):
   if hasattr(sys, 'ps1') or not sys.stderr.isatty():
      # we are in interactive mode or we don't have a tty-like
      # device, so we call the default hook
      sys.__excepthook__(type, value, tb)
   else:
      import traceback, pdb
      # we are NOT in interactive mode, print the exception...
      traceback.print_exception(type, value, tb)
      print
      # ...then start the debugger in post-mortem mode.
      pdb.pm()

sys.excepthook = info

Here's what I'm currently running:

# code snippet, to be included in 'sitecustomize.py'
import sys

def info(type, value, tb):
   if (#hasattr(sys, "ps1") or
       not sys.stderr.isatty() or 
       not sys.stdin.isatty()):
       # stdin or stderr is redirected, just do the normal thing
       original_hook(type, value, tb)
   else:
       # a terminal is attached and stderr is not redirected, debug 
       import traceback, pdb
       traceback.print_exception(type, value, tb)
       print
       pdb.pm()
       #traceback.print_stack()

original_hook = sys.excepthook
if sys.excepthook == sys.__excepthook__:
    # if someone already patched excepthook, let them win
    sys.excepthook = info

The original ActiveState script doesn't debug if you are running in interactive mode. To me, this makes no sense at all - thats a case where I specifically want to debug.

Alas, if the debugger shows an obvious error that could be fixed, python exceptions cannot be resumed. The code listed here will let you see the stack and move around in it, but your program is no longer running and can never finish where it left off. Allowing this requires call/cc, and as far as I can tell there is no plan to ever support that in python.

If this was helpful, or should be changed, let me know. I've just started using it myself.

Closures in Python

2007-08-12T09:37:00.001-04:00

A closure is data attached to code (pretty simple, eh?). I use them for:

Replacing hard coded constants
Eleminating globals
Providing consistent function signatures
Implementing Object Orientation

(Isn't it funny that people rarely tell you what closures are good for?)
Here is a closure in python:

def makeInc(x):
  def inc(y):
     # x is "closed" in the definition of inc
     return y + x

 return inc

inc5 = makeInc(5)
inc10 = makeInc(10)

inc5 (5) # returns 10
inc10(5) # returns 15

Closures in python are created by function calls. Here, the call to makeInc creates a binding for x that is referenced inside the function inc. Each call to makeInc creates a new instance of this function, but each instance has a link to a different binding of x. The example shows the closure of x being used to eliminate either a global or a constant, depending on the nature of x.

import time
keepRunning = True
updates = []
def runLoop():
   while (keepRunning):
       for u in updates:
           u()

class foo:
   def __init__(self, x = 0):
       self.x = x

   def update(self):
       print self.x
       self.x += 1

f = foo()
g = foo(2)

updates.extend([f.update, g.update])

In python, all methods (but not functions) are closures ... sort of. The method definition foo.update closes the class foo. The value of g.update is a closure that stores the value of g and passes that as the first argument of foo.self, hence the first argument of a method in python is self. Details aside, it is important to note that the designers of python have gone out of their way so that you can pass g.update by itself to another function and have it continue to work correctly.

Caveats

In some languages, the variable bindings contained in a closure behave just like any other variables. Alas, in python they are read-only. This is similar to Java, and has the same solution: closing container objects. Closure of a dictionary or array won't let you assign a new dictionary or array, but will let you change the contents of the container. This is a common use pattern - every time you set a variable on self, you are changing the contents of a closed dictionary.

A Visualization of Visualizations

2007-08-03T01:39:00.000-04:00

You really need to see this one first hand, so check it out.

Working with Python/C

2007-07-06T19:13:00.000-04:00

Some very short notes for working with python from C. I'm using Python 2.4.x, YMMV.

Getting the current error state inside your debugger:

_PyThreadState_Current->curexc_type

Instantiating a "New Style" class (a subclass of "object"):

PyObject* args = PyTuple_New(0);
PyObject* dict = PyDict_New();

// works only with new style classes (subclasses of "object")
PyObject* result = PyObject_Call(classObject, args, dict);

Open Network Sockets on Mac OS X

2007-06-15T12:54:00.000-04:00

This is one of those things that constantly annoys me. On Linux, netstat tells me the list of currently active network connections, including (often most importantly) listening connections. Just knowing that something is running on port 8080 tips me off that I probably have an Apache or Java EJB process running (or maybe a swiki) - a trip to localhost:8080 will answer my question. But what if this port isn't an HTTP server, and doesn't speak when connected to? Now you have a dilemma - there's no way of knowing what has this port open.

Fortunately this is a solved problem on Linux. netstat has some options that tell you the PID of the process with the port open. From here you can use ps to find the name of the process, its path, the user who started it, etc. BTW, you need to run netstat as root to see other people's PID's and those of services.

Well, you know what comes next - netstat on the Mac doesn't show PID's! WTF! Speaking of commands that Mac OS X doesn't have, fuser is missing as well. fuser on Linux tells you which processes have a specific file open - very useful if you're cleaning up files and one has is locked. Well, Mac OS X (BSD really) doesn't have fuser... but it does have a command called lsof. lsof isn't quite as user friendly as fuser. It only has one mode, which is to list every open file that is visible to you (this is a subtle hint that you should run it as root to see more files). This means that fuser <filename> roughly translates to lsof | grep <filename>. Very useful for finding that stray service that has outlived its welcome and is holding files hostage.

Still, the problem at hand is finding the PID of network sockets. It turns out that in POSIX, network sockets are pretty much the same as files. This means that they show up in lsof if you ask nicely. And since lsof shows PID's (it even gets fancy and shows the process name), it turns out to be the solution. So, here's the money shot:

sudo lsof -i -P

This produces something that looks like:


COMMAND    PID  USER   FD   TYPE     DEVICE SIZE/OFF   NODE NAME
launchd      1  root    9u  IPv4 0x01d96e10      0t0    UDP *:137
launchd      1  root   10u  IPv4 0x020dbe8c      0t0    TCP *:139 (LISTEN)
launchd      1  root   11u  IPv4 0x020dbb38      0t0    TCP *:445 (LISTEN)
launchd      1  root   12u  IPv6 0x01d99c50      0t0    TCP *:22 (LISTEN)
launchd      1  root   13u  IPv4 0x020db7e4      0t0    TCP *:22 (LISTEN)
mDNSRespo   43  root    7u  IPv4 0x01d96ad0      0t0    UDP *:5353
mDNSRespo   43  root    8u  IPv6 0x01d96a00      0t0    UDP *:5353
mDNSRespo   43  root    9u  IPv4 0x03379d40      0t0    UDP *:5353
mDNSRespo   43  root   13u  IPv4 0x03379860      0t0    UDP 192.168.1.100:53891
mDNSRespo   43  root   14u  IPv4 0x032f82a4      0t0    TCP *:* (CLOSED)
mDNSRespo   43  root   15u  IPv4 0x02edb554      0t0    TCP 192.168.1.100:5000 (LISTEN)
mDNSRespo   43  root   16u  IPv4 0x020da098      0t0    TCP *:* (CLOSED)

... (lots more here)

If you don't have root access, you can still use lsof, but you won't see the plethora of system services and other users processes.

Python Techniques You've Been Googling For

2007-05-29T12:43:00.000-04:00

I've been doing it, I bet you have too. Here's some topics that have come up lately, I'll be writing about each of them in time. Since you probably got here via Google, add a comment for the subject that got you here.

Closures
Debugging
Creating functions
Metaclasses
Performance Optimization
Objects in depth

asdf-install on openMCL

2007-03-17T13:45:00.001-04:00

A Note: This has been sitting in my drafts folder for 10 months. I'm
posting it unedited because its better out then in, right?

---

This should be pretty trivial... after all, its included with the
latest openMCL, right? (I'm using the latest CVS)

Well, I've been fighting to get it working for almost a week, most
likely because I tend to ignore files named README. Having some
experience with CPAN and rubygems, it should really be possible to get
packages from a package repository without even knowing the language
that you are using. Why? Because newbies want to see results before
they have to invest time in learning new things. In the worst case,
an end user doesn't know, or even want to know, how to program at all.
Package repositories should really just work.

Here are my notes so far:

You need to compile asdf-install. To do so, you should be in the
directory containing asdf-install.asd . Then run:
(asdf:operate 'asdf:compile-op :asdf-install)
(asdf:operate 'asdf:load-op :asdf-install)
Now, make your asdf registry:

mkdir /usr/local/share/ccl/asdf-registry
cp asdf-

You need to create ~/openmcl-init.lisp with:
(require 'asdf)
(setf asdf:*central-registry*
'(*default-pathname-defaults*
#p"/usr/local/asdf-install/site-systems/" ; where
asdf-install puts things
#p"/usr/local/share/ccl/asdf-registry/" ; where i put
asdf-install.asd
))
(asdf:operate 'asdf:load-op :asdf-install)

;;; its really best to install GPG
(setf asdf-install:*VERIFY-GPG-SIGNATURES* nil)

The central registry setup is important - maybe
asdf-install/site-systems should be the same folder as the asdf
registry... I dunno. What I do know is that they both need to be in
the asdf search path (ie asdf:*central-registry*) or you won't get
very far.

I hope that I'm misguided, this is all a bad dream, and if I only
followed some magic instructions, this would be trivial. For now,
this works for me, and thats all that really matters.

Old And In The Way (stale fasl's)

2006-12-03T21:37:00.000-05:00

SBCL 1.0 has been released, and as expected, it breaks some things on my Powerbook. One specific problem is stale "fasl" files. Honestly, I don't even know what they are... I suspect them to be some kind of optimized bytecode used by ASDF to improve loading speed, but really (as long as they work) I don't care what they are.

SBCL will stutter something like this:
debugger invoked on a SB-FASL::INVALID-FASL-VERSION: #<SB-SYS:FD-STREAM for "file /usr/local/lib/sbcl/site/rt-20040621/rt.fasl" {11A024F9}> is in native code fasl file format version 70, but this version of SBCL uses format version 71.
Searching cliki revealed that these are intermediate files that can simply be deleted. They suggest that deleting files by hand is tedious and have a nice code snippit that should make things magically refresh in the future... I'm a little more pragmatic:
find /usr/local/lib/sbcl -name '*.fasl' -exec rm {} \;
Works fine afterwards.

Update: I was a little quick to declare victory. After executing this, i was unable to (require 'asdf), since the fasl for asdf had been deleted. I reinstalled (executed sbcl-1.0/install.sh) and the problem is fixed. This seems to invalidate my previous suggestion... Maybe I should have filtered the search to only include files older than a certain date. The cliki code snippit is probably the better way to go for now.

Python Developers Wanted: Dead or Alive

2006-11-21T23:13:00.000-05:00

I've been working at a new gig in Boulder, Colorado. We're writing an earth rendering engine, a la Google Earth. Why bother? Unlike the competition, we're going to be putting lots of our imagery on hard drives so that you can take it off the net: in your car, on a plane, in your backpack. You can sync changes when you get home, or use a cellular connection for low bandwidth data like local searching and geolocating your friends. If you've wanted it, hopefully we're on it (but feel free to send suggestions or leave comments).

I think 3d earth rendering is going to be big (maybe I've just read Snow Crash too many times), and recent announcements by Microsoft seem to confirm my suspicion. This blog doesn't get many comments, but usage statistics betray the silent masses who frequent it looking for KML tools. Are you tired of writing web services and pounding KML into new Google Earth features? Does writing a car navigation system 12 hours a day in python using a real 3d world API (we won't poke too much fun) sounds like your cup of tea? Maybe you're tired of Carmack taking credit for your OpenGL code. I heard somewhere that start-ups aren't en vogue anymore, but I know there's other people out there who regret missing the bubble. If you're confident in your abilities and can write the code to prove it, we're looking for you.

It would be nice if you lived in the US and wanted to move to Colorado (its sunny and warm even when the slopes are open and there's plenty of outdoors to go around. Send us an email.

How Python is Killing Lisp

2006-11-19T10:11:00.000-05:00

These are tumultuous times for lisp. Once the bastion of language academics and hackers and shunned by those of industry, it's received much lip service from successful dot commers, most notably Paul Graham. Lisp has been around a long, long time (its the second oldest language still in use, behind fortran), and its gotten very mature in that time. Industry languages continue to steal features from lisp, and some modern languages like python and ruby are more lisp than C. Still, lisp does not have the popularity one might expect. What is it that's holding it back while python prospers?

Lets start with what it gets right. It is said that lisp is very expressive, allowing extremely complicated ideas to be expressed succinctly (and hopefully quickly) by those fluent in the language. That's a rather heady idea, especially considering it is being applied to a practical industry. What this means in practice is that a great programmer is less limited by lisp than C++ or Java. This is really important because there are no natural limits on the productivity of a programmer. Unlike artisans and construction workers, there are no physical processes that limit how fast he can accomplish a task. The best person of another trade might be hundreds of times better than the novice, but they are limited by nature. Take away that limitation, and a handful of the best people could accomplish amazing things.

What makes lisp expressive aren't complex features like Object Orientation, but powerful simple ones like closures. The simplicity of these features let you build new complex ones as times change. Aspect Oriented Programming was part of the original Object Oriented lisp system based on closures. Continuations are a great way to deal with the non-linear flow of web applications.

Many of these powerful features have been adopted by new languages: Python can generate closures and has list comprehensions. Still, they don't contain all of lisp's power, and yet a search for open-source lisp applications on sourceforge returns 345 results to python's 6,571. Where are the hordes of elite programmers looking to change the world? Its easier to write things in lisp, right?

What lisp is getting wrong, opening the door for other lisp-like languages, is the new user experience. This means documentation, discoverability, sample code, libraries, and books. Its forums with solutions and smiling people. Unfortunately the lisp community is confident that their horse will win in the end, and they're just not willing to put in the effort. Other languages whose success is not so certain are more motivated to please. Python is one guy's foolish attempt to change the world, and you know that he spent a lot of time working to that end.

What lisp needs is is dedicated people behind it, laying the foundation of what people want. Specifically, people who are not currently lisp users. Given a python executable, some syntax, and knowledge of "import", "dir", and "help", anyone can learn python. Lisp should also be easy for newbies; there is even less syntax! SLIME is a great package, but please, stop hurting the cause... people who don't use Emacs are terrified by it.

Even the first step of a newbie has too many decisions: picking a lisp interpreter. There are simply too many of them, and their features are often inconsistent and incomplete. Perl, python, and ruby all managed to get one interpreter running on many platforms fairly uniformly. I don't worry about the networking or threading model inconsistencies between python on my mac and python on my linux server.

They also all have package managers that just work. This is a critical feature that just needs to work. CPAN really set the bar for package mangers in general. Asdf-install is okay, but it can be inconsistent across runtimes and packages don't always install correctly.

There are simple things as well. I am constantly irritated by the lack of prompt features in lisp interpreters. I'm lucky if they even recognize control characters... command history is a blessing. I haven't seen one that has any manner of auto-complete (something based on list-all-packages and do-symbols would be fine, thank you).

Its the unfortunate nature of lispers to work on the really long term things which are totally useless to their contemporaries. While I really appreciate that Paul Graham has decided to work on his own version of the language, I suspect that his lisp oriented incubator company would benefit more from a simple, usable IDE for sbcl and openmcl.

Ultimately, I am hopeful. Lisp is something that I believe in, and I'll keep trying to do more with it. If any language is ready to transcend old habits, this one is. Most things have already been tried in it once (distributed computing, hardware implementation, graphical editing), the next big thing is probably already in there somewhere waiting to be harnessed.

Back to Basics: Attending UC Berkeley From Your Lap

2006-05-31T23:40:00.000-04:00

Background

You're three years into high school, and bored out of your mind. You live in, have always lived in, have never left, central Connecticut. Its a far cry from the mid-west or the deep south, but you still find it difficult to learn more than they print in your text books, or what 9 out of 10 people know. An hour due north east and you might have been the subject of a half-indie blockbuster movie... but you aren't nearly that lucky. You're looking for an out, but you aren't even old enough to drive a car.

As it turns out, you're a lucky bastard. If you haven't yet guessed, this was my youth - years of frustration in trying to pursue something that no one understood. The benefit? Being able to gripe about how you pulled yourself out of nowhere by your own bootstraps. My verdict? Griping gets you no where, just do a better job than I did. After all, you're a lucky bastard. The reason that you are a lucky bastard - that you have no excuses - that this situation simply no longer exists - is simple: the Internet.

The Appetizer

If you are an athlete, painter, musician, or gear head, the Internet is probably quite useful. Sites like Wikipedia provide you with information that you would have never had access to (personally, I would have killed at least one or two people for that kind of resource). They'll help you sharpen your technique and learn the latest trends, even if you live in a cave (one with net access). But these areas require physical skills, and probably coaching. You must practice them for years, especially if they are competitive. They are intrinsically tied to the people available to you. On top of that, the best information for you is probably not available on the net at all - the best athletes and painters probably don't write web pages.

Ah, but maybe you're not. Maybe you are a computer geek, as I am. Spending more time trying to describe yourself in terms of bits and algorithms than figuring out who you are in the first place (btw: not healthy). If this is the case, then you sir (or hopefully, madam: 'cause computing need chicks) are in luck. While you might live in the sticks of Alaska, Peru, or Iowa, you are on the forefront of the computer industry. Don't lament not being in the Valley (crowded), don't make excuses of Route 128 (cold). The secret that expensive real estate doesn't want you to know is simple: the computer industry elite is everywhere are nowhere, wherever there is 'net. Does high school seem more interesting yet?

The Entree

So, young padawan - where to start. Some rules of thumb:

Don't use commercial products
Don't worry that your work doesn't look professional
Don't listen to what most people say - they don't know anything anyway. Especially if you are in High School.
Don't ignore everyone - some people know more than you (or they) think they do.
Don't get involved in politics or religion too much.
For the love of God, don't write C++. (You'll thank me later)

Read a lot. Paul Graham, Golden Age SciFi, Cyberpunk, the classics. I regret not reading more.
Exercise regularly. I fought what my parents forced on me, but you picked a mostly sendentary career and the human body requires exercise. Learn it early, and you'll appreciate it forever.
Learn Java. It has its problems, but its your best bet. Download src.jar and learn how everything works.
Finish things. When you're teaching yourself, its easy to leave things half done. Make a list of things you are working on and when they will be finished.
Use open source tools. Use Linux. Download source code for things and see how they work.
Talk to people. Just as a storyteller needs an audience, you need users. Programming is not about computers, its about solving people's problems.
Contribute to open source projects. You are contributing to the wealth of the world, and you can put that on your resume. Plus, you don't have to interview for them (suits bad!).
Think about what you are doing. Part of what you are doing is learning, so pay attention. Honestly, you don't want to succeed too much too early, or the allure of skipping college will be too great.
Oh yeah, go to college. You won't want to, and you might not enjoy it, but if you want to succeed (and who doesn't?), you have to. Its hard to explain why, so you'll just have to trust me. If you find it too easy, there will be a world of seriously cutting edge projects looking for volunteers. If you find it too hard...

Finally, the title of this post - attend UC Berkeley. Its easy! Really, anyone can do it - I do it at work. Plus, its free - how can you say no to that?

In a bid to put other schools out of business, the top programs are putting scores of class resources online - including podcast audio of lectures. I have been listening to CS 61A with Brian Harvey at work. Being a little underwhelmed with my GeorgiaTech undergrad education, I was curious to see what (if anything) I missed out on. So far, I like the clarity and confidence that Harvey has - but then, I thought the same of Kurt Eiselt. The course seems significantly faster and denser than my Tech experience. I suppose a program like UCB can expect more of its undergrads.

If you are interested in learning Lisp (or anything for that matter) or testing the breadth of your skills, I recommend checking out the plethora of courseware available. If you're trapped in High School computer class, step up and ask your teacher to incorporate intro level college courses. The first couple months of a class like CS 61A will teach you more than a year of AP Computer Science. And I cannot stress enough that all of this is free. If you have the time, you are losing nothing by checking it out.

Just Desserts

Convinced that you are alone in your thirst for knowledge? You could not be more wrong. Get together with your friends and work on things together. Networking is the name of the game in software. If you are in the Atlanta area, check out atlHack.org. We're like a support group for caffeine addicts and computer programmers. But it isn't hard to start your own - websites and time are both cheap now. If you do, drop us a line and we can start a network of hackers.

Whatever you do, make sure that you're in it for the right reasons. If you enjoy what you do, you'll get more out of it and do a better job. No startup has been founded by people who don't enjoy what they do. Plus, I don't like working with people who don't enjoy what they do, so do me a favor.

Airport ICAO SQL

2006-05-22T00:55:00.000-04:00

Getting simple information on flights is a real pain in the ass. There are airport codes (IATA and ICAO), airplane codes, and airline codes (IATA and ICAO). Here's the good news:

There is an existing database for IATA and ICAO airport codes.
There aren't too many airplane codes, and they aren't as important
IATA airline codes seem to be out of fashion, and will be officially replaced by ICAO codes soon.

I've scraped Wikipedia to put together ICAO airline codes in SQL. I won't be maintaining this list, but if you just want to cover the basics, you're welcome.

Game Lispers

2006-05-09T00:34:00.000-04:00

I've made some pretty technical blog posts so far. They've covered Objective-C, Java, Lisp, KML / GoogleEarth, python, ruby on rails, Mono, C++.... lots of topics that I don't generally talk about IRL because if I did, I wouldn't have any friends ( atlHack aside). At the same time, I don't get many (well, any) comments to any posts, so for a long time I was pretty certain that no one was listening to the blog either. Well, it turns out that I was wrong, and I get a lot more blog hits than I ever suspected. Some of them are via search engines, so in addition to page hits I have the search queries that brought you guys here.

My last post / rant focused mainly on some of the queries that I see, so I won't indulge myself with yet another rant. Instead, I'd like to talk about one query that seems to show up every month for a least a couple of hits: "naughty dog lisp". Of every statistic on the awstats page (and there are a lot of them) this is my favorite. Every time I see the count increment I smile, knowing that my blog is getting hit by some of the most elite hackers in the world. Why do I say that? Lets break this down:

Game hackers:

Are excited about their work. Any time people are excited about their work, they will be naturally better at it than if it bores them.

Deal with a lot of hard problems: real-time performance optimization, media (sound, graphics, physics), and concurrency.
Generally push the hardware to its absolute limits, just for the fun of it.

Lisp hackers:

Are looking for ways to make themselves more efficient.
Aren't scared of complex systems, but probably appreciate simplicity and elegance
Are working on the bleeding edge of development, even if that edge happens to be 50 years old. Lisp does not have hundreds of Dummies books.
Value function over form. There is no flashy IDE for it. It doesn't generate documentation from your comments. Its value is not in decoration, but in raw substance.

Given that people searching for "Naughty Dog Lisp" are in both of these groups, I would say they are a formidable group. But enough about them, I have real substance for this post.

I have started experimenting with OpenMCL's OpenGL bindings. For a long time, I have battled with graphics programming. The majority opinion is to use C and OpenGL, or C++ and a scenegraph library (Ogre3d is nice, and cross-platform). I appreciate the performance of these solutions, but I have always felt limited once my programs exceeded a few hundred lines. There isn't much you can do in a few hundred lines of code, especially if you've got visions of Oni in your head. So I have played with a number of programmer-efficient alternatives, like python, and my own 1/2 of common lisp, Commotion. (Commotion has somewhat transformed into a graphical programming language, akin to Max/MSP, which might be best implemented on top of lisp - but more on that later.) But PG's talks are eating away at the back of my mind - its time to leave the cheap substitutes behind and take on the real deal.

Which brings us to the listener interactive part of this post. I know that there are people out there reading this, and some of them are chasing the same unicorns that I am. If this is you, I want to hear about it. There's a comment button down there that's getting lonely. Anonymous comments are allowed, so there are no account-related excuses. Have you written a game in lisp? Screensaver? Music? Maybe you're just interested, or your Intro to Computing class is in scheme and you want to do something useful with it. Anything goes, except for comment spam, which recieves swift death.

A Little Break From Coding

2006-04-04T02:43:00.000-04:00

AWStats has brought to my attension some incidents that I find a little disturbing... Let me present to you the current search queries for ynniv.com in April:

Term	Hits
recursive drawing java	2
naughty svn	1
rails .htaccess rewrite	1
itunes 5	1
dependency injection frameworks	1
recursive drawing	1
moving to another country	1
coding autorun	1

First off... "naughty svn"... 3rd page, baby (due to a mishap link to another blog post, "Naughty Dog Software"). ... Why would someone search for this? Is your versioning system being a bad little boy? Does it need discipline? Is it wearing black panties or something? People, it versions your text files - leave it alone! Nothing good will come of molesting svn, its only there to serve your basic versioning needs!

Next up, "itunes 5". Apparently, a blog of mine made it to page 13 of "itunes 5" on google. Which begs the quesiton, What were you doing on page 13?! After the first 10 or so pages, would you not reconsider your search term? And more importantly, why am I even on page 13?!? The only thing I had to say was that iTunes 5 looked a lot like iTunes 4 with the graphical contrast cranked up. I'm not sure if my google ranking says more about Google's PageRank system, or people's opinions of iTunes 5.

Now, my favorite: "moving to another country". I made page 4 on this one. I like this for a number of reasons. First, I think more people should move to another country. Not in the "this is my country, get a move on" sense, but the "everyone cool is going somewhere else" sense. Why? Because I think that the United States is simply too cocky. We're pretty far behind on a lot of new things (the painfully obvious, glaring example is Broadband 'net access), and yet, we think we're the bomb dug. For instance, were you aware that the United States is Ferrari's largest market?

Plain and simple, we have too much money and no idea what to do with it. So, if you haven't yet, and are interested in, I highly recommend spending some time in another country. No, don't go on vacation - thats just moving your US lifestyle to Ibiza for the week. The French call this "le bullshite". If you have the opportunity, study abroad. Trust me, you have no idea how important this is until you've done it.

Go to a place that has a high cost to quality-of-life ratio, and you'll understand how real people live. In places like Costa Rica and Peru, you could retire on the cost of that Ferrari - and live a good life too. Guess what - there's even Internet access. Think about that - with $150k in the bank, you could buy your own castle in Costa Rica, and still read Slashdot. Now, you should be warned that you'll need to own and know how to use a shotgun. But, how hard do you really have to work to bank $150k for retirement? I met a guy from California who did just this.

Questions? Comments? ynniv-web@this-url.

Blending KML

2006-03-20T23:56:00.000-05:00

Update: We're hiring python programmers to write a car navigation system in our earth rendering engine (a la Google Earth).

Google is the best thing that's happened to Keyhole. They've transformed a cool visual toy into a platform for GIS information, and the key to their success is twofold. First is the open KML (XML) file format - anyone can make simple content. Second, Google Earth is free for limited use. The combination is magic - developers are eager to create content, consumers are eager to view this content, and Google gets rapid desktop application adoption and the ability to roll out future advertising.

The KML format is easy enough to read and write by hand for small amounts of meaningful data, like placemarkers and pointers to raster files for ground overlays. What's currently lacking is a free way to convert lots and lots of meaningless data, like 3d models. There are a few commercial tools out there, most notably SketchUp (recently purchased by Google, 8 hour trial available), but nothing free for the casual hacker... until now.

kml_export.py is a python script for blender that supports meshes and diffuse lighting. Not much, I know, but KML doesn't support much 3d. This is really a quick-release, rough cut, leaves much to be desired release - but it gets the job done.

Installation:

Put kml_export.py in your Blender scripts directory
(windows = "c:\Program Files\Blender Foundation\Blender\.blender\scripts",
mac = "/Applications/Blender/blender.app/Contents/MacOS/.blender/scripts",
linux = "~/.blender/scripts"?)
Edit kml_export.py to adjust the placement constants (lat, lon, altitude, scale)
If you're using Windows, you'll need to download PyXML. Unarchive the PyXML folder and move the "xml" subdirectory to the same folder as kml_export.py.

Your scene needs to contain at least one mesh object to produce useful output.