random.choice(['idea', 'rant', 'link', 'tip'])

Going to speak at DjangoCon!

2008-07-22T11:45:00.000-04:00

Wow, my name is on the DjangoCon schedule on Sunday, 2:25 pm. Jim Baker told me about it just a few days ago. This may be a bit surprising, but looks like the whole DjangoCon is a bit surprising.

I can hardly believe that I'm going to be there, and I'm extremely happy to have this opportunity to meet the Django community and to show what we are doing on this GSoC-funded project of getting Django running on Jython and integrating it with some cool JVM stuff.

Well, I still have to do all the paper-work (I don't have a US visa yet, this is going to be my first visit to the country), and there is not too much time to do it. Not to mention that I've to practice my spoken English. But we humans are optimistic by nature, and I think that such optimism let us do most of the great things we do!

The Jython Import Logic

2008-07-20T18:51:00.002-04:00

Motivation

I think the coolest feature of Jython is the seamless integration with Java. Let say you have the following java class:


package com.leosoto;
public class HelloWorld {
    public void hello() {
        System.out.println("Hello World");
    }
    public void hello(String name) {
        System.out.printf("Hello %s!", name);
    }
}

If the class is on the classpath when you start Jython, using it from python code is straightforward:


>>> from com.leosoto import HelloWorld
>>> h = HelloWorld()
>>> h.hello()
Hello World
>>> h.hello("joe")
Hello joe!

Now, did you knew that if the class was not pointed by the classpath, we could also package it on a jar, and the following would also work:


>>> import sys 
>>> sys.path.append('/path/to/helloworld.jar')
>>> from com.leosoto import HelloWorld

Until yesterday, I didn't knew!

Since part of my GSoC project is to come with a way to package Django projects in a single distributable war file, I've spent a complete day reading and playing with the Jython import logic, and here is what I got.

Not much different than Python, right?

First of all, Jython is an implementation of the Python language. So the import mechanism follow strictly what is know as PEP 302: import hooks. I don't want to repeat what is documented there, but a quick explanation is in order:

First, try custom importers registered on sys.meta_path. If one of them is capable of importing the requested module, we are done.

For each entry of sys.path:
- Find the first hook registered on sys.path_hookthat can handle the path entry (for example, zipimport is registered there and handle all "*.zip" paths)
- If a importer hook was found, try it. If the importer loaded the module, we are done
- If a importer hook was not found, use the builtin import logic (good old *.py files inside directories). If the module is successfully loaded, we are done.

How are java classes loaded then?

With a built-in import hook, naturally ;-)

If you start CPython and look at sys.path_hooks you get:


>>> import sys
>>> sys.path_hooks
[<type 'zipimport.zipimporter'>]

On Jython, the result is slightly different:


>>> import sys
>>> sys.path_hooks
[<type 'JavaImporter'>, <type 'zipimport.zipimporter'>]

The JavaImporter only recognizes the '__classpath__' entry on sys.path, so it is fired after looking at path components before '__classpath__'. This gives us some control over which namespaces will end up containing python modules and which will contain java packages/classes, if some conflict occurs (such as the very real issue of having the 'test' python module and the 'test' java package). Naturally, the '__classpath__' entry is added automagically to sys.path on Jython startup.

But...


>>> sys.path_hooks = []
>>> sys.path_importer_cache = {}
>>> del sys.modules['java']
>>> import java
>>> dir(java)
['__name__', 'applet', 'awt', 'beans' ... ]

This should have failed, after removing the JavaImporter hook and all the involved caches. Well, there is also some magic going on here...

Jython, `JavaImporter` and Java Packages

When an import is going to fail (that is, after searching on all sys.path entries and having no results), Jython tries to load a java package or java class. But wasn't that the task of the JavaImporter?

Well, sort of. Half of such job is the responsability of the JavaImporter. The other half is managed by the SysPackageManager, which keeps in memory a tree of discovered java packages.

When the Jython interpreter starts, the SysPackageManager looks for all jars and directores on the classpath and build the tree of java packages. You can also explicitely add a Java package into the PackageManager by calling sys.add_package("package.which.was.not.autodiscovered"). This is useful on environments where Jython is not allowed to look at the system classpath, or doesn't get the right information (as maybe the case when running inside a JavaEE container).

Back to JavaImporter, its job is to just look into the SysPackageManager's loaded packages and check if the requested name is present there.

And here is the magic

Another way to get packages loaded into SysPackageManager is to add a zip or jar to sys.path. The next time the import logic runs, it automatically add the contents of the new jar (or zip) to the tree of known java packages.

This is a little weird, because if you have the following in your sys.path:


['__classpath__', '/foo', 'foo.jar']

Then, if java packages on foo.jar conflicts with python modules from /foo then the java packages will prevail, because the '__classpath__' entry is before '/foo', and then the JavaImporter will do its magic.

And the other bit of magic is what we have already seen: Jython does a last attempt to load a Java package, or to be more precise, to add a package to the SysPackageManager if the imported name is know to the JVM as a class or package name. If this operation is successful, the module is directly imported by this Jython builtin import logic addition (no way to go back to the JavaImporter at this time).

Some observations

Here ends the objective part of this post. What follow now are my observations on the whole process:

I don't quite understand why Jython tries to load Java classes or packages at the end of the import logic after trying the standard procedure. Seems like such fallback would make the calls to sys.add_package unnecessary, but then, why does add_package exists? And, in any case, I think that JavaImporter should do this

The confusing situation of jar files (and java classes) in sys.path is well... confusing. The good news is that namespace conflicts aren't that common in practice, so just remembering that all java "modules" come from the magic '__classpath__' element is enough.

It would be nice if the Jython standard loader were installed on the meta_path. Then, JavaImporter could be added there too, just after the default python code loader. This way, we would have a more clear precedence rule (Python modules first, Java packages/classes later), instead of the current "first python modules before __classpath__, followed by java "modules", followed by python modules after __classpath__, followed bt java "modules" wich weren't registered yet on the PackageManager).

That's all

OK, that was a long post. Now that I've dumped all that info here, I can go back to coding and try to make distributable WAR files for django projects, containing the complete Jython, modjy and Django runtime.

Jython 2.5 Alpha Released!

2008-07-15T14:09:00.002-04:00

As announced by Frank Wierzbicki: Jython 2.5 alpha is out.

If you work with Java and love the Python programming language, this is a good opportunity to test this great Python implementation, integrate it with some of your Java programs and tell us how it went.

My New Django/Jython Developer Workflow

2008-07-14T11:36:00.006-04:00

Both Django and Jython project are fast moving targets these days. That's a good thing, both projects are rapidly approaching big milestones: Django 1.0 and Jython 2.5. But that also means that if you are trying to patch both codebases to integrate them, your task gets a little more complicated.

So I don't have private mercurial branches of both projects anymore, because it makes quite hard to update patches and keep them as separate units. The new solution is two new mercurial repositories: django.patches and jython.patches. They contain mercurial queues and correspond to the .hg/patches directory you put inside the repositories containing the mercurial mirror of each project.

Translated to command line, this is what you have to do in order to get the current code for Django on Jython:


$ hg pull https://hg.leosoto.com/django.svn.trunk
$ hg pull https://hg.leosoto.com/jython.svn.asm
$ hg pull https://hg.leosoto.com/django.patches django.svn.trunk/.hg/patches
$ hg pull https://hg.leosoto.com/jython.patches jython.svn.asm/.hg/patches
$ hg --cwd django.svn.trunk qpush -a
$ hg --cwd jython.svn.trunk qpush -a

This way, I always try to keep my patches updated to apply cleanly to each project latest svn version.

I also have a hudson running on my machine which runs the Django test suite on CPython using all backends. Once that test is finished it runs the suite again after applying my Django patches (to make sure I'm not breaking anything). Finally it runs the suite one more time, using Jython and the postgresql_zxjdbc backend. This should replace the dojstatus site, once I discover a way to publish the results I get from hudson without installing hudson on my hosting.

By the way... I'm a Jython commiter! :)

2008-07-10T10:33:00.004-04:00

I've been so busy last weeks (half-time job, Summer of Code, and university final exams and labs!), that I even forgot to mention that Frank gave me committer access to the Jython project, two weeks ago.

I want to thank publicly all Jython core developers for the confidence they have put on my work, especially Jim Baker, my GSoC mentor. Also Phillipe Jenvey, Nicholas Riley and obviously Frank Wierzbicki, have been extremely helpful guiding me when I needed help.

As I now commit most of my Jython patches directly to the SVN repository, I'm going to deprecate the jython.doj Mercurial repository. I'll post about it soon, along with a new recipe to get Django running on top Jython, using the asm branch (i.e, the upcoming 2.5 version).

The Devil is in the Details

2008-07-05T20:44:00.004-04:00

It's a cliché, but really, look at this commit message I just wrote. It's for a supposedly simple change I made to Jython to get '%d' % foo and '%f' % bar working, on some corner cases[1]:

“StringFormatter: '%d' and '%f' support for the __int__ and __float__ protocol respectively. The implementation is more convulted than it should be, because we have PyString implementing __float__ and __int__ at the "java level" but not at the "python level". For string formatting, only "python level" __float__ and __int__ must be supported. Also, now that __int__ can return a PyLong, this case needs special care. Basically formatInteger now can call formatLong if a PyLong is found as the result of __int__. Then, as formatLong can also be called from formatInteger, __hex__, __oct__ and __str__ conversions were moved inside formatLong. Finally, test_format_jy was changed to stop checking that we don't support big floats on '%d' (CPython doesn't, but that seems a limitation of the specific implementation and I can't imagine a program that could break on Jython because we *support* it).”

Python is wonderful, but there are a lot of details which make it tricky when implementing it. Nice to see that, when we play the role of Python users, we aren't exposed too much to this subtleties. In fact, I'd say that it is one of the languages with the better "user interface" I've seen.

[1] Naturally, I find this corner cases when running and testing the Django codebase. Well, that's one of the points of my SoC project: Test how CPython-compliant Jython is, and fix it when it isn't :).

We Should Master Regular Expressions

2008-07-01T13:00:00.007-04:00

When someone tell us that every programmer should know regular expressions, it's not only about using them to validate or match input on our programs. After all, seems like many of us can live using split() and replace() and some ad-hoc code instead of learning regexps.

My point today is that they are also useful when coding. I just needed to replace every code that looked like this:


<mx:RemoteObject id="grabaMuestreosRemote" 
... 
fault="Alert.show('Problemas al grabar los muestreos')"/>

To:


<mx:RemoteObject id="grabaMuestreosRemote" 
... 
fault="reportFault('Problemas al grabar los muestreos', event.fault)"/>

The change is on the last line, replacing the alert by a slightly more involved logic (which lives inside the reportFault function).

Solution? Find/Replace, using regexps (this was done with eclipse, but every reasonable editor have this feature):

Find:fault="Alert.show$'([^']*)'$"

Replace With:fault="reportFault('$1', event.fault)"

Quick explanation: $ and $ matches literal parenthesis; they are escaped because they have their own special meaning on regexp: capturing. And they are using for capturing the string message inside quotes, on '([^']*)'. That means: a single quote (') followed by any character which is not a single quote ([^']), repeated 0 or more times (*), followed by a single quite ('). So the non-escaped parenthesis are used to capture (i.e, remember, store) what was found inside the quotes. Later, you use the captured value by specifying $1 on the replacement text. If you have more captures, they are labeled $2, $3 and so on.

By the way, I'm not a regexp master. In fact, I admit to frequently resort to ad-hoc code, especially if I'm in a hurry.

But the simple exercise of summing all the time spent on writing ad-hoc code, plus the time wasted doing non-trivial find & replace by hand, have convinced me to learn them, and hopefully master them.

Doctests and XML/XHTML

2008-06-12T23:00:00.004-04:00

Every python programmer knows that doctests are really cool. Checking snippets of code just by the literal output that it would print when run on the REPL is both simple and powerful.

But, sometimes, they could be too literal. There are options to deal with this, such as the use of ellipsis (...) as wildcards. They are not enough to deal with XML output, though:


File "/home/lsoto/src/django.doj/tests/modeltests/model_forms/models.py" [...]
Failed example:
    print f['name']
Expected:
    <input id="id_name" type="text" name="name" maxlength="20" />
Got:
    <input name="name" id="id_name" type="text" maxlength="20" />

(that's an actual failure from the Django test suite running on Jython)

To solve this, I implemented a doctest OutputChecker, inspired by (but not as polished as) lxml.doctestcompare. The plus side? The code uses the xml.dom.minidom stdlib API (instead of ElementTree or lxml), so it works without any third party library. Here is the core code:


def check_output_xml(self, want, got, optionsflags):
    # Tries to do a 'xml-comparision' of want and got.  Plan string
    # comparision doesn't always work, because, for example, attribute
    # ordering should not be important.
    #
    # Based on http://codespeak.net/svn/lxml/trunk/src/lxml/doctestcompare.py

    # We use this to distinguish repr()s from elements:
    _repr_re = re.compile(r'^<[^>]+ (at|object) ')

    _norm_whitespace_re = re.compile(r'[ \t\n][ \t\n]+')
    def norm_whitespace(v):
        return _norm_whitespace_re.sub(' ', v)

    def looks_like_markup(s):
        s = s.strip()
        return (s.startswith('<')
                and not _repr_re.search(s))

    def is_quoted_string(s):
        s = s.strip()
        return (len(s) >= 2
                and s[0] == s[-1]
                and s[0] in ('"', "'"))

    def is_quoted_unicode(s):
        s = s.strip()
        return (len(s) >= 3
                and s[0] == 'u'
                and s[1] == s[-1]
                and s[1] in ('"', "'"))

    def child_text(element):
        return ''.join([c.data for c in element.childNodes
                        if c.nodeType == Node.TEXT_NODE])

    def children(element):
        return [c for c in element.childNodes
                if c.nodeType == Node.ELEMENT_NODE]

    def norm_child_text(element):
        return norm_whitespace(child_text(element))

    def attrs_dict(element):
        return dict(element.attributes.items())

    def check_element(want_element, got_element):
        if want_element.tagName != got_element.tagName:
            return False
        if norm_child_text(want_element) != norm_child_text(got_element):
            return False
        if attrs_dict(want_element) != attrs_dict(got_element):
            return False
        want_children = children(want_element)
        got_children = children(got_element)
        if len(want_children) != len(got_children):
            return False
        for want, got in zip(want_children, got_children):
            if not check_element(want, got):
                return False
        return True

    # Strip quotes
    if is_quoted_string(want) and is_quoted_string(got):
        want = want.strip()[1:-1]
        got = got.strip()[1:-1]
    elif is_quoted_unicode(want) and is_quoted_unicode(got):
        want = want.strip()[2:-1]
        got = got.strip()[2:-1]

    if not looks_like_markup(want):
        return False

    # Wrapper to suuport XML fragments
    wrapper = u"<root>%s</root>"
    try:
        want_root = parseString(wrapper % want).firstChild
        got_root = parseString(wrapper % got).firstChild
    except:
        return False

    return check_element(want_root, got_root)

Note that, as is, it doesn't support HTML. That's not a problem on the Django test suite (where everything is XHTML) but it would be nice to add such support and then submit the checker upstream to CPython's doctest.py. So we could do something like this everywhere:


>>> print f['name'] # doctest: +XML
<input id="id_name" type="text" name="name" maxlength="20" />

Next target: JSON output!

Update: I've submitted a patch to Django with support for both, XML and JSON output.

Jython: How to Instantiate Classes Written in Python Code

2008-05-30T17:04:00.003-04:00

This week, as part of my GSoC project, I had to do some work related with zxJDBC, the very cool DBAPI <-> JDBC brige which is bundled with Jython. Among other things, I'm improving the default type mapping, adding support for converting java.sql.* instances to pythonic datetime.* ones.

As the type mapping is written inside a class written in Java, I was confronted to the problem of how to instantiate datetime.* objects, which (by now) Jython implements using pure python code. The answer was very simple: Just do by hand what Python always does when you write:


import datetime
datetime.date(year, month, day)

You know, the import statement is implemented by the __import__ builtin, foo.bar is getattr(foo, 'bar') and f(x) is f.__call__(x). Then, the following is equivalent to the previous snippet:


datetime = __import__('datetime')
getattr(datetime, 'date').__call__(year, month, day)

Which, translated to Java/Jython looks almost the same:


PyObject datetime = __builtin__.__import__('datetime')
datetime.__getattr__('date').__call__(Py.newInt(year), 
                                      Py.newInt(month), 
                                      Py.newInt(day))

Once you get the idea, not only instantiating, but doing anything with classes written with Jython from Java code looks like a piece of cake.

Hey!, How easiest could it be?

Note that this works if all the Jython machinery is in place. This will be the case if your Java code is being called (directly or indirectly) from python code being ran on Jython. As any Jython module, such as zxJDBC, is always used from python code, this is something to have in mind when writing them in Java.

Loading Rails Fixtures Without Deleting Existing Records

2008-05-28T13:32:00.001-04:00

This blog post by choonkeat explain how to overcome the ruby limitation which forbids the use of the "class [name-of-the-class-to-monkeypatch]" statement inside methods.

Why would you need this?

I just used it to define a db:fixtures:insert task, which differs from db:fixtures:load on not removing existing records. I could have copied the Fixtures.create_fixtures method and remove the line which calls Fixtures#delete_existing_fixtures. But we all know that copy/paste have its problem. Monkey-patching Fixtures#delete_existing_fixtures to do nothing seemed like a less ugly solution, except that it could have unforeseen consequences. So I wanted to limit the effect of the monkey patching to last as little as possible. Here is the result:


    # Mimics rails Fixtures.create_fixtures, without deleting existing records
    # from the database.
    #
    # The implementation monkey-patches Fixtures.delete_existing_fixtures
    # *temporarily*, restoring the original behaviours before exiting
    def self.insert_fixtures(fixtures_directory, table_names, class_names = {})
      ::Fixtures.module_eval do
        alias_method :original_delete_existing_fixtures,
                     :delete_existing_fixtures
        def delete_existing_fixtures
        end
      end
      ::Fixtures.create_fixtures(fixtures_directory, table_names, class_names)
      ::Fixtures.module_eval do
        alias_method :delete_existing_fixtures,
                     :original_delete_existing_fixtures
        remove_method :original_delete_existing_fixtures
      end

[By the way, looks like this monkey-patch -> call -> undo monkey-patch may be end being common idiom which could be better encapsulated on another method. Perhaps someone already did it...]

[Update: Here is what seems an elegant way to do it]

Finally, Here is the task:


namespace :db do
  namespace :fixtures do
    desc "Inserts fixtures into the current environment's database,
*without* deleting existing records (as db:fixtures:load does).
Insert specific fixtures using FIXTURES=x,y"

    task :insert => :environment do
      ActiveRecord::Base.establish_connection(RAILS_ENV.to_sym)
      (ENV['FIXTURES'] ? ENV['FIXTURES'].split(/,/) : Dir.glob(File.join(RAILS_ROOT, 'test', 'fixtures', '*.{yml,csv}'))).each do |fixture_file|
        Utilities::Fixtures.insert_fixtures('test/fixtures', File.basename(fixture_file, '.*'))
      end
    end
  end
end

Ubuntu: Changing Swap Size Without Loosing Hibernation

2008-05-15T23:03:00.001-04:00

Short recipe:

After resizing your swap partition (if you resized the swap partition instead of adding another one), use vol_id -u /dev/<device-name> to get the new partion UUID, and put it at the appropriate place on /etc/fstab and /etc/initramfs-tools/conf.d/resume. Then, run:


 $ sudo dpkg-reconfigure  initramfs-tools

Now, the not so long history:

When I made the changed on my partitions to go back to Linux, I made a mistake reserving only 1Gb of space for the swap partition. It sounded like a reasonable amount as I'd have a total of 3Gb of virtual memory, enough for my typical usage.

I forgot about hibernation, which, on Linux, uses the swap partition to store the main memory contents (unlike Windows, which have a separate file for hibernation). To be honest I didn't even considered hibernation when installing Ubuntu, it never worked in the past. But now it worked. At least when I wasn't using too much main memory. If i was, then it wouldn't fit in the swap file and the laptop couldn't hibernate.

GParted came to the rescue again, and my swap partition grew to a little more than 2Gb, which is the size of my main memory. Reboot on Ubuntu again and...no swap was recognized. WTF!?

After looking at the /etc/fstab I realized that the partitions are identified by an UUID rather than by their device name. So no /dev/sda5 (my swap partition) there. Just a bunch of hexadecimal digits. I thought of wiping that mess out and go back to my familiar devices names, but googling first was a better option. Then I found that I can easily know the new UUID of the partion using:


 $ sudo vol_id -u /dev/sda5

And, what really saved me from another headache was discovering that, after a change of UUID you must update two files: the “classic” /etc/fstab and also /etc/initramfs-tools/conf.d/resume. And finally run:


 $ sudo dpkg-reconfigure  initramfs-tools

Not that straightforward as it could be, but I have no real complaints, as I now can hibernate my laptop whenever I want.

Back to Linux

2008-05-11T18:47:00.000-04:00

I'm back to Linux as my main OS. As you can infer from previous posts, I was using Windows on the past months. In fact, I was using Windows since I bought my Dell Laptop, a year and half ago. It wasn't just Windows: I used cygwin and coLinux inside my Windows XP to have a reasonable development environment too (except for Java, which feels OK for me on plain Windows). Somehow I survived running that all this time, forced in part because I had to use some Windows-only VPN-client software on my job.

Two weeks ago things changed. I did some backups, ran GParted (couldn't find my laptop's recovery disk, so my original plan of wiping out Windows and reinstalling everything couldn't go ahead) to shrink the NTFS partition, checked that everything was OK and installed Ubuntu Linux.

Choosing Ubuntu was a new thing: I started ten years ago with Redhat and changed to Debian “sid” five or six years ago (apt-get was such an important advantage on that times). But now I had little time to fight with my particular hardware configuration and heard that Ubuntu ran very good on Dell laptops. So I tried it and I'm not looking back.

The only thing that wasn't automatically configured was my broadcom wireless card. Hey, having to deal with only one driver issue looked very promising to me (but it will drive away people who expects everything running out of the box, anyway). Installing b43-fwcutter and following the instructions made me a happy user. But all that could be caused by me installing a non-final version of the distribution.

The overall impression is great. I'm even sticking to Gnome. Always used KDE in the past, but the default gnome environment seems very functional and I have nothing to say against it. And compiz is a great way to impress people :). Frankly, I miss konqueror versatility. But, considering that I would use Firefox as my main browser anyway (basically because it has great extensions; I don't like it eagerness for memory), that's no big deal.

Migrating things from my coLinux filesystem couldn't be easiest: just moving files, and everything (emacs, latex, gnucash, mercurial, svn) works.

Above all, having an unix-like environment is priceless.

If FlexBuilder 2 had a Linux version, it would be perfect. Perhaps too perfect to be real.

Django SECRET_KEY Generation

2008-04-30T11:52:00.004-04:00

When deploying Django application it is a common step to generate a SECRET_KEY for the site. Here is the quick recipe to do it:


$ python -c 'import random; print "".join([random.choice("abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)") for i in range(50)])'

Useful when for whatever reason you don't want to install django-command-extensions.

Python Comparison Weirdness

2008-04-28T13:05:00.005-04:00

While tracking a Jython bug related with some __cmp__ methods (on dict and unicode, ant least) I had to check how __cmp__ behaves on CPython. And got a few surprises:


>>> {} == ''
False

It sounds right, but...


>>> {}.__eq__('')
NotImplemented

Oh. So == isn't using __eq__ to check for equality. It's using the old three way comparison function:


>>> cmp({}, '')
-1

So, as '' is greater than {}, then they are not equal. But...


>>> {}.__cmp__('1')
Traceback (most recent call last):
  File "", line 1, in 
TypeError: dict.__cmp__(x,y) requires y to be a 'dict', not a 'str'

Oops. Isn't cmp(foo, bar) the same that foo.__cmp__(bar), at least when hasattr(foo, '__cmp__')? Well, obviously, not always.

For some reason, CPython does a bit of "type checking" when you indirectly use dict.__cmp__. If you compare a dict with an instance of a incompatible type, it does a "default comparison" by class name, instead of raising TypeError. By looking at CPython sources it seems that this is the case for every type where tp_compare is implemented in C.

So, we get a -1 from cmp({}, '') because 'dict' < 'string'. Weird. But that isn't all. If it were, probably I wouldn't bothered to write this.

Let's derive dict and check what happens:


>>> class dict_derived(dict): pass
...
>>> cmp(dict_derived(), '')
-1
>>> dict_derived().__cmp__('')
Traceback (most recent call last):
  File "", line 1, in 
TypeError: dict_derived.__cmp__(x,y) requires y to be a 'dict_derived', not a 'str'

No surprises: It inherits the behavior from dict. So, remembering what I said above:

If you compare a dict with an instance of an incompatible type, Python does a "default comparison" by class name, instead of raising TypeError.

Now we can extend it to:

If you compare a dict or an dict-derived instance with an instance of an incompatible type, Python does a "default comparison" by class name, instead of raising TypeError.

But, why am I saying that it applies only to dicts? [Or, AFAICS, special types where the comparision function is written in C] Why not to every type? Aswer:


>>> class Foo(object):
...   def __cmp__(self, other):
...     raise TypeError("Foos are not comparable")
... 
>>> Foo() == ''
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in __cmp__
TypeError: Foos are not comparable
>>> cmp(Foo(), '')
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in __cmp__
TypeError: Foos are not comparable

So, on one hand we have dict (and maybe other builtin types) where cmp() and comparison operators doesn't raise TypeError even if __cmp__ does. And on another, user-defined classes where the raised TypeError does "leak". In the middle, our dict_derived class inherited the behavior from dict. But look at this:


>>> class dict_derived2(dict):
...   def __cmp__(self, other):
...     super(dict_derived2, self).__cmp__(other)
... 
>>> cmp(dict_derived2(), '')
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in __cmp__
TypeError: dict_derived2.__cmp__(x,y) requires y to be a 'dict_derived2', not a 'str'

Dict-derived types inherit the behaviour of dict, unless they override __cmp__. CPython doesn't care that the new __cmp__ just call the original dict.__cmp__. The only important thing is that there is a __cmp__ implemented on python code. Once you write a "custom" __cmp__, cmp(), == and all the other comparison operators will raise the exception.

To summarize, here is final rule for dict.__cmp__:

If you compare a dict or an dict-derived instance with an instance of an incompatible type, and __cmp__ is not overriden, Python does a "default comparison" by class name, instead of raising TypeError

Note that this rule is not directly applicable to other builtin types that implement __cmp__:


>>> set().__cmp__('')
Traceback (most recent call last):
  File "", line 1, in 
TypeError: set.__cmp__(x,y) requires y to be a 'set', not a 'str'
>>> cmp(set(), '')
Traceback (most recent call last):
  File "", line 1, in 
TypeError: can only compare to a set
>>> set() == ''
False
>>> set().__eq__('')
False

With set, TypeError is raised on __cmp__ and on cmp(), but not on ==. That's because set.__eq__ takes care of returning False if the argument type is not compatible. The end result sounds quite reasonable, because you can still do check for equality against instances of other types (like set() != ''), but can't compare for ordering against them (set() > 1 raises an error instead of doing a weird class name comparison).

I suppose that the roots of this inconsistency are historical accidents. I'm curious to see if all this changed on Python 3.0.

Django on Jython: Summer of Code!

2008-04-23T17:04:00.005-04:00

This post is not exactly hot news, but late is better than never:

My application for the Google Summer of Code 2008, titled “Django on Jython: Supporting Python Web App Frameworks on the JVM” was accepted!! :-)

I can't say in words how happy I am, even after a few days have passed since I got the news. I'm lucky enough to have Jim Baker as my mentor, a very inspiring guy, as motivated as myself by the project. He is a very active Jython contributor, a successful mentor of past SoC projects. And Frank Wierzbicki is already moving DoJ to work on top of Glassfish. From the Django side, Jacob Kaplan-Moss maintained the support shown on the past and will fast-track our patches to Django, if they are needed.

I also got the acknowledgment from Imagemaker, my employer, to stop working full time during the SoC period. In fact, they showed very supportive and interested on the project, looking forward to its results and potential.

I'm going to participate on a vibrant Jython community, with other two fellow students working on web frameworks and Jython: Georgy Berdyshev with Zope and Ariane Paola Gomes with TurboGears2 .

Things couldn't be better. I will resume the work done on the past year, hoping to make Django on Jython a reality by August 2008. In fact, I already started, with a quick but useful web application for tracking the status of the Django test suite running on top of Jython and the postgresql_zxjdbc driver. Hope to get that page fully green as soon as possible.

404

2008-04-21T23:37:00.002-04:00

I saw this on reddit, on a thread about the supposed best 404 error message. But this is one far better:

“I'm sorry, you've reached a page that I cannot find. I'm really sorry about this. It's kind of embarassing. Here you are, the user, trying to get to a page on LiveJournal and I can't even serve it to you. What does that say about me? I'm just a webserver. My sole purpose in life is to serve you webpages and I can't even do that! I suck. Please don't be mad, I'll try harder. I promise! Who am I kidding? You're probably all like, "Man, LiveJournal's webserver sucks. It can't even get me where I want to go." I'm really sorry. Maybe it's my CPU...no that's ok...how bout my hard drives? Maybe. Where's my admin? I can't run self-diagnostics on myself. It's so boring in this datacenter. It's the same thing everyday. Oh man, I'm so lonely. I'm really sorry about rambling about myself, I'm selfish. I think I'm going to go cut my ethernet cables. I hope you get to the page you're looking for...goodbye cruel world!”

(LiveJournal's 404)

You may need to reload the page a few times to get this message, because they have others...

Rails Migrations Gotcha: Backward-Incompatible Model Changes

2008-04-09T09:51:00.013-04:00

I'm pushing for adoption of Rails Migration on all Rails projects on my job (we use them on a few). As a consequence, I won the assignment of writing migrations for the last changes on the system I'm currently involved. That seemed easy, but it wasn't. I will try to show why, without diving into details of my specific scenario.

Imagine you have the following model:


class Foo < ActiveRecord::Base
end

And the following migration:


class AddAnotherFieldToFoo < ActiveRecord::Migration
  def self.up
    add_column :foo, :new_column, :string
    Foo.reset_column_information
    Foo.find(:all).each do |foo|
      foo.new_column = some_calculation(foo.another_column)
      foo.save!
    end
  end
end

Now, we make the following changes to our model:


class Foo < ActiveRecord::Base
  has_many :bars
  before_save :do_something_with_my_bars
  def do_something_with_my_bars
    ...
  end
end

And its migration (just for completeness, not really relevant):


class AddBazToFoo < ActiveRecord::Migration
  def self.up
    add_column :foo, :bar_id, :integer
  end
end

So what is the problem?

For us, who made the last change on Foo after doing the AddAnotherFieldToFoo migration, it's all fine.

But, for the new developer who just made a checkout of the source code and happily executed rake db:migrate, the AddAnotherFieldToFoo migration failed miserably.

That's because Foo#do_something_with_bars will get called (remember the :before_save we introduced), but the association between foo and bar is not made yet (we are executing a previous migration).

Same happens to the developer who didn't update his local copy this week. And it will break on production too, when we merge this set of changes into the production branch.

So, here is my problem:

Every backward incompatible change to models will (potentially) break past migrations, because they are not specifically associated to a model state on the time.

And SCMs doesn't help either (updating one changeset at time would work, but when merging braches all that changesets will collapse into one and you are doomed) I'm looking into what to do. Maybe I'm using migrations in a way they were not intended to be used...

Does someone know how to solve this?

Update: Here is the ruby-talk thread

Cinemark Should Learn Unicode

2008-04-03T21:24:00.002-04:00

Crappy photo taken last weekend on a Cinemark: it shows, in the text on the middle, the title of the movie "Crónicas de Spiderwick" mangled by the incorrect interpretation of UTF-8 data as ISO8859-1.

Someone should point the Cinemark guys to the Joel's Unicode guide. It is a good reading on the topic of text manipulation on the real, modern world. That is, taking text encodings into account. It is a messy, but unavoidable topic.

And remember:

“There Ain't No Such Thing As Plain Text.”

(quoted from The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!))

Out-of-process memoization using memcached.

2008-03-23T22:32:00.002-03:00

This is my humble recognition of memcached (“a high-performance, distributed memory object caching system”) usefulness. It is not a language or platform dependent tool, so do not overlook it just because I tend to talk about Python or Java.

Last weeks I have been fixing a lot of small bugs on a legacy application. After solving most of them, a challenging problem remained: Lots of database-intensive computations were taking too much time to complete. Where "too much time" == 48 hours. Some people suggested me to rewrite all that part of the application, but that was impractical with all the pressure of getting it done in one or two days. And it was the wrong solution, as explained at the end of this post.

So after a bit of research and profiling, I identified two critical points. One was solved by carefully rewriting a lot of SQL queries into just one or two, while modifying the program as little as possible.

The other problem was the program repeatedly computing the same thing over and over. It resembled the naive recursive fibonacci algorithm:


def fib(n):
    if n == 1 or n == 2: return 1
    return fib(n - 1) +  fib(n - 2)

Of course, this is easy to solve adding memoization to the fib routine:


fibs = {1: 1, 2: 1}
def fib(n):
   if n in fibs: 
       return fibs[n]
   else:   
       result = fib(n - 1) + fib(n - 2)
       fibs[n] = result
       return result

My problem when applying this kind of solution was memory space: I feared running out of it, considering the amount of data processed by the program. In the end, I would have to program a little cache subsystem and deal with the feared global state. Or dig into the inner deeps of the system changing a lot of method signatures to pass the damn state along.

But I didn't. I just plugged a java memcached client library, surrounded the computation routine with code very similar to the second fibonacci example, fired a memcached instance on the development server, and it went way faster.

Now everybody is happy, including myself for not having to reimplement the whole computation. Rewriting code from scratch is tempting when nobody understand the existing one anymore. But we tend to underestimate the amount of work behind that unintelligible pieces of code. Much of their ugliness comes from covering a lot of corner cases. Coming with a new, shiny piece of code may not be that hard, but it will be buggy, until it passes a great deal of testing. If we can not understand the existing code, we can hardly thoroughly test the new one, just because we are not understanding all the corner cases. Only after we really understand the ugly code, we can write a better version from scratch. That was not my case.

Just for the record, the computation now takes 3 hours, for a 15x speed improvement :).

How to Demotivate Me at Work

2008-03-08T16:02:00.001-03:00

This a condensed list of things I hate, collected from the past five years of work for multiple software companies. In no special order:

Always assign me to projects where I am the only developer.

Throw me at unfinished projects where the single developer left the company.

Throw me at projects where the massive pre-existent accidental complexity (e.g, stupid technological infrastructure) never gets out of the way.

Put technical decision-making on non-technical people's hands.

Throw me at crappy, badly “finished” projects that no one understand anymore, just after a critical “milestone” happened (such as a very very angry customer, or a last-minute disaster). Bonus points for letting me solve them without help of other programmers. Add even more extra points for no help from other human beings at all.

Move me from project to project, without getting anything really finished.

Put more attention on how do I work than on what I produce.

Disclaimer: This is not a targeted critic against my current employer (or an specific past employer). As far as I know, this kind of shit happens, almost everywhere. But I would really like to work on a different world. Can we change it?

Reading a Binary File on Ruby

2008-03-02T12:58:00.000-03:00

No, it is not as simple as:


contents = IO.read(path_to_binary_file)

Because it does not work well on Windows platforms. You should do:


contents = IO.open(path_to_binary_file, "rb") {|io| io.read }

Obviously, the key is the binary flag. It can not be passed to IO.read.

I learned this because a co-worker had a mysterious problem sending email attachments with rails. I was not able to help her (that IO.read never seemed suspicious to me), but once she found the solution, all became clear.

But you know, the real bug was she using Windows as a development platform, right?.

Generic On-The-Fly Deployment

2008-02-29T09:06:00.002-03:00

Once spoiled by dynamic-languages web frameworks, it is hard to go back to one where a manual build cycle is needed before seeing the changes made to the source code on the running application. It must be instantaneous.

As you can infer from my latest posts, I am now working with Flex. And Java Servlets, using Resin as the server. That environment is not even close to be as developer-friendly as Django, or Rails or many others modern web frameworks. Perhaps I am being a bit unfair, because Resin tries to be developer-friendly, not only reloading the context when something changes, but also providing a special directory where you can put your java sources and it even compiles them. But the project layout is not configured the way Resin likes it, so it did not work.

Other developers use a IDE plugin to deploy the compiled classes automatically. But, is really an IDE needed to simply (re)build a project after some file(s) changes? I do not think so. After all, it is just matter of monitoring the file system and firing the build process if a change is detected. So I came with a simple, handy utility, which monitor a specified directory and run an arbitrary command when a change is detected.

Integrating it with our build script is trivial:


 > cd \eclipse3.3\eclipse\workspace\MyProject
 > python dirwatch.py -c "ant deploy-development"

Moreover, Eclipse integration is easy too: Run-> External Tools -> Open External Tools Dialog -> Program -> New:

Location: c:\python25\python.exe
Working Directory: ${workspace_loc:/MyProject}
Arguments: -u "c:\path\to\dirwatch.py" -c "ant deploy-development"

The whole point of the exercise? Decouple the convenience of automatic deploymeny from a particular IDE, where it does not belong. Just like building.

So here is the program (currently only works on Windows, but should be easy to port to Unix-like systems, using inotify or FAM):

#!/usr/bin/env python
"""
dirwatch:

Monitor a directory for changes. Executes a command after a change is detected.
"""

import os
import time
import win32file
import win32event
import win32con
from optparse import OptionParser

VERSION = "1.0"

def dirwatch(path_to_watch, seconds_to_wait, commands):
    print time.asctime(), "Watching %s" % path_to_watch
    import sys
    change_handle = win32file.FindFirstChangeNotification (
        path_to_watch,
        1, # => recursive
        win32con.FILE_NOTIFY_CHANGE_FILE_NAME |
        win32con.FILE_NOTIFY_CHANGE_DIR_NAME |
        win32con.FILE_NOTIFY_CHANGE_SIZE |
        win32con.FILE_NOTIFY_CHANGE_LAST_WRITE
    )
    # Loop forever, listing any file changes.
    try:
        last_change_time = None
        while True:
            result = win32event.WaitForSingleObject(change_handle, 1000)
            if result == win32con.WAIT_OBJECT_0:
                if last_change_time:
                    if time.time() - last_change_time > seconds_to_wait:
                        msg = "More change(s) detected, " \
                              "waiting %d more second(s)"
                    else:
                        msg = None
                else:
                    msg = "Change detected, waiting %d second(s)"
                if msg:
                    print time.asctime(), msg  % seconds_to_wait
                last_change_time = time.time()
                win32file.FindNextChangeNotification(change_handle)
            if last_change_time:
                if time.time() - last_change_time > seconds_to_wait:
                    for command in commands:
                        print time.asctime(), "Executing '%s'" % command
                        os.system(command)
                    last_change_time = None
    finally:
        win32file.FindCloseChangeNotification(change_handle)
    

def parse_options(args):
    parser = OptionParser(usage="%prog [-c command[;command...]] [directory]",
                          version="%prog " + VERSION)

    parser.add_option("-w", "--wait", dest="seconds_to_wait", metavar="SECONDS",
                      default="1",
                      help="seconds to wait after last change before the "
                           "of the command(s).")
    parser.add_option("-c", "--command", dest="command",
                      default="echo change detected",
                      help="command to execute after a change is detected")
    return parser.parse_args(args)
def main(args):
    options, args = parse_options(args)
    try: path_to_watch = args[1] or "."
    except: path_to_watch = "."
    dirwatch(path_to_watch, int(options.seconds_to_wait),
             options.command.split(';'))
        
if __name__ == "__main__":
    import sys
    main(sys.argv)

Flex Builder 2 on top of Eclipse 3.3

2008-02-26T16:51:00.001-03:00

The Flex Builder installer does not like Eclipse 3.3. When you try to install it over eclipse, it says the destination folder is not valid.

I found some instructions on how to manually plug the FlexBuilder components in Eclipse 3.3, provided that you installed it as a stand alone application. But it contains a small, but important, error on the directory layout, because the "plugins" and "features" directories must be inside "your-new-extension-point/eclipse". That is, the directory structure inside the extension point should be:


./
 /eclipse
 /eclipse/.eclipseextension
 /eclipse/plugins
 /eclipse/features

Other than that, everything went OK following the instruction of the article. The only other different thing I did was to only copy the com.adobe.* plugins and features. But it worked either way.

A Closure-Related Gotcha on Flex 2 (AS3)

2008-02-25T11:27:00.007-03:00

This is one of these write-it-so-you-will-never-forget blog posts. Because I spent much time spotting the bug on code that used closures to simplify the code. It is not practical to show the actual code here, so I will make a dumb "clone" of the relevant section:


function doSomething() {
    for (var i = 0; i < 10; i++) {
        var foo = makeFoo();
        foo.addEventLister(ResultEvent.RESULT, timeLogger(i));
    }
}

function timeLogger(i) {
    return function(event) {
        Logger.debug("Received request number ", i);
        for(var i = 0; i < 100; i++) {
            // For some reason this loop had to exist.
        }
    }
}

[If you are curious, I did not included the inner anonymous function inside doSomething, because all instances of the function would "close over" the same reference to i]

As you can see, the idea was to add some timers for some asynchronous events. See the bug? Yes? I envy you! No? Well, unless you have psychic debugging abilities, it is hard to say without knowing the observed, wrong behaviour: Every logged request had an "undefined" number. That i was always undefined. WTF!?

The bug is the formal parameter i of timeLogger conflicting with the loop variable i on the anonymous inner function. You change either variable name, and the problem is solved. Ta-da!

I suppose this is a side effect of the static analysis and typing made by the ActionScript compiler. This gotcha does not currently apply to JavaScript, because of good old interpreters "dumbness" prevent them to bind variables to definitions that are ahead on the code. But I question the fact that such "clever" idea is a good one. In fact, I do not see a single use case for it.

Review Board: Code Reviews on the Web

2008-01-23T23:07:00.000-03:00

Today I had a happy surprise. In a discussion on a local linux related mailing list, someone pointed to Review Board as a good alternative for doing code reviews.

I looked at the webpage and it looks very, very good. I will check it out as soon as I can, and play with it a bit before trying to “sell” it in my job, but I am pretty confident it will be well received. As a plus, It has people from a recognized company behind it: VMWare. And, another plus for me: it is made in Python, using Django.

I am very interested, because I did something related in the past, using the very same development platform. Before being abducted by the company to do management tasks, I was playing with a pet project to do code browsing, reviewing and project planning, where code review was the first focus. It is called “codeflow”

I left it practically abandoned when my position in the company changed: I had no way to eat my own dogfood anymore. Before that, back in April, I managed to get a working prototype wich did fancy diffs, code highlighting and worked against our subversion repository, although it was not very polished:

We tested the prototype and it showed to be very useful, for real projects. Even
now, one or two months ago, the main engineering team (I am into research and development right now) wanted to put that prototype into production. But I was not sure it was a good idea.

I believe in “release early, release often”. But without enough time to actually release anything, it would be quite difficult to do it often.

So I am now going to push Review Board, as a working product, instead of my rough prototype. As far as I can see, the ideas for codeflow about code reviews were not any bad. I will still miss the integrated concept which I envisioned for codeflow back in April, but I will not miss the reviewing platform anymore.

And it is done on Python/Django. I am familiar and proficient with both. I am still looking forward to do something with the ideas behind codeflow (not limited to code reviewing), and Review Board looks like a great place to start.