Coder Who Says Py

Python 3.1 is out

2009-06-27T15:46:00.003-07:00

As Benjamin Peterson tweeted, Python 3.1 is now available. As always you can read about what's new. Anyone using Python 3 for anything should upgrade as there are some major performance improvements in 3.1 that will make your life better (that includes the new --with-computed-gotos configure flag).

And just to remind people, PEP 370 exists to help you manage your Python installation.

Lazy imports in 25 lines of code

2009-06-02T11:38:00.003-07:00

For some unknown reason to me, I found myself thinking about lazy importing on my way home from school. Now I have never actually dived into the topic before so I had no preconceived notions of how to solve the problem. I do know that people like to have imports be lazy so that the startup time of their applications are not dominated by imports of modules that they do not need immediately, if at all. Many projects have replicated their own solution to this problem such as bzr and hg. This has even gone as far as Greg Ewing proposing actual syntax support on python-ideas and Christian Heimes using lazy imports as a motivating factor for post import hooks.

While the solution varies, the general agreed upon solution is to call some custom function that will return a module proxy. That proxy triggers the actual import once the module is actually used in some meaningful way, such as when an attempt is made to access an attribute. I also assumed that people either used a custom importer (meta path or not) or a custom __import__ function. But beyond that I had no clue how people solved the issue. The importer solution can be seen in a cookbook recipe and the custom function can be seen in peak.util.import (source).

Going in blind on this, I started to think about what it would take to pull this off using importlib in Python 3.1 for Python source. I was not going to worry about built-in modules and the like as importing those types of modules are much cheaper compared to importing Python source or bytecode. And I wanted to solution to be as simple as possible since I was doing this for fun, not as an actual robust solution to a problem I had.

With those goals in mind, I realized that I needed to come up with a lazy loader that could be subclassed for ease of use. I also knew I needed to come up with module proxy object that would trigger an import the instant an attempt was made to access an attribute. Seems simple enough, but obviously it isn't obvious.

First problem I had to solve was how the module proxy would get at the real loader when it was eventually needed. That was quickly solved by assigning the lazy loader to __loader__ and treating the class as a mix-in. Assuming the lazy loader class name was LazyMixin and I set up the inheritance correctly, I could get at the real loader from within the module proxy with super(LazyMixin, self.__loader__). By re-assigning the resulting object to __loader__ I would continue to have the actual loader doing the loading on the module as well have a consistent way to get at the loader I was after. Assuming a module proxy class named LazyModule, the code looks like this:


class LazyMixin:

    def load_module(self, name):
        module = LazyModule(name)
        module.__loader__ = self
        sys.modules[name] = module
        return module

With that problem solved I turned my attention to the module proxy. After some mishaps with trying to dynamically set __getattr__ I had an epiphany: I could simply reset __class__ to a class that lacked the __getattribute__. By having both the lazy module class with __getattribute__ defined and the module class that __class__ gets set to inherit from types.ModuleType I could make sure that the module instance was always consistent in terms of its type. This all led to the following module objects:


  class Module(types.ModuleType):
    pass


  class LazyModule(types.ModuleType):

    def __getattribute__(self, attr):
        self.__class__ = Module
        self.__loader__ = super(LazyMixin, self.__loader__)
        self.__loader__.load_module(self.__name__)
        return getattr(self, attr)

The final problem to work out was how to make sure that the module instance was what got initialized by the import so that all pre-existing references to the module instance worked after the import. The obvious way is to make the module proxy act as an actual proxy; you could store the initialized module in the proxy and forward all attribute requests to the initialized module using __getattribute__. But that has a performance penalty that seems unnecessary and I had a gut feeling there had to be a more elegant solution.

That more elegant solution is module reloading. If you look at PEP 302 you will notice that to support module reloading that a loader is expected to reuse the module found in sys.modules when performing a load. This makes sure that existing references to a module continue to work as the module instance doesn't change, just its __dict__. So by sticking the module proxy instance into sys.modules the real loader would use that module object for the initialization, letting all pre-existing references to continue to be valid.

It turns out that all of this was enough for a proof-of-concept to work! By creating a class that inherited both the lazy loader and importlib's private Python source file loader I was able to defer actual importing of a source file until an attribute on the module was accessed. And the best part was that the only requirement of the real loader was that it follow the rules set out in PEP 302 which importlib provides with importlib.util.module_for_loader. And an even cooler thing is that since the lazy loader is a mix-in that only overrides load_module all other methods from the real loader are accessible, meaning the lazy loader allows for the use of introspection APIs such as importlib.abc.InspectLoader to continue to work! And the greatest perk of all is that since it works with loaders it is completely transparent to the import statement, removing any need to use a custom function to make this work!

To see the proof-of-concept which is NOT PRODUCTION QUALITY -- remember it is using private APIs from importlib which could easily disappear at any time -- see this paste bin. I am sure the code should be more robust somehow, but I am rather pleased with the solution is only 20 source lines and appears to work at least in a really dead-simple example. If people actually like this approach and think they would find it useful then leave a comment and I will see if I can be persuaded to package it up and write the proper tests so that I would be willing to put this up on the Cheeseshop.

The staleness of the standard library and adding new things

2009-05-30T13:37:00.004-07:00

If you happen to follow certain members of python-dev on Twitter you may have noticed a discussion going on over Zed Shaw's "Curing Python's Neglect" post about inconsistencies in the language and the standard library, focusing mostly on the latter. This caused some people to say Zed should be lindberg'd (link is currently down) for complaining about the state of Python's standard library without offering solutions. This then led to Zed responding about how there is a double-standard for accepting changes between Python committers and everyone else which has led to crap making it into the standard library.

There seems to be a misunderstanding here that I don't think is isolated to Zed (and thus this post is not meant to be picking specifically on Zed, he just happened to spark the discussion) involving Python's history and how things operate over on python-dev that I would like to clear up.

I want to start off by acknowleding the fact that the standard library is far from perfect. There are modules in there that are of subpar quality when compared to other parts of the standard library. And even those that are in there and are considered good can have some inconsistencies in their APIs that can make them enfuriating to use on occasion. There is work that could stand to happen to clean up some things.

Today, to get something added to the standard library, we ask that the code be considered best-of-breed by the community, that the developer promise to maintain the code, and that the maintenance happen within Python's development process and not outside of it. There can be a PEP involved depending on how large the code is and how much of a fight there is to get the code included. And of course this is all dependent on whether the core developers believe the module has enough widespread usefulness to warrant adding it to the standard library.

But historically these requirements are a recent turn of events. I would say that these requirements for inclusion into the standard library did not start to be seriously enforced until some time during the Python 2.5 development cycle which started in November 2004 and ended in September 2006. If you look at the standard library a lot of it predates 2.5 by several releases. If you look at the modules that Zed said could use some improvements you will notice that none of them were added to the standard library recently (or even in the standard library as Zed mistakenly thought that setuptools and easy_install were):

os: 1992 (I don't even know what release that is)
time: 1990 (less than a year after Python's creation!)
datetime: 2.3
email: 2.2

My point is that if you look at a module in the standard library and it seems a little stale and that the API could have possibly used some more public vetting then it probably could have but was accepted anyway. People need to realize that Python was started by Guido to scratch an itch and just happened to turn into this massively popular programming language. Python's popularity took off starting shortly after the first PyCon in 2003, IMO. But because the growth was organic python-dev did not realize how much more careful we had to be with standard library inclusions until a few years later. Back before we instituted more stringent acceptance requirements things were added simply if people offered up the code, were willing to maintain it, and python-dev thought it would be useful.

But admittedly, even today some modules get to short-circuit the acceptance process. Importlib is a perfect example of this as I didn't have to put it out there for a year to make sure people thought it was best-of-breed. I didn't have to write up a PEP to vet the API. The only thing I had going for me was a known need for what importlib provides. Otherwise the code went in because I wanted it to go in and I happen to be in a position where I can make that simply happen.

But I skipped the steps only because of other things I did. I skipped the year-long acceptance by the community as I was targetting Python 3.0 which was not released as final yet. I also blogged extensively about importlib and its API so it was at least discussed publicly somewhere. There were several core developers I talked with about the API and who also watched every commit I made with interest and provided feedback. I talked with people at PyCon about this stuff. There is a reason I spent years working on importlib.

But importlib is an exception, not the rule. If my bootstrapping goal did not exist importlib would have existed externally for at least a year before I tried to bring it into the standard library. And if I had gotten any realistic pushback from python-dev I would have waited for inclusion, but thankfully I didn't receive any. And trust me when I say that people on python-dev are quite happy to share their opinion if they think something is a bad idea. While people who have proven themselves do get to skip some steps, don't think that they get to do whatever they want to the standard library.

With all of this in mind, how should modules get fixed? If an API is truly deemed poor and in need of a replacement it can either grow a new API next to the old one or we introduce a new module to make the old module obsolete. Doing the former is nice for stupid little API mistakes, but does require working properly with the pre-existing module which might be a hinderance.

Adding a new module breaks cleanly from the old module, allowing a new API to exist from scratch. But this has the issue of being a much larger burden for support as there will simply be more code. There are possible stability issues, etc. Even if code has existed out in the public for a year does not guarantee top-notch quality.

And of course this all assumes that an API is actually fundamentally broken. Minor issues can be overlooked in the name of prior knowledge. When we add something to python-dev we are essentially asking every Python developer out there to learn about this new code and consider using it which is quite the mental burden if you have been using the older API for years. And it is also possible that you are in a minority in thinking it is broken.

This is all rather complicated and nuanced. Coming up with a solution that pleases everyone is impossible. But python-dev continues to try to do its best and hopefully that does please most people.

Month one of my python-dev sabbatical

2009-05-08T15:46:00.002-07:00

It now has been roughly a month since I disabled subscriptions for practically every mailing list I belong to. Initially I had to get used to the fact that checking Gmail constantly for new mail was a fruitless exercise. Otherwise it's been nice.

I definitely have more time on my hands. Python-dev has been my hobby for so long I had somewhat forgotten what it was like to just work on a coding project purely for fun and just for myself. Not having to answer to the Python community for the quality of my code or test coverage, not having to document everything, or have to argue for why I did something is rather freeing. I can see why Neal Norwitz is on perpetual sabbatical from python-dev.

But I do miss being plugged in. Having to get my news through Twitter about what is going on still feels odd -- this was made especially true when PEP 383 was accepted and I didn't even know the PEP existed until people tweeted about the acceptance.

And I still think about my long-standing projects. I am thinking about how to handle the Mercurial transition and what workflow we should end up with. I still think about what I might need to implement in C for importlib in order to prevent people from lynching me when I bootstrap it in at the implementation of import. This means that once I am done defending my thesis proposal I will be coming back to python-dev.

But I suspect I won't rejoin every mailing list I disabled delivery for. And I also expect I will be much quicker on telling people to move over to python-ideas along with very liberally using the muting feature in Gmail. I have had a taste of freedom from senseless arguments and whining and I don't plan to go back to it without a fight.

One month down, two more to go.

Does XHR lead to better testing/abstraction?

2009-05-04T14:34:00.003-07:00

I was talking with a friend of mine who is a Ruby programmer who does Rails development and I asked her how best to handle a form submission that was malformed. I am thinking of the situation when I have a URL that accepts a POST from a web page and some argument is in the wrong format or an argument is entirely missing. I wanted some way to signal the POST submission was bad directly from the HTTP status code or something. I mean you should have some clear way to know that something failed along with a message as to why it failed.

But she said it should return a 200 with response page specifying what went wrong. When she said that my TDD sensibilities along with my separation of concerns training recoiled in horror. To test the URL response I have to parse the HTML for error output?!? That means I have to inspect the view to know that an error occurred! And this is simply not testing the view to make sure an error was made visible to the user, this was to test the error was caught, period.

After I recovered from my shock at this suggested practiced and realized most sites worked like this I realized why it offended me so much. Just like everyone else I was taught that MVC was generally the best way to structure GUI applications, and in general I agree with the assessment. In my opinion an application should function regardless of what the GUI iss, making the front-end a separate component of the overall application. One should be able to swap out the "V" from MVC and have things still work. That's just good abstraction in my opinion. That means I should be able to test the core components of an application -- the M and C -- without a GUI. And yet with most web apps we do not get this separation thanks to most forms signaling a failure based on what is displayed in the reply.

With this in mind I began to think about how I could rectify this situation for my web app I was developing. And that's when I realized that using XHR to send a form's data to a URL and use the response to handle errors was, from a testing perspective, the best way to go. It might make the client-side code more complicated as a click in a form would no longer simply be the case of the web browser redirecting to a specific page but require processing a response -- probably JSON -- and handling the reply. But from a testing perspective I would be able to test the submission URL in isolation from the web page, thus separating the V from the M and C. Plus it would give me a REST API upfront and thus not require me to create it later -- if I chose to document the API and make it public.

That's when I said to myself, "OK, so how does everyone else handle signifying an error when it comes to errors in a REST API"? And that's when I found out everyone does it there own little way. About the only thing I found that was consistent is that if everything went well the URLs returned a 200 and when it went to hell they returned a 404. But from there there was no consistency. Only looking at JSON responses, some included the HTTP status code in the error message while others did not. Some had the idea of a class of error and a message while others only had a classification. Some had no answer whatsoever (I'm looking at you XML-RPC), and some went over the top (that would be SOAP).

So I began to wonder about what I would do to signify a failure for a REST call. The first thing I agreed upon was that returning 200/404 was reasonable. It does irk me slightly since it is just like returning 0/1 in C to signal an error, but it works in this situation where one does not have proper exceptions nor other status codes to use to signal different types of errors. I could inspect the returned JSON object for an 'error' attribute to realize an error object was returned, but this works just as well and I would rather dispatch on status code than have to introspect the returned value.

Having chosen a way to signify an error, I then thought about what the JSON reply would require. I thought about how Python signifies errors and almost went down the road of something like exceptions complete with inheritance. But I quickly realized that would require loading JavaScript code just for defined exceptions and I didn't like that for an exposed API; for a REST API one should be able to just read the API docs to make a call and not require including any special JavaScript code. I also realized that this was an API issue, not a programming issue. I was more interested in how Python handled errors when calling functions than how bad syntax was flagged.

That means that I wanted something like TypeError when an argument is missing and ValueError when some input is malformed. That's when I figured why can't I just do something like that? If I had a JSON object have an 'error' attribute that specified the type of error, like "TypeError" or "ValueError", I would have my TypeError/ValueError in JSOn. A 'message' attribute would work as BaseException.args[0]. And then I could tack on arbitrary attributes on this error object for error-specific information such as what argument was missing or what kind of format was expected for a field.

And so all of this is what I plan to do the next time I have any data that needs to be sent to a URL for something that is public-facing and and not a hack (although honestly, isn't everything hack and we just happen to be willing to chance having the public use it?). Yes, it complicates the client-side stuff, but actually the JS code could be made into a library to nicely flag errors, etc. if one standardized on the type of errors that would be returned. Plus it makes unit testing the controller much easier. And it forces you to have a REST API upfront so you don't have to design it later.

Oh, and if anyone says, "what about XForms?", I will say, "get it in all of the major web browsers and then we can talk".

Designing the model schema for part of my father's web site

2009-04-28T20:53:00.002-07:00

Since I am still on sabbatical from python-dev and my personal web app is not ready for public consumption or discussion yet I have decided to keep this blog going by talking about the steps I am planning to take to re-implement my father's business site from a static PHP site to an App Engine/Django one (plus hopefully fix it so it no longer reflects its initial conception in the late 1990's).

The first part of the site I plan to redo is the wall coverings section. Looking purely from a model schema view, the wall coverings fall under one of two baseline paper collections (China Wall and Caravan). Here is how I see the model for the collections looking (I am using shorthand here but it should be obvious how this will translate into an App Engine model definition).


BaseCollection:
    """A base paper collection (e.g. Caravan and China Wall)."""
    name = String
    one_liner = String
    description = Text

Nothing fancy and covers the basic text needed for the collections. Now within each collection there are color lines and what I call "fancy" lines (i.e. paper lines with more than just color added). For the color lines (e.g. Imperial Scrolls), the model is very much like the BaseCollection model.


ColorLine:
    """A color line (e.g. Imperial Scrolls)."""
    base_collection = Reference(BaseCollection)
    name = String
    one_liner = String

Beyond the reference to BaseCollection there is really nothing special here. By adding the reference back to BaseCollection instances of the class will grow a colorline_set attribute the contains the models that reference that instance.

But having some text to explain a color line is rather useless without the individual papers that make up a color line.


PaperColor:
    """A color of paper."""
    name = String
    picture = Blob
    color_line = Reference(ColorLine)

With this model I will be able to store the scanned color samples of the paper and associate them with the proper color line. This will give me a papercolor_set attribute on ColorLine instances, making it easy to generate those color swath pages.

But all of this was the easy stuff. The tricky part is the fancy lines (e.g. Linea). For these fancy lines they are associated with a specific color line along with the customization options. Since my father has been in the wall covering business for over a decade and so far every fancy line they have has no more options than patterns and one other custom thing, I think I can handle those cases with three models.


Pattern:
    """A single instance of a pattern for a FancyLine (e.g. a single pattern
    for Linea)."""
    fancy_line = Reference(FancyLine)
    name = String
    picture = Blob

FancyOption:
    """An option for some paper (e.g. pen ink color)."""
    name = String
    description = Text (?)
    # fancyline_set
    # option_set

Option:
    """A single option for a FancyOption (e.g. slate pen color)."""
    fancy_option = Reference(FancyOption)
    picture = Blob
    name = String

Pattern is an instance of a pattern for a fancy line. Since patterns are not shared I don't think a model to collect the patterns separately from a FancyLine instance is needed. As for FancyOption, I made that collecting model since it can be shared between lines (e.g. all the lines that use pens have the same color options since the same set of pens are used). Each option is thus covered by an Option instance which will store some scanned representation of the option.

But this is where my rudimentary web site experience ends, and unfortunately there is more to do. I still have to have installation examples. I am thinking of doing a model that takes an upload of a full-size photo and auto-generates the thumbnail upon upload (should be easy with the images API from App Engine). Trick is then somehow pulling from these to show up in the sidebar of their respective paper line pages. I am thinking that adding a boolean to the model so that a simple filter for the photos with the boolean set to true will be enough to pull up the preferred installation photos for the main pages.

I also have to deal with the flame spread reports. There are only three of them: one for each base paper and one for a specific fancy line. That means dealing with the single outlier. I am tempted to make a model for the reports, have a blob for a PDF, and then having a generic reference in the model to the most generic line it represents. That way I can check if for each line if the instance has a flamespread_set or whatever with an instance to know whether a special flame spread report exists.

And then there are installation instructions which are in a similar spot as the flame spread reports.

I think all of these models cover what I need. Hopefully I have not generalized too much or painted myself in a corner. I don't think any of this will make adding or editing content by my father on his own too difficult (which is the entire reason I am redoing the site), especially if I make sure that the forms to change stuff is dead-simple.

And if I am doing something silly, please let me know.

PyCon 2009 recap: Best PyCon ever!

2009-04-03T16:44:00.003-07:00

I landed in Chicago midday in hopes of catching the tail-end of the VM summit. After grabbing my luggage and finding the shuttle to the hotel idling at the curb -- which took a little to find as the signage pointing you to where the shuttles wait is not good at ORD -- I got to the hotel, checked in, and went on down to the summit. There I found a room with a bunch of VM implementors standing around talking.

Turns out that the entire summit had been like that. After brief introductions by each team -- including the public unveiling of Unladen Swallow -- apparently everyone just started chatting in small groups and that continued until people broke up for dinner.

When I arrived I instantly started chatting with the CPython guys in the back of the room. One of the great things of PyCon is getting to see friends in person that I typically only get to see at the conference. It's one of those situations where you just start talking almost instantly without a huge build-up of what anyone has been up to since we read each other's blogs, IM or Twitter, etc. Chatting went on for several hours as more people began to show up at the hotel. Eventually we stopped for dinner at a place a block away from the hotel, after which I just slept.

Thursday was the first Python language summit. Originally there was going to be a morning session of general discussion followed by a 2.7/3.1 discussion with parallel tracks midday/afternoon, but those plans were scrapped. Instead we ended up all staying in the same room talking all day.

During the initial session we all just talked about various things. For me it became very obvious that the DVCS was going to be a running topic of interest for the rest of my stay at PyCon. I don't remember anything specific coming up in the first session.

For the 2.7/3.1 session several things came up. One was that it is very likely 2.7 will be the end of the line for the 2.x series for Python. While the idea was floated of canceling 2.7 entirely, that was disliked and instead the preferred approach of strongly stating the opinion that 2.7 will be the end of 2.x was chosen.

It was also strongly stated that new features cannot exist only in 2.7. Even more important, though, was that people should strongly consider adding new features only to 3.x in order to help entice people to move to 3.x.

This discussion segued into the idea of 3to2. The idea sprung up at PyCon 2008 but never went anywhere other than being an idea. During the summit, though, the various framework implementors said that they would love to get to code in 3.x and then run 3to2 over their code to get 2.x code for their compatibility requirements. Thanks to 3.x having clearer semantics -- clear separation of Unicode and bytes, nicer syntax, etc. -- it is actually easier to go from 3.x to 2.x than the other way around. There was general excitement that this idea could pan out and be a great way to get adoption of 3.x to pick up.

In terms of Python 3.0 it was decided that 3.0.2 would be the last release. We will not even bother with security fixes for the release as the user-base is so small and 3.1 such a nice step up in performance that it will not be worth our time to maintain even security patches for 3.0. Obviously this is a very unique situation and we are not about to start doing this on a regular basis.

After lunch was the alternative VM session where we discussed how other VMs -- basically Jython, IronPython, and PyPy -- fit into all of this. With Unladen Swallow effectively acting as a long-term branch of CPython the discussion focused around the other VMs and how to put them on equal footing with CPython. My big suggestion, which was surprisingly well-received, was to pull out the part of the standard library implemented in pure Python, all of the tests for things that all VMs are expected to implement, and the documentation and place it all as a top-level project within CPython's code repository. The idea would be for all VMs, including CPython, to pull in the standard library from this project along with the tests they are expected to pass. That way we can pool development resources on the standard library along with making sure the test suite properly reflects what a Python VM must pass to be called a true implementation of Python. The versions of the language will still be tied to CPython but at least the language will seem to be less of "what CPython does" and more "what the test suite and standard library say it is". And to help facilitate this the various major contributors to the VMs have been given commit privileges to help start cleaning up the test suite. This should start to happen once we switch over to Mercurial.

The last session of the day was on packaging. The general consensus of that session is that distutils needs to be cleaned up and morph into a support library that third-party tools can be built off of. So tools like virtualenv or buildout won't go away and be replaced by distutils, but they should be able to harness more of distutils to have a more common base. It also looks like an extensible, declarative format is going to be developed for those situations where a setup.py script is overkill. Tarek is spearheading this endeavor and hopefully he will be enough of a dictator to make sure it happens.

Friday was the first day of the conference where I got to give the first lightning talk on why Git was no longer being considered in the morning (I start at 3:47 in the video). I also had my talk on how Python is developed along with the Python VM panel. I also somehow got pulled into giving my shorter-than-I-remembered intro talk on sprinting on Sunday along with a tutorial on how to contribute to Python.

As per usual for me at PyCon I missed most of the talks at the conference. The ones that I saw which have videos online as I write this are for Pinax, IronPython implementation, IronPython status update, intro to multiprocessing, and concurrency and distributed computing in Python. All of those talks were entertaining and are worth watching if you are interested in the subject matter. Otherwise I just attended all of the lightning talks and the keynotes -- the one thing of interest during the keynotes was Guido's talk when he said he was "tiring" but not "retiring".

In terms of ancillary activities during the conference I had the usual Python Software Foundation stuff to deal with. We elected a lot of new members along with expanding the board to ten people -- I am still on it -- and keeping the trademark of the Python logo.

I also was the co-conspirator of the "VAN LINDBERG'D" catch-phrase. Jesse Noller was my roommate and we were up late one night and somehow we were saying we wished we could force the idiots online who put time and effort into trash-talking Python and spend that time instead attempting to contribute to help fix the situation they were unhappy with. Somehow we thought about changing the Python software license such that if you complained without helping we would send Van Lindberg, the PSF lawyer, after you and take you down. We dubbed it being "VAN LINDBERG'D" and it led to lindbergd.info and Jesse's very entertaining lightning talk.

Then came the sprints. The core, as usual, had a good turn-out. This year, though, I at least was able to tell people to work on tests to raise test coverage if they didn't have anything to work on. Having the new developer docs that I have written helped a lot since I finally had something concrete to point people to in order to get up and running.

I personally had no agenda beyond try to help people where needed and to clear out my backlog of issues (which I actually did in the end). Various people worked on things and last I heard from Georg Brandl we had closed over 125 bugs Wednesday night, of which just under half were closed by Georg himself.

But the great thing that happened during the sprints was not the coding but the camaraderie that broke out amongst Django and Pinax Tuesday. The core team was next door to the Pinax sprint room and at some point on Tuesday we heard some laughing continually break out. Turns out a little "war" had broken out between the Django and Pinax teams, culminating in several domains being purchased and the Django pony being made to do things no stuffed animal should have to do. Eventually there was a peace accord. James Tauber plans to write up a blog post about what happened so I will let him tell the story. For the core's part some of us got pulled into all of it, leading to ponysex.us coming into existence (the site is Safe For Work).

The night that ponysex.us came into existence was also the night before April Fool's. The idea for the Python April Fool's for this year took a couple of turns before it ended up at what was posted online. I originally came up with the idea of doing a blog post where I helped lead a coup against Guido to usurp him as dictator and install Barry Warsaw as my puppet with a future statement for adding back in the diamond operator to massage Barry's ego to help keep him in line. I had that idea nearly a year ago and told Guido about it during my internship. He liked it so I did the future statement on the flight to PyCon and let Barry in on the idea at the conference.

That's when Barry decided he wanted to tweak it. Because of Guido's whole "not retiring, just tiring" bit in his keynote, Barry wrote up PEP 401, suggesting Guido was retiring and would be replaced by Uncle Tim. That made the use of the diamond operator not quite as poignant as Tim has never expressed a dislike for '!=' while Barry and Thomas Wouters have both stated that they preferred '<>', but it still would be funny.

Then Guido decided to tweak it to have Barry be the successor and throw in the idea of Barry switching us to Bazaar instead of Mercurial. I am sure getting back his precious diamond operator in Python 3.1 has made up for any possible disappointment from Mercurial being chosen over Bazaar for Barry.

And then there was Wednesday night. At some point the Django folks relocated to the Pinax sprinting room. Since there were not that many core sprinters at the time, we decided to relocate to the room where the laughter was coming from. After sitting down, one of the core developers, who will remain nameless to prevent embarrassment with any potential future employers, seemed determine to get his drink on -- a lot of us were happy to join in of course. Eventually some people went out and bought some liquor, which led to the core team finishing off a bottle of Knob Creek and a little bit of scotch -- most of it was drunk by one individual and myself. I actually managed to check in code that night, although there was an unfortunate svnmerge mishap that deleted the metadata for the tool. But otherwise it was a fun night of drinking and joking all around (read Twitter of the folks attending to get an idea of what the heck was going on).

In the end PyCon 2009 turned out to be the best PyCon I have attended. The language summit was fun and seemed to produce some results. I got in a great amount of socializing during the conference. And the sprints were just awesome. PyCon 2010 in Atlanta has a lot to live up to for me personally.

I should also probably mention I am going to take three months off from Python development in order to focus on my thesis proposal which is due by July 1. It will also give me a chance to finally redo my father's web site in Django -- maybe even Pinax -- so that he can update the information on the site himself in a manner easy enough that even his partner could do it -- he just learned about copy and paste, although he can navigate the web fine -- while leaving me in complete control of layout. In other words this blog's focus will be shifting for the near term to web programming stuff and I might be suffering from Python withdrawl.

Guido has retired as BDFL

2009-04-01T00:01:00.000-07:00

PEP 401 says it all: Guido has retired. If you attended PyCon you heard Guido's keynote where he mentioned he was tired but not retiring. Well, "tired" turned to "retired" rather quickly once the sprints started.

To replace him as BDFL is Barry Warsaw, Friendly Language Uncle For Life (FLUFL). Read the PEP for details, but the things on the FLUFL's agenda is:

Switch to bzr over hg.
Move to '<>' over '!='.
Back to 'print' as a statement.
Shut down 3.x and backport everything to 2.7.
Move over to Parrot as the VM for Python.
Disband the PSF.

To help with the transition I have implemented a new future statement to make the diamond operator -- '<>' -- the one true way to write NOTEQUAL over '!=':

from __future__ import barry_as_FLUFL

This has been checked into Python 3.1.

Why Python is switching to Mercurial

2009-03-31T11:59:00.003-07:00

Starting at PyCon 2008 thanks to Barry Warsaw and the Bazaar team I started thinking about moving Python over to a distributed version control system (DVCS). While I wanted to get offline commits for the benefit of non-core developers along with easier merging from 2.6 to 3.0 (ah, the days when there are only three branches under development), I knew that would not necessarily be enough of a reason for others to switch.

Come October I started a PEP for switching off of svn to a DVCS. Originally it was going to be hg vs. bzr, but enough of an outcry on python-dev led to me to relent to adding git to the PEP. With the list of DVCSs decided I began writing up common use cases that I and other developers have come up against in developing Python. I then had a representative for each DVCS fill in the PEP with the best solution to the use case (who I am grateful to for helping). This all became PEP 374.

And for me that's when the stress began. I was being bombarded on all sides on this pretty regularly. I quickly realized that choosing a DVCS was like choosing a code editor; it's a very personal thing for a lot of people. Plus I forget how big Python is now; when what I was doing hit the Net I ended up talking with developers from all three DVCSs which I didn't expect.

Time past and I tried to absorb all three DVCSs as much as I could, although with my internship and trying to finish importlib for 3.1 I only had so much time. I ran a survey of the core developers where I asked them to rate the three DVCSs as either better, equal, or worse than svn if they felt they had enough experience to have an opinion.

Based on the results of that survey where git was clearly the most disliked tool of the core developers, having the weakest Windows support, and not being implemented in Python, I decided to eliminate git from the running and announce its elimination at the first lightning talk at PyCon.

When I arrived at PyCon pretty much everyone asked me about the DVCS PEP. People wanted to know how it was going, who was going to win, and giving me support/pity for what I was going through. Guido noticed this and decided to end my misery by saying he wanted to make a decision by the end of PyCon. I said I was fine with that as one was already about to be eliminated and I knew my personal preference at that exact moment aligned with Guido's.

So I did my lightning talk eliminating git. Luckily that went well with only about two people telling me directly they disliked the decision.

But the more telling thing was what everyone else told me after that lightning talk. I ended up with a surprisingly large number of people telling me -- including core developers -- they wanted, preferred, or guessed that hg would now win. Now the guesses could be explained away by Guido having publicly stated he likes hg, but to me the amount of people telling me they wanted hg to be chosen was surprisingly large. And honestly no one told me they preferred bzr (although no one said I better choose hg over bzr either).

So Monday morning came around and I walked into the sprint. I asked Guido if he was ready to make a decision. He said yes, we both said hg, and so Guido tweeted the decision before telling python-dev that we chose Mercurial.

There has been a lot of speculation as to why Guido pronounced the way he did. On Twitter Guido said to read PEP 374 for the reasons. Since I helped write my PEP my reasons are reflected in the PEP.

Obviously community preference as shown at PyCon played a role. No one wants to choose a DVCS that causes the community to not want to contribute to Python. And I would never choose a VCS that would cause Guido to not want to work on Python. Some people seem surprised that something non-technical played a role, but ignoring social issues is to ignore how much open source is a social phenomenon. And we are not the first project to take social preference into consideration: I know both GNOME and Pinax chose git because their developers preferred git.

And there are technical reasons. Having hg being faster than bzr by 2x to 3x does matter to some extent. No one wants to cause someone to not contribute because they didn't want to wait for a checkout. And having personally experienced long checkout times because of a subpar connection to a specific server I know this can occur. The performance margin between hg and bzr is within reason typically and is not a flat-out deal-breaker, but it doesn't help either.

Bazaar also has its short timespan of format stability working against it. The tool has changed its format at least three times based on what the man page says (1.0, 1.6, and 1.9). Mercurial, on the other hand, has been stable since I think it went public or near that time. They take great pride in the fact they have not changed it. And that stability more aligns with python-dev's sensibilities regarding stability.

Stephen Turnbull's explanation of why on the bzr mailing list is also a good explanation of why we chose hg. Basically no one is saying bzr is bad, just that hg is a better fit for our needs on python-dev.

But the thing I really love about having made this decision -- other than I don't have to stress about this anymore -- is that everyone seems to be flat-out happy we made a decision to switch as well. Once again the Python community stands out as being friendly and understanding about stuff like this with no one really seeming to be upset that we made the decision we did.

As for when the switch will happen, I don't know. We are hoping by summer, but that is just a hope at the moment. We have to figure out the best way to convert our history as well as what workflow we want to have.

Consider volunteering at PyCon

2009-03-19T15:34:00.002-07:00

Doug has sent out a request for help at PyCon. If you will be attending PyCon and are willing to help, please do! PyCon is volunteer-driven so we need all the volunteers we can get.

Last day for online registration to PyCon

2009-03-18T13:23:00.001-07:00

Helping to spread the word that today is the last day to register for PyCon online. You can still register at the door, although obviously it is a little bit more expensive.

Use XMPP for microblogging?

2009-03-16T19:14:00.004-07:00

I had something interesting happen to me today: people noticed my status message on Google Talk. Normally I figure no one really pays attention to status messages for IM beyond whether someone is online or not (we all know people who are always listed as away or have not changed their status message in ages). But for some reason today I decided to use my status message to bitch about the car issues I was having.

And people responded. I had various friends IM me throughout the day inquire about what was going on (short answer: transmission is shot and the repair bill is more than the value of the car, so I am trying to get the car recycled; not easy when you drove down to visit a parent in another country). It reminded me of what can happen when I post to Twitter or FriendFeed when I get to have a topical conversation about what is going on in my life that exact second; instantaneous blogging in snippet form. And it was nice to get to have a conversation with some friends about what was happening. That's something I don't necessarily get to have on Twitter since the threading of the conversation gets all out of whack. And unlike FriendFeed the conversations were private and happening in real-time.

But the other nice thing is that these conversations were with people I normally don't get to interact with on a regular basis online. Typically I interact with people online through FriendFeed, Twitter, and Google Reader. IM has a much broader reach than any of these newfangled services have.

For me, the difference between the former services and IM is one takes more proactive participation while the latter is usually only for when I have a specific need or desire to talk with someone. If the people I follow on FF or Twitter post something I will eventually see it when I actively check the application (although obviously you can use notifiers so that the engagement is passively triggered). But with IM, it is typically relegated to the background, sitting there until either I initiate a conversation with someone or vice-versa.

And yet today that didn't happen. It makes sense that ones status in the world at any moment be tied to their IM persona. I mean my status of being online or not is listed in IM, why shouldn't what I am thinking about or reading also be displayed? We talk about microblogging and lifestreaming services, and yet the one service we all use that portrays our status online is not doing more than showing a green, red, or gray dot next to some picture. It feels like an unneeded disconnect to me that my IM persona does not tie into what else I might share with the world at that moment.

There is also a case that IM is a better model for microblogging/lifestreaming. It's very much a push model; I decide to say something to the world, it gets pushed to the people who care to listen. And yet all of these other services use a pull model to get the information. Why should my Twitter client have to refresh to find out that there is something new to read? Can't it just be pushed to me? Can't the pull from a web site be a side-effect of providing access while the actual system is push?

If you listen to FLOSS Weekly episode 49 you will hear Peter Saint-Andre mention how he thinks XMPP is a good system to use for microblogging. I personally buy his arguments. Apparently I am not the only one as people have blogged about this idea and gone as far as to sketch out a proposal of microblogging over XMPP. But what exactly would microblogging tied to IM act like?

First thing first, though, is to realize how microblogging versus IM is different. For one, anyone can follow me on Twitter or FF but not on IM. If some stranger wants to follow me on FF that's fine, but I don't necessarily care when they are online or want to have a personal conversation with them. So any microblogging service would need to make the idea of subscribing to one's microblog feed separate from getting to interact with them directly over IM. Another difference is the temporal nature of IM status messages versus microblogging. When I change my IM status, that last one vanished into the digital ether. But for microblogging, I don't necessarily want that to happen. So a ticker would be needed to keep track of the messages that have happened throughout the day that I may have missed.

So what are the benefit of tying microblogging to IM? For one, I could easily respond directly to someone about something and have a truly real-time conversation with them about it. Yes, Twitter has direct messages, but the fluidity of an IM conversation, I feel, is much better. It also simplies things by not forcing me to run yet another service.

Jaiku actually tried this approach somewhat. While being a lifestreaming app, they also had their S60 client which updated your online status and your location in the world. But I don't think it included IM services which would have been nice.

Or maybe all I really want is better support for longer IM status messages. What I honestly get out of Twitter is what people are currently up to and some conversations. If longer status messages could easily be displayed and a public group chat around my status message could take place that would be close to what I get out of Twitter with the only thing missing is people overhearing my public conversation.

Google Moderator pages for PyCon VM panel

2009-03-15T10:20:00.003-07:00

I organized a VM panel at PyCon this year where a representative from each of the major VMs (CPython, Jython, IronPython, and PyPy) will be in one room to answer your various questions. To help get questions from people who cannot attend along with making our esteem MC's job easier, Jacob (who is the moderator) and I have set up a Google Moderator page for the panel to propose qauestions.

Please propose questions or vote questions up and down so that Jacob can come with a battery of questions for the panel to fill in lulls during the Q&A. And I think all talks are being recorded this year so the panel should eventually end up online.

Oh, and if you are not attending PyCon, you still have a chance to register and come!

Time deltas between Python releases

2009-03-11T16:58:00.004-07:00

I have been thinking about what needs to be covered during my session at the language summit at PyCon which will cover 2.7/3.2 issues. One thing I have been wondering about is whether Python should consider shifting to a time-based release schedule. The rule of thumb has been that we do a minor release (e.g 2.7 is the next minor release for on the 2.x series) every 18 months.

To see if that 18 month idea was true, I created a chart (thanks to the Google Chart API for existing and Doug Napoleone for always hyping up the API). The x-axis is number of days and each section of the graph is the length of time to develop a micro release, although developing x.y.0 starts from the release of x.y-1.0 so that's why the initial values are so large.

It would seem that we are trending away from 18 months between minor releases rather quickly. While I would never propose a hard deadline (that's just stupid for open source when you are not trying to get a packaged piece of software done by a specific date for shipment), I wouldn't mind planning out release dates after the last minor release went out so we know when exactly stuff needs to get out (minus any bumps; see Python 3.1 and its planned 6 month release cycle).

The chart also points out some interesting things. One is that 2.3 was a trouble-maker. =) It has the most micro releases (2.30 through 2.3.5) on top of having the shortest amount of time between micro releases (10 days between 2.3.1 and 2.3.2 thanks to a bug). The 2.5 branch has been actively maintained the longest (4 years, 24 days from 2.4.0 to 2.5.4). And a random fact for myself: 2.2.3 was the first release which occurred while I had commit privileges.

Interacting between and Java

2009-03-09T14:02:00.028-07:00

[2009-03-15: This blog post has been thoroughly rewritten after I found out some details were wrong and people wanted more details]

I don't like Java. If care to find out why you can search my blog, but it's no secret Java is not exactly at the top of my list of favorite programming languages. But I am not stupid enough to think that Java is about to go away. And I have nothing against the JVM, just the primary language that runs on it.

Thanks to the JVM, just because I have to work with Java code does not mean I have to write Java code. Thanks to various languages being ported to the JVM there are now multiple options for working with Java code in languages other than Java:

Jython - port of Python.
JRuby - port of Ruby.
Rhino - port of ECMAScript.
Clojure - Lisp dialect.
Groovy - Scripting language for the JVM.
Scala - object-oriented, functional language for the JVM.

By targeting the JVM all the above languages can call Java code in order to be relevant in a Java-heavy world. But just because they can easily consume Java code does not mean that the reverse is true. What I am looking for is not just a way to call into Java, but to call other languages from Java.

To better explain this, take the following three files. First, I have Spam.java:


public class Spam {
 public String serves() {
   return "spam";
 }
}

I have another class that implements a subclass, BaconSpam.java:

public class BaconSpam extends Spam {
 @Override
 public String serves() {
    return "bacon " + super.serves();
 }
}

And finally, the class that runs the show, Waitress.java:

public class Waitress {
 public static void main(String[] args) {
   BaconSpam menu = new BaconSpam();
   String food = menu.serves();
   System.out.println("We serve " + food + "!");
 }
}

What I am after is a language to rewrite BaconSpam.java in another JVM language such that I don't touch Spam.java. I also want the changes to Waitress.java to be minimal or non-existent while still having to store an instance of the class and the value returned by serves() to show how objects would be stored and used in a long-running Java application (i.e. no cheating by inlining some call in the println() call to make the example a little tougher and more "real world").

Jython

Thanks to the Jython guys I was pointed to a Jython Monthly article from October 2006 that explains how best to go about accessing Jython code from Java.

To start I wrote BaconSpam in Python:

import Spam

class BaconSpam(Spam):
 def serves(self):
   return " ".join(["bacon", Spam.serves(self)])

While that was rather simple, there is still the issue of getting an instance of the class. Because Jython dynamically interprets Python code you can't simply drop in BaconSpam.py and have it work with Waitress.java. You need to create an instance of org.python.util.PythonInterpreter and interface with it to get at the Python code:


import org.python.core.__builtin__;
import org.python.core.PyObject;

public class Waitress {
 public static void main(String[] args) {
   PyObject BaconSpam = __builtin__.__import__("BaconSpam").__getattr__("BaconSpam");
   Spam menu = (Spam)BaconSpam.__call__().__tojava__(Spam.class);
   String food = menu.serves();
   System.out.println("We serve " + food + "!");
 }
}

Because Jython dynamically loads Python code, Waitress.java has to be modified. Luckily a large chunk of the code is boilerplate that can be extracted out into a factory class to help simplify things.

And just a warning for anyone wanting to run the above code: for some odd reason the above code only worked for me with Jython 2.5b3, not 2.2.1.

JRuby

Writing the BaconSpam subclass was simple:


class BaconSpam < Spam
  def serves
    "bacon " + super
  end
end

But as of right now you need to use JSR 223 to interface with the class which is worse than the Jython approach, so I am not going to go through the steps here.

But Charles Nutter has blogged about adding signature support to JRuby's in-dev compiler2. It looks like as soon as inheritance is handled properly in compiler2 that JRuby will be in the same position at Groovy and Scala for ease of integration (see below for details on those two languages).

Rhino

I don't like JavaScript, so I am skipping Rhino. =) But it's another JSR 223 approach.

Clojure

I actually didn't get Clojure to work. I tried to follow the gen-class example from the Clojure wikibooks, but ran into several issues that included having to manually execute compilation for the Eclipse plug-in to even get an error message and putting the code in a package to get Clojure to not assume I was working off of java.lang when inheriting from Spam.java.


(ns pkg.BaconSpam
(:gen-class
 :extends pkg.Spam
 :exposes {serves servesSuper}))

(defn -serves [this]
(str "bacon " (.servesSuper this))
)

With an error of "java.lang.IllegalArgumentException: Don't know how to create ISeq from: Symbol", I just stopped trying to make this work. I assume Waitress.java will need to be changed beyond just being put in a package anyway in order to deal with Clojure's dynamic typing.

Groovy

Writing the Groovy version of BaconSpam was very easy thanks to the language having been designed for the JVM from the outset. The only trick was that serves() needed to have the return type specified for the method instead of being dynamic:


public class BaconSpam extends Spam {
 // Using 'def' makes return value dynamic.
 String serves() {
   return "bacon " + super.serves()
 }
}

With the typed method there is no need for modifying Waitress.java. Groovy basically ends up looking like Java with some syntax removed.

One issue that did come up with writing the Groovy example was that the Eclipse plug-in is in some bad shape; I couldn't get it to run the project. This drove me to download and use NetBeans since it has Groovy support included. That was a much better experience since it actually worked.

Oh, and the docs suck. Took way too long to figure out how the darn language is even structured. Just had to read various examples to figure things out.

Scala

Much like Groovy, Scala was easy to use to rewrite BaconSpam.


class BaconSpam extends Spam {
 override def serves(): String =
   return "bacon " + super.serves()
}

And just like Groovy there was no need to change Waitress.java in order to interact with the class. But unlike Groovy the Eclipse plug-in for Scala actually allowed me to execute the application. Plus the docs are better so I didn't have to go digging around to figure out what I needed to do.

Importlib is now useful to other people

2009-03-08T23:20:00.003-07:00

I have always had two goals for importlib. The docs for importlib say they were to provide a reference implementation of import and to make it easier for people to implement their own importers. The former goal is sort of true; the real goal is to not just be a reference implementation of import but to actually become THE implementation of import. The latter goal, though, is spot-on and I finally checked in a big chunk of code tonight that gets me closer to helping other people really harness the flexibility of import.

The cron job to rebuild the docs has not kicked in yet (I think it runs twice a day), but once it does you will discover there is now a new module: importlib.abc. Within the module there are ABCs for everything specified in PEP 302. Back in January I asked for help to name those classes. The fruition of that discussion is now finally live.

But simple ABCs to require load_module exist as a method isn't THAT helpful. What I really wanted to provide was something to make it as easy as possible to write their own custom loaders such that they didn't have to worry about the little details that are consistent between all loaders (and there a lot of details; just look at the "see also" section of the importlib docs). Way back in August 2007 I came up with an idea called handlers which would deal with stuff like setting __file__, making sure bytecode is recreated when it's older than the source code, etc.

Unfortunately handlers turned out to be somewhat burdensome. They required a bunch of information upfront to be passed into them that loaders had to provide. On their own they couldn't do much and just became internal delegates for loaders. So I set about trying to come up with a way to merge the handler concept into loaders.

And it turns out PEP 302 got me part way there. If you look at what it takes to import Python source code (I am ignoring bytecode), you essentially create an empty module, read in the source code, compile the source into a code object, and then execute the code object in the __dict__ of the empty module. After that it's just details like __file__ and such. When you take these basic steps you will notice they roughly align with some APIs that PEP 302 defined as optional protocols loaders could implement. So I asked myself if I could somehow harness the optional protocols to get all the information I need so that I can simply provide a loader that uses those protocols.

Looking at source only, it would seem like the answer is "yes" since there is a get_source method which obviously returns source code for a module. One would think then an implementation for load_module would simply call get_source, compile it, and then use it to create a module. But of course life is not simple.

First of all, as I have discussed before, reading source from disk was not working for me as I didn't have a simple way to get source from disk in the proper decoding thanks to PEP 263. Everything I came up with was on the complicated side. That meant get_source was not exactly a nice thing to rely upon.

But the other issue of relying on get_source is it doesn't tell me the path the source came from. That's needed for __file__ to be set. That completely kills solely relying on get_source.

You could potentially rely on another part of PEP 302 which defines a get_code method which is supposed to return the code object for a module. But that puts more burden on a developer than I wanted to.

At this point I realized I was probably going to have to add to the APIs that PEP 302 provided. I didn't want to do this as that just makes it that much more difficult to implement a loader, but I realized that the PEP 302 protocols simply did not provide all the information needed to create a module from scratch. So I started to think about what the minimum amount I needed to add to the API.

And I thought. And I thought. And I thought. Whenever I have blogged about APIs while working on importlib, it has been in regards to this conundrum of building off of PEP 302's protocols with something that is simple and useful no matter what the storage back-end for modules happened to be.

Eventually I had an epiphany. Using get_source was not an option because it was missing the path to set __file__ to. Somehow the loader needed to have some concept of paths to set __file__ in some meaningful fashion, even if it wasn't really a file path. If the loader provided some concept of a path, then I could use the loader as if it was using a file-based back-end. If I went with that assumption I could use get_data from PEP 302 in order to get at the source code; get_data(source_path('module')).

But I hesistated for a long time at using get_data to fetch source code. Having get_source sitting right there was just so tempting! But then I started to consider how to handle reading bytecode. Should I have a get_bytecode method? But I then run into the problem of needing a bytecode_path method to be able to set __file__ probably for modules loaded from bytecode (and when no source is available; new semantics of 3.0). Going that route means I would have added source_path, bytecode_path, and get_bytecode just to read source and bytecode. This still doesn't deal with getting the modification time for source to see if the bytecode is stale or writing bytecode to the storage back-end through the loader.

Realizing that going the get_bytecode route duplicated functionality needlessly, I went with using get_data to read source and bytecode based on what source_path and bytecode_path return. This keeps the functionality per method simple and mostly unique. It's definitely a "misuse" of get_data as it was not meant to be used this way, but it makes sense and keeps the API simple.

With all of this put together I can provide an ABC that implements load_module in terms of the PEP 302 protocols and just a couple of other methods that handles all the stuff that is not specific to the back-end being used to store the source or bytecode. For instance, to implement a source loader, one needs to implement:

get_data
source_path
is_package

With those three methods, you get a bunch of other methods for free:

get_code
get_source (eventually; actually figured this out just before starting this blog post)
load_module

As you can see, the methods one needs to implement are rather simple to do for a storage back-end. It does follow a path-like API which isn't really needed for non-file back-ends, e.g. databases, which is unfortunate. But since most people blindly assume __file__ and items in __path__ are paths anyway, I don't think you can get around this without breaking people's code.

But the big one is when handling source and bytecode together. There you add the above methods plus:

bytecode_path
source_mtime
write_bytecode

Since write_bytecode can be a no-op, you really only need to implement the first two methods to be able to use bytecode. Heck you could implement all three as dud methods and you would end up with a source-only loader that just happened to always try for bytecode. The point is that I have implemented the other methods so that no one else should have to care about what the format for bytecode files are or when to use bytecode or source.

With all of this done, that leaves just two parts left to implement for the public API (get_source in terms of get_data and source_path along with a decorator I found useful). Once that is done, importlib is semantically done for CPython. I do need to talk to Jython, IronPython, and PyPy to see what they might be missing that I rely on from CPython such that if someone implements a source/bytecode loader it will still work on those VMs even if bytetcode happens to be present (this worry is thanks to PEP 370).

My first academic research paper is now available online

2009-03-08T14:05:00.002-07:00

I just got back from AOSD.09 in Charlottesville, VA. Now that the conference is over my paper that I presented at the conference is now online at the ACM Portal. I have also put up a PDF here for those that are not ACM members.

You can read the abstract at the ACM page to see what it is about. But just a heads-up that the paper has nothing to do with Python; it's all Java/AspectJ in terms of coding stuff and assumes you know both.

Last day for PyCon early bird registration!

2009-02-21T11:11:00.002-08:00

PyCon 2009 has its last day for early bird registration today (Feb 21)!

Want some help porting to Py3K? Come to PyCon!

2009-02-19T17:29:00.002-08:00

Ted Leung made a good point that the sprints at PyCon this year are an ideal time to get help with porting your code to Python 3.0. With a good number of core developers showing up this year there will be plenty of experienced people with 3.0 to answer questions in whatever room the core sprint will be in (once I get around to registering the core sprint).

Three days left for PyCon early bird registration!

2009-02-19T11:44:00.002-08:00

Just a reminder that PyCon early bird registration ends in three days! There are 439 people publicly registered at the moment which, last time I heard, puts us on track to be over 1,000 attendees again! Looks like the economy is not going to suppress attendance at PyCon which is great!

Actually, this PyCon will probably have the highest number of core developers in attendance than any other previous PyCon thanks to the language summit. That should make the conference interesting to me just from the standpoint of the hallways conversations that will most likely spring up.

Benchmark numbers for DVCSs on Python's repository

2009-02-10T10:52:00.001-08:00

Robert Fendt benchmarked Python's repository on various DVCSs. It confirms my numbers that Bazaar, Mercurial, and Git are all fast enough for our needs.

Invited speakers at PyCon announced

2009-02-05T20:45:00.002-08:00

When the list of accepted talks for PyCon 2009 was released, it wasn't the whole story in terms of scheduled content. This year the program committee decided to do something different and have invited talks. Now, short of the keynotes which have not been announced, people have a very clear picture of the great content that is set to be shown at PyCon this year.

[Full disclosure: I am on the PyCon program committee as well as an invited speaker, although I didn't know I was invited until I got the invitation =) ]

importlib.import_module() packaged up for PyPI

2009-02-02T22:01:00.004-08:00

Importlib is now on the Cheeseshop. I took what was in the trunk for Python 2.7 and tossed it up. At the moment I know it can go as far back as Python 2.5, but I bet it will go back even farther without issue; just need to get my hands on older interpreters to see how compatible it is.

With import_module() one does not need to call __import__ and try to comprehend how to use it. Instead you use import_module() and get a much saner API.

Snakebite announced

2009-01-27T12:27:00.003-08:00

Well, I am definitely not the first person to blog about Snakebite, but it is such a big thing I figured I would do my part to get the word out. Having a huge server farm available to the Python developers to fix platform-specific bugs is going to be great.

A perk of being the chairman of the infrastructure committee for the PSF is I got to know about this when Trent first started this. But even knowing about it didn't prepare me for the scope of what he has done with Titus' (and others) help.

And so I big "thank you" from me to Trent for doing this! The man shelled out a lot of time and cash to do this for the benefit for the Python community and I think he deserves all the kudos he has been getting for this.

importlib is now in Python 3.1

2009-01-17T16:29:00.004-08:00

[update: fixed a typo and a broken link]

Back in the summer of 2006 I interned at Google under Neal Norwitz. Part of what I did that summer was try to figure out how to potentially secure the Python interpreter for embedding into Firefox. I did finally figure out how to secure the interpreter for embedding to protect resources, culminating in a paper for a security course I took at UBC.

Part of the solution I developed required controlling import such that you couldn't import arbitrary built-in, frozen, and extension modules. As import is currently implemented, that is simply not possible to do in a secure fashion. That meant reworking or rewriting import. Since the import code was known to be a little difficult to work with I decided to rewrite it in pure Python.

Work began on October 4, 2006. At the time I was planning on making my Python security work my thesis topic with the long-term goal of making my rewrite of import the official implementation of import. Little did I know how massive of a project this would turn out to be.

Two years, three months, and 13 days later, importlib came into being for Python 3.1 in revision 68698. Between the beginning and now my security work stopped being my thesis topic (as did anything directly relating to Python), I dropped support for Python 2.x, and I learned part of the reason the C implementation is so difficult to work with is that import's semantics are rather nuanced and require juggling a lot in your head at once. Oh, and allowing different source encodings is evil.

This is easily the longest amount of time I have ever spent on a single piece of code. One of the surprising things is that the thing is not even 2,000 LOC, including tests! It just took forever to get the semantics fully backwards-compatible (short of some assumptions in the tests, you can currently run the entire test suite for Python with importlib as __import__ and have things work).

The other thing that held up checking importlib in was being too much of a perfectionist. I think I implemented importlib twice, and I still have plans on how to clean things up. Importlib has become the perfect example that your initial implementation might work, but it most definitely will not be the best implementation you can do. Heck, I still have some things to change to make the code easier to work with and more useful to users.

The perfectionist part also came out through worrying about the public API. I know people want to have access to all of the code I have written for their own importers. So I have constantly worried about how to expose it in a sane way. But API design is hard, especially when it is in Python's standard library. Get something wrong and you have to live with it for at least one extra release when you add a deprecation. This is why I am going to expose the API slowly over time and probably blog about it so that I can get feedback from people.

Now that the code is in, what are the long term plans? Well, I have notes with the code that cover what I plan to do. They start with documenting importlib.import_module. That is to be the function that everyone has asked for: a usable interface over __import__. As it stands now the interface is ``import_module(name, package)`` where 'name' is what to import, including relative imports, and 'package' is the package for the calling module. Calling the function returns the specified module, not the top module like __import__; no more fake values in fromlist! I might change the argument names, but I can't think of any other way to make it the API simpler and straight-forward.

Past that is cleaning up some things through better refactorings, exposing more code, and then exposing more of the code. But the end goal is still to get this all bootstrapped into Python 3.1 so that importlib becomes the actual implementation of __import__.