Doug Hellmann

Book Review: IronPython in Action

2009-07-06T08:05:00.001-04:00

IronPython in Action by Michael Foord and Christian Muirhead covers the version of Python built to run on Microsoft's CLR and explains how to use it with the .NET framework.

Disclaimer: I received a review copy of this book from Manning through the PyATL Book Club.

There are two target audiences for this book: experienced Python developers wanting to learn .NET, and experienced .NET developers wanting to learn Python. Both groups will find plenty of interesting material and learn a lot. After some relatively basic introductory chapters, the authors dig right in building a complex GUI application, and then implementing a web interface for the same desktop application.

Along the way they introduce topics such as different programming models in Python, navigating the MSDN documentation (very important for understanding the scope of features available in .NET), packaging your app for distribution under Windows, data persistence, XML parsing, design patterns, automated testing, system administration, relational databases, and two separate GUI libraries.

All of the code is clear, concise, and useful -- there are no fluffy, throw-away code snippets that fall short in the real world. While many of the examples given are specific to IronPython or .NET, the techniques being illustrated are definitely not.

I recommend this book for any Windows developer interested in learning about Python, and for Python developers looking into deploying an application under Windows. If you don't fall into either of those groups, I can still recommend that you pick up a copy for some excellent advice on general programming topics and the solid example code.

New command line interface to PyMOTW

2009-07-05T09:30:00.001-04:00

The 1.95 release of PyMOTW includes a command line interface to access the documentation for a module.

The package can be installed via easy_install or pip:

$ pip install PyMOTW
Downloading/unpacking PyMOTW
  Downloading PyMOTW-1.95.tar.gz (2.2Mb): 2.2Mb downloaded
  Running setup.py egg_info for package PyMOTW
    warning: no files found matching 'ChangeLog'
    warning: no files found matching '*.py' under directory 'sphinx/templates'
    no previously-included directories found matching 'utils'
Installing collected packages: PyMOTW
  Running setup.py install for PyMOTW
    changing mode of build/scripts-2.6/motw from 644 to 755
    warning: no files found matching 'ChangeLog'
    warning: no files found matching '*.py' under directory 'sphinx/templates'
    no previously-included directories found matching 'utils'
    changing mode of /Users/dhellmann/.virtualenvs/testpymotw/bin/motw to 755
Successfully installed PyMOTW

and then to use the command line interface, run motw.

$ motw -h
Usage: motw [options] 

Options:
  -h, --help  show this help message and exit
  -t, --text  Print plain-text version of help to stdout
  -w, --web   Open HTML version of help from web
  --html      Open HTML version of help from installed file

For example, motw abc opens the local version of this week's article. You can also use the "-w" option to go to my web site instead of reading the local version, so you always have the latest version of an article.

PyMOTW: abc - Abstract Base Classes

2009-07-05T09:18:00.001-04:00

abc – Abstract Base Classes

Purpose:	Define and use abstract base classes for API checks in your code.
Python Version:	2.6

Why use Abstract Base Classes?

Abstract base classes are a form of interface checking more strict than individual hasattr() checks for particular methods. By defining an abstract base class, you can define a common API for a set of subclasses. This capability is especially useful in situations where a third-party is going to provide implementations, such as with plugins to an application, but can also aid you when working on a large team or with a large code-base where keeping all classes in your head at the same time is difficult or not possible.

How ABCs Work

abc works by marking methods of the base class as abstract, and then registering concrete classes as implementations of the abstract base. If your code requires a particular API, you can use issubclass() or isinstance() to check an object against the abstract class.

Let’s start by defining an abstract base class to represent the API of a set of plugins for saving and loading data.

import abc

class PluginBase(object):
    __metaclass__ = abc.ABCMeta
    
    @abc.abstractmethod
    def load(self, input):
        """Retrieve data from the input source and return an object."""
        return
    
    @abc.abstractmethod
    def save(self, output, data):
        """Save the data object to the output."""
        return

Registering a Concrete Class

There are two ways to indicate that a concrete class implements an abstract: register the class with the abc or subclass directly from the abc.

import abc
from abc_base import PluginBase

class RegisteredImplementation(object):
    
    def load(self, input):
        return input.read()
    
    def save(self, output, data):
        return output.write(data)

PluginBase.register(RegisteredImplementation)

if __name__ == '__main__':
    print 'Subclass:', issubclass(RegisteredImplementation, PluginBase)
    print 'Instance:', isinstance(RegisteredImplementation(), PluginBase)

In this example the PluginImplementation is not derived from PluginBase, but is registered as implementing the PluginBase API.

$ python abc_register.py
Subclass: True
Instance: True

Implementation Through Subclassing

By subclassing directly from the base, we can avoid the need to register the class explicitly.

import abc
from abc_base import PluginBase

class SubclassImplementation(PluginBase):
    
    def load(self, input):
        return input.read()
    
    def save(self, output, data):
        return output.write(data)

if __name__ == '__main__':
    print 'Subclass:', issubclass(SubclassImplementation, PluginBase)
    print 'Instance:', isinstance(SubclassImplementation(), PluginBase)

In this case the normal Python class management is used to recognize PluginImplementation as implementing the abstract PluginBase.

$ python abc_subclass.py
Subclass: True
Instance: True

A side-effect of using direct subclassing is it is possible to find all of the implementations of your plugin by asking the base class for the list of known classes derived from it (this is not an abc feature, all classes can do this).

import abc
from abc_base import PluginBase
import abc_subclass
import abc_register

for sc in PluginBase.__subclasses__():
    print sc.__name__

Notice that even though abc_register is imported, RegisteredImplementation is not among the list of subclasses because it is not actually derived from the base.

$ python abc_find_subclasses.py
SubclassImplementation

Dr. Andr√© Roberge has described using this capability to discover plugins by importing all of the modules in a directory dynamically and then looking at the subclass list to find the implementation classes.

Incomplete Implementations

Another benefit of subclassing directly from your abstract base class is that the subclass cannot be instantiated unless it fully implements the abstract portion of the API. This can keep half-baked implementations from triggering unexpected errors at runtime.

import abc
from abc_base import PluginBase

class IncompleteImplementation(PluginBase):
    
    def save(self, output, data):
        return output.write(data)

PluginBase.register(IncompleteImplementation)

if __name__ == '__main__':
    print 'Subclass:', issubclass(IncompleteImplementation, PluginBase)
    print 'Instance:', isinstance(IncompleteImplementation(), PluginBase)

$ python abc_incomplete.py
Subclass: True
Instance:
Traceback (most recent call last):
  File "abc_incomplete.py", line 22, in <module>
    print 'Instance:', isinstance(IncompleteImplementation(), PluginBase)
TypeError: Can't instantiate abstract class IncompleteImplementation with abstract methods load

Concrete Methods in ABCs

Although a concrete class must provide an implementation of an abstract methods, the abstract base class can also provide an implementation that can be invoked via super(). This lets you re-use common logic by placing it in the base class, but force subclasses to provide an overriding method with (potentially) custom logic.

import abc
from cStringIO import StringIO

class ABCWithConcreteImplementation(object):
    __metaclass__ = abc.ABCMeta
    
    @abc.abstractmethod
    def retrieve_values(self, input):
        print 'base class reading data'
        return input.read()

class ConcreteOverride(ABCWithConcreteImplementation):
    
    def retrieve_values(self, input):
        base_data = super(ConcreteOverride, self).retrieve_values(input)
        print 'subclass sorting data'
        response = sorted(base_data.splitlines())
        return response

input = StringIO("""line one
line two
line three
""")

reader = ConcreteOverride()
print reader.retrieve_values(input)
print

Since ABCWithConcreteImplementation is an abstract base class, it isn’t possible to instantiate it to use it directly. Subclasses must provide an override for retrieve_values(), and in this case the concrete class massages the data before returning it at all.

$ python abc_concrete_method.py
base class reading data
subclass sorting data
['line one', 'line three', 'line two']

Abstract Properties

If your API specification includes attributes in addition to methods, you can require the attributes in concrete classes by defining them with @abstractproperty.

import abc

class Base(object):
    __metaclass__ = abc.ABCMeta
    
    @abc.abstractproperty
    def value(self):
        return 'Should never get here'


class Implementation(Base):
    
    @property
    def value(self):
        return 'concrete property'


try:
    b = Base()
    print 'Base.value:', b.value
except Exception, err:
    print 'ERROR:', str(err)

i = Implementation()
print 'Implementation.value:', i.value

The Base class in the example cannot be instantiated because it has only an abstract version of the property getter method.

$ python abc_abstractproperty.py
ERROR: Can't instantiate abstract class Base with abstract methods value
Implementation.value: concrete property

You can also define abstract read/write properties.

import abc

class Base(object):
    __metaclass__ = abc.ABCMeta
    
    def value_getter(self):
        return 'Should never see this'
    
    def value_setter(self, newvalue):
        return

    value = abc.abstractproperty(value_getter, value_setter)


class PartialImplementation(Base):
    
    @abc.abstractproperty
    def value(self):
        return 'Read-only'


class Implementation(Base):
    
    _value = 'Default value'
    
    def value_getter(self):
        return self._value

    def value_setter(self, newvalue):
        self._value = newvalue

    value = property(value_getter, value_setter)


try:
    b = Base()
    print 'Base.value:', b.value
except Exception, err:
    print 'ERROR:', str(err)

try:
    p = PartialImplementation()
    print 'PartialImplementation.value:', p.value
except Exception, err:
    print 'ERROR:', str(err)

i = Implementation()
print 'Implementation.value:', i.value

i.value = 'New value'
print 'Changed value:', i.value

Notice that the concrete property must be defined the same way as the abstract property. Trying to override a read/write property in PartialImplementation with one that is read-only does not work.

$ python abc_abstractproperty_rw.py
ERROR: Can't instantiate abstract class Base with abstract methods value
ERROR: Can't instantiate abstract class PartialImplementation with abstract methods value
Implementation.value: Default value
Changed value: New value

Unfortunately, the decorator syntax does not work for read/write abstract properties the way it does with concrete properties.

import abc

class Base(object):
    __metaclass__ = abc.ABCMeta
    
    @abc.abstractproperty
    def value(self):
        return 'Should never see this'
    
    @value.setter
    def value_setter(self, newvalue):
        return


class Implementation(Base):
    
    _value = 'Default value'
    
    @property
    def value(self):
        return self._value

    @value.setter
    def value_setter(self, newvalue):
        self._value = newvalue


i = Implementation()
print 'Implementation.value:', i.value

i.value = 'New value'
print 'Changed value:', i.value

Notice that the caller cannot set the property value.

$ python abc_abstractproperty_rw_deco.py
Implementation.value: Default value
Traceback (most recent call last):
  File "abc_abstractproperty_rw_deco.py", line 40, in <module>
    i.value = 'New value'
AttributeError: can't set attribute

Collection Types

The collections module defines several abstract base classes related to container (and containable) types.

General container classes:

Container
Sized

Iterator and Sequence classes:

Iterable
Iterator
Sequence
MutableSequence

Unique values:

Hashable
Set
MutableSet

Mappings:

Mapping
MutableMapping
MappingView
KeysView
ItemsView
ValuesView

Miscelaneous:

Callable

In addition to serving as detailed real-world examples of abstract base classes, Python’s built-in types are automatically registered to these classes when you import collections. This means you can safely use isinstance() to check parameters in your code to ensure that they support the API you need. The base classes can also be used to define your own collection types, since many of them provide concrete implementations of the internals and only need a few methods overridden. Refer to the standard library docs for collections for more details.

See also

abc: The standard library documentation for this module.
PEP 3119: Introducing Abstract Base Classes
collections: The collections module includes abstract base classes for several collection types.
collections: The standard library documentation for collections.
PEP 3141: A Type Hierarchy for Numbers
Wikipedia: Strategy Pattern: Description and examples of the strategy pattern.
Plugins and monkeypatching: PyCon 2009 presentation by Dr. Andr√© Roberge

PyMOTW Home

The canonical version of this article

Book Review: Hello, World!

2009-07-04T18:45:00.001-04:00

Hello, World! Computer Programming for Kids and Other Beginners by Warren and Carter Sande is an introduction to programming in general (and Python specifically) aimed at pre-teens or young teens.

Disclaimer: I received a review copy of this book from Manning through the PyATL Book Club.

Although the book is designed for a young audience, it is not condescending as many kids books tend to be so it remains readable by adults who need a very basic text on how computer programs work. And by "basic" I mean from the ground up. The book covers using an editor to create and modify program files, numbers, strings, variables, branching, and looping. It doesn't stop with basic topics, though. By the mid-point of the book, the authors have built up to the point where introducing PyGame and graphics programming isn't a stretch, and by the end of the book they have covered the GUI, animation, and sound techniques needed to create two simple computer games.

The writing style is clear and friendly without coming off as cutesy. Each chapter is relatively short, with review questions at the end in the style of a text book (the answer guide is available in the appendix). There is a liberal use of sidebars to break up longer sections or highlight related digressions. And the authors also don't shy away from showing "broken" versions of programs as they evolve, which teaches the reader how to understand error messages and debug problems -- an extremely important skill for a programmer.

I recommend checking out Hello, World! if you have a young person in your life who is interested in learning about programming. Writing the book was a father/son project, and reading it together seems like a fun parent/child activity for the summer.

PyMOTW: pyclbr

2009-06-28T09:02:00.001-04:00

pyclbr – Python class browser support

Purpose:	Implements an API suitable for use in a source code editor for making a class browser.
Python Version:	1.4 and later

pyclbr can scan Python source to find classes and stand-alone functions. The information about class, method, and function names and line numbers is gathered using tokenize without importing the code.

The examples below use this source file as input:

"""Example source for pyclbr.
"""

class Base(object):
    """This is the base class.
    """
    
    def method1(self):
        return

class Sub1(Base):
    """This is the first subclass.
    """

class Sub2(Base):
    """This is the second subclass.
    """

class Mixin:
    """A mixin class.
    """
    
    def method2(self):
        return

class MixinUser(Sub2, Mixin):
    """Overrides method1 and method2
    """
    
    def method1(self):
        return
    
    def method2(self):
        return
        
    def method3(self):
        return

def my_function():
    """Stand-alone function.
    """
    return

Scanning for Classes

There are two public functions exposed by pyclbr. readmodule() takes the name of the module as argument returns a mapping of class names to Class objects containing the meta-data about the class source.

import pyclbr
import os
from operator import itemgetter

def show_class(name, class_data):
    print 'Class:', name
    print '\tFile: {0} [{1}]'.format(os.path.basename(class_data.file), class_data.lineno)
    show_super_classes(name, class_data)
    show_methods(name, class_data)
    print
    return

def show_methods(class_name, class_data):
    for name, lineno in sorted(class_data.methods.items(), key=itemgetter(1)):
        print '\tMethod: {0} [{1}]'.format(name, lineno)
    return

def show_super_classes(name, class_data):
    super_class_names = []
    for super_class in class_data.super:
        if super_class == 'object':
            continue
        if isinstance(super_class, basestring):
            super_class_names.append(super_class)
        else:
            super_class_names.append(super_class.name)
    if super_class_names:
        print '\tSuper classes:', super_class_names
    return

example_data = pyclbr.readmodule('pyclbr_example')

for name, class_data in sorted(example_data.items(), key=lambda x:x[1].lineno):
    show_class(name, class_data)

The meta-data for the class includes the file and line number where it is defined, as well as the names of super classes. The methods of the class are saved as a mapping between method name and line number. The output below shows the classes and methods listed in order based on their line number in the source file.

$ python pyclbr_readmodule.py
Class: Base
        File: pyclbr_example.py [10]
        Method: method1 [14]

Class: Sub1
        File: pyclbr_example.py [17]
        Super classes: ['Base']

Class: Sub2
        File: pyclbr_example.py [21]
        Super classes: ['Base']

Class: Mixin
        File: pyclbr_example.py [25]
        Method: method2 [29]

Class: MixinUser
        File: pyclbr_example.py [32]
        Super classes: ['Sub2', 'Mixin']
        Method: method1 [36]
        Method: method2 [39]
        Method: method3 [42]

Scanning for Functions

The other public function in pyclbr is readmodule_ex(). It does everything that readmodule() does, and adds functions to the result set.

import pyclbr
import os
from operator import itemgetter

example_data = pyclbr.readmodule_ex('pyclbr_example')

for name, data in sorted(example_data.items(), key=lambda x:x[1].lineno):
    if isinstance(data, pyclbr.Function):
        print 'Function: {0} [{1}]'.format(name, data.lineno)

Each Function object has properties much like the Class object.

$ python pyclbr_readmodule_ex.py
Function: my_function [45]

See also

pyclbr: The standard library documentation for this module.
inspect: The inspect module can discover more meta-data about classes and functions, but requires importing the code.
tokenize: The tokenize module parses Python source code into tokens.

PyMOTW Home

The canonical version of this article

PyMOTW: robotparser

2009-06-21T10:59:00.001-04:00

robotparser – Internet spider access control

Purpose:	Parse robots.txt file used to control Internet spiders
Python Version:	2.1.3 and later

robotparser implements a parser for the robots.txt file format, including a simple function for checking if a given user agent can access a resource. It is intended for use in well-behaved spiders or other crawler applications that need to either be throttled or otherwise restricted.

Note

The robotparser module has been renamed urllib.robotparser in Python 3.0.
Existing code using robotparser can be updated using 2to3.

robots.txt

The robots.txt file format is a simple text-based access control system for computer programs that automatically access web resources (“spiders”, “crawlers”, etc.). The file is made up of records that specify the user agent identifier for the program followed by a list of URLs (or URL prefixes) the agent may not access.

This is the robots.txt file for http://www.doughellmann.com/:

User-agent: *
Disallow: /admin/
Disallow: /downloads/
Disallow: /media/
Disallow: /static/
Disallow: /codehosting/

It prevents access to some of the expensive parts of my site that would overload the server if a search engine tried to index them. For a more complete set of examples, refer to The Web Robots Page.

Simple Example

Using the data above, a simple crawler can test whether it is allowed to download a page using the RobotFileParser‘s can_fetch() method.

import robotparser
import urlparse

AGENT_NAME = 'PyMOTW'
URL_BASE = 'http://www.doughellmann.com/'
parser = robotparser.RobotFileParser()
parser.set_url(urlparse.urljoin(URL_BASE, 'robots.txt'))
parser.read()

PATHS = [
    '/',
    '/PyMOTW/',
    '/admin/',
    '/downloads/PyMOTW-1.92.tar.gz',
    ]

for path in PATHS:
    print '%6s : %s' % (parser.can_fetch(AGENT_NAME, path), path)
    url = urlparse.urljoin(URL_BASE, path)
    print '%6s : %s' % (parser.can_fetch(AGENT_NAME, url), url)
    print

The URL argument to can_fetch() can be a path relative to the root of the site, or full URL.

$ python robotparser_simple.py
  True : /
  True : http://www.doughellmann.com/

  True : /PyMOTW/
  True : http://www.doughellmann.com/PyMOTW/

 False : /admin/
 False : http://www.doughellmann.com/admin/

 False : /downloads/PyMOTW-1.92.tar.gz
 False : http://www.doughellmann.com/downloads/PyMOTW-1.92.tar.gz

Long-lived Spiders

An application that takes a long time to process the resources it downloads or that is throttled to pause between downloads may want to check for new robots.txt files periodically based on the age of the content it has downloaded already. The age is not managed automatically, but there are convenience methods to make tracking it easier.

import robotparser
import time
import urlparse

AGENT_NAME = 'PyMOTW'
parser = robotparser.RobotFileParser()
# Using the local copy
parser.set_url('robots.txt')
parser.read()
parser.modified()

PATHS = [
    '/',
    '/PyMOTW/',
    '/admin/',
    '/downloads/PyMOTW-1.92.tar.gz',
    ]

for n, path in enumerate(PATHS):
    print
    age = int(time.time() - parser.mtime())
    print 'age:', age,
    if age > 1:
        print 're-reading robots.txt'
        parser.read()
        parser.modified()
    else:
        print
    print '%6s : %s' % (parser.can_fetch(AGENT_NAME, path), path)
    # Simulate a delay in processing
    time.sleep(1)

This extreme example downloads a new robots.txt file if the one it has is more than 1 second old.

$ python robotparser_longlived.py

age: 0
  True : /

age: 1
  True : /PyMOTW/

age: 2 re-reading robots.txt
 False : /admin/

age: 1
 False : /downloads/PyMOTW-1.92.tar.gz

A “nicer” version of the long-lived application might request the modification time for the file before downloading the entire thing. On the other hand, robots.txt files are usually fairly small, so it isn’t that much more expensive to just grab the entire document again.

See also

robotparser: The standard library documentation for this module.
The Web Robots Page: Description of robots.txt format.

PyMOTW Home

The canonical version of this article

Exception Handling Techniques

2009-06-19T09:23:00.001-04:00

Last week when a colleague was reviewing some code I had written, he had a few questions about an unusual exception handling technique I was using. I had been fixing a bug in which the cleanup code from one error was generating another error, and the second exception masked the first. I went through a few contortions before I came up with a solution that allowed the cleanup code to fail but still preserved the original error and traceback for the user.

After our conversation, I decided to write up a quick example to share. I prefaced the description with some basic exception handling information to put it in context, and I posted the resulting article on my site.

PyMOTW: gettext

2009-06-14T11:47:00.001-04:00

gettext – Message Catalogs

Purpose:	Message catalog API for internationalization.
Python Version:	2.1.3 and later

The gettext module provides an all-Python implementation compatible with the GNU gettext library for message translation and catalog management. The tools available with the Python source distribution enable you to extract messages from your source, build a message catalog containing translations, and use that message catalog to print an appropriate message for the user at runtime.

Message catalogs can be used to provide internationalized interfaces for your program, showing messages in a language appropriate to the user. They can also be used for other message customizations, including “skinning” an interface for different wrappers or partners.

Note

Although the standard library documentation says everything you need is included with
Python, I found that pygettext.py refused to extract messages wrapped in the
ungettext call, even when I used what seemed to be the appropriate command line
options. I ended up installing the GNU gettext tools from source and using
xgettext instead. YMMV.

Translation Workflow Overview

The process for setting up and using translations includes five steps:

Mark up literal strings in your code that contain messages to translate.
Start by identifying the messages within your program source that need to be translated,
and marking the literal strings so the extraction program can find them.
Extract the messages.
After you have identified the translatable strings in your program source, use
xgettext to pull the strings out and create a .pot file, or translation
template. The template is a text file with copies of all of the strings you identified
and placeholders for their translations.
Translate the messages.
Give a copy of the .pot file to the translator, changing the extension to .po. The
.po file is an editable source file used as input for the compilation step. The
translator should update the header text in the file and provide translations for all of
the strings.
“Compile” the message catalog from the translation.
When the translator gives you back the completed .po file, compile the text file to
the binary catalog format using msgfmt. The binary format is used by the runtime
catalog lookup code.
Load and activate the appropriate message catalog at runtime.
The final step is to add a few lines to your application to configure and load the message
catalog and install the translation function. There are a couple of ways to do that, with
associated trade-offs, and each is covered below.

Let’s go through those steps in a little more detail, starting with the modifications you need to make to your code.

Creating Message Catalogs from Source Code

gettext works by finding literal strings embedded in your program in a database of translations, and pulling out the appropriate translated string. There are several variations of the functions for accessing the catalog, depending on whether you are working with Unicode strings or not. The usual pattern is to bind the lookup function you want to use to the name _ so that your code is not cluttered with lots of calls to functions with longer names.

The message extraction program, xgettext, looks for messages embedded in calls to the catalog lookup functions. It understands different source languages, and uses an appropriate parser for each. If you use aliases for the lookup functions or need to add extra functions, you can give xgettext the names of additional symbols to consider when extracting messages.

Here’s a simple script with a single message ready to be translated:

import gettext

# Set up message catalog access
t = gettext.translation('gettext_example', 'locale', fallback=True)
_ = t.ugettext

print _('This message is in the script.')

In this case I am using the Unicode version of the lookup function, ugettext(). The text "This message is in the script." is the message to be substituted from the catalog. I’ve enabled fallback mode, so if we run the script without a message catalog, the in-lined message is printed:

$ python gettext_example.py
This message is in the script.

The next step is to extract the message(s) and create the .pot file, using pygettext.py.

$ xgettext -d gettext_example -o gettext_example.pot gettext_example.py

The output file produced looks like:

# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2009-06-14 11:39-0400\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=CHARSET\n"
"Content-Transfer-Encoding: 8bit\n"

#: gettext_example.py:16
msgid "This message is in the script."
msgstr ""

Message catalogs are installed into directories organized by domain and language. The domain is usually a unique value like your application name. In this case, I used gettext_example. The language value is provided by the user’s environment at runtime, through one of the environment variables LANGUAGE, LC_ALL, LC_MESSAGES, or LANG, depending on their configuration and platform. My language is set to en_US so that’s what I’ll be using in all of the examples below.

Now that we have the template, the next step is to create the required directory structure and copy the template in to the right spot. I’m going to use the locale directory inside the PyMOTW source tree as the root of my message catalog directory, but you would typically want to use a directory accessible system-wide. The full path to the catalog input source is $localedir/$language/LC_MESSAGES/$domain.po, and the actual catalog has the filename extension .mo.

For my configuration, I need to copy gettext_example.pot to locale/en_US/LC_MESSAGES/gettext_example.po and edit it to change the values in the header and add my alternate messages. The result looks like:

# Messages from gettext_example.py.
# Copyright (C) 2009 Doug Hellmann
# Doug Hellmann <doug.hellmann@gmail.com>, 2009.
#
msgid ""
msgstr ""
"Project-Id-Version: PyMOTW 1.92\n"
"Report-Msgid-Bugs-To: Doug Hellmann <doug.hellmann@gmail.com>\n"
"POT-Creation-Date: 2009-06-07 10:31+EDT\n"
"PO-Revision-Date: 2009-06-07 10:31+EDT\n"
"Last-Translator: Doug Hellmann <doug.hellmann@gmail.com>\n"
"Language-Team: US English <doug.hellmann@gmail.com>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"


#: gettext_example.py:16
msgid "This message is in the script."
msgstr "This message is in the en_US catalog."

The catalog is built from the .po file using msgformat:

$ cd locale/en_US/LC_MESSAGES/; msgfmt -o gettext_example.mo gettext_example.po

And now when we run the script, the message from the catalog is printed instead of the in-line string:

$ python gettext_example.py
This message is in the en_US catalog.

Finding Message Catalogs at Runtime

As described above, the locale directory containing the message catalogs is organized based on the language with catalogs named for the domain of the program. Different operating systems define their own default value, but gettext does not know all of these defaults. The default locale directory is sys.prefix + '/share/locale', but most of the time it is safer for you to always explicitly give a localedir value than to depend on any default behavior.

The language portion of the path is taken from one of several environment variables that can be used to configure localization features (LANGUAGE, LC_ALL, LC_MESSAGES, and LANG). The first variable found to be set is used. Multiple languages can be selected by separating the values with a colon (:). We can illustrate how that works by creating a second message catalog and running a few experiments.

$ cd locale/en_CA/LC_MESSAGES/; msgfmt -o gettext_example.mo gettext_example.po
$ python gettext_find.py
Catalogs: ['locale/en_US/LC_MESSAGES/gettext_example.mo']
$ LANGUAGE=en_CA python gettext_find.py
Catalogs: ['locale/en_CA/LC_MESSAGES/gettext_example.mo']
$ LANGUAGE=en_CA:en_US python gettext_find.py
Catalogs: ['locale/en_CA/LC_MESSAGES/gettext_example.mo', 'locale/en_US/LC_MESSAGES/gettext_example.mo']
$ LANGUAGE=en_US:en_CA python gettext_find.py
Catalogs: ['locale/en_US/LC_MESSAGES/gettext_example.mo', 'locale/en_CA/LC_MESSAGES/gettext_example.mo']

Although find() shows the complete list of catalogs, only the first one in the sequence is actually loaded for message lookups.

$ python gettext_example.py
This message is in the en_US catalog.
$ LANGUAGE=en_CA python gettext_example.py
This message is in the en_CA catalog.
$ LANGUAGE=en_CA:en_US python gettext_example.py
This message is in the en_CA catalog.
$ LANGUAGE=en_US:en_CA python gettext_example.py
This message is in the en_US catalog.

Plural Values

While simple message substitution will handle most of your translation needs, one of the special cases handled explicitly by gettext is pluralization. Depending on the language, the difference between the singular and plural forms of a message may vary only by the ending of a single word, or the entire sentence structure may be different. There may also be different forms depending on the level of plurality. To make managing plurals easier (and possible), there is a separate set of functions for asking for the plural form of a message.

from gettext import translation
import sys

t = translation('gettext_plural', 'locale', fallback=True)
num = int(sys.argv[1])
msg = t.ungettext('%(num)d means singular.', '%(num)d means plural.', num)

# Still need to add the values to the message ourself.
print msg % {'num':num}

$ xgettext -L Python -d gettext_plural -o gettext_plural.pot gettext_plural.py

Since there are alternate forms to be translated, the replacements are listed in an array. Using an array allows translations for languages with multiple plural forms (Polish, for example, has different forms indicating the relative quantity).

# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2009-06-14 11:39-0400\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=CHARSET\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=INTEGER; plural=EXPRESSION;\n"

#: gettext_plural.py:15
#, python-format
msgid "%(num)d means singular."
msgid_plural "%(num)d means plural."
msgstr[0] ""
msgstr[1] ""

In addition to filling in the translation strings, you will also need to describe the way plurals are formed so the library knows how to index into the array for any given count value. The line "Plural-Forms: nplurals=INTEGER; plural=EXPRESSION;\n" includes two values to replace manually. nplurals is an integer indicating the size of the array (the number of translations used) and plural is a C language expression for converting the incoming quantity to an index in the array when looking up the translation. The literal string n is replaced with the quantity passed to ungettext().

For example, English includes two plural forms. A quantity of 0 is treated as plural (“0 bananas”). The Plural-Forms entry should look like:

Plural-Forms: nplurals=2; plural=n != 1;

The singular translation would then go in position 0, and the plural translation in position 1.

# Messages from gettext_plural.py
# Copyright (C) 2009 Doug Hellmann
# This file is distributed under the same license as the PyMOTW package.
# Doug Hellmann <doug.hellmann@gmail.com>, 2009.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: PyMOTW 1.92\n"
"Report-Msgid-Bugs-To: Doug Hellmann <doug.hellmann@gmail.com>\n"
"POT-Creation-Date: 2009-06-14 09:29-0400\n"
"PO-Revision-Date: 2009-06-14 09:29-0400\n"
"Last-Translator: Doug Hellmann <doug.hellmann@gmail.com>\n"
"Language-Team: en_US <doug.hellmann@gmail.com>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=n != 1;"

#: gettext_plural.py:15
#, python-format
msgid "%(num)d means singular."
msgid_plural "%(num)d means plural."
msgstr[0] "In en_US, %(num)d is singular."
msgstr[1] "In en_US, %(num)d is plural."

If we run the test script a few times after the catalog is compiled, you can see how different values of N are converted to indexes for the translation strings.

$ cd locale/en_US/LC_MESSAGES/; msgfmt -o gettext_plural.mo gettext_plural.po
$ python gettext_plural.py 0
In en_US, 0 is plural.
$ python gettext_plural.py 1
In en_US, 1 is singular.
$ python gettext_plural.py 2
In en_US, 2 is plural.

Application vs. Module Localization

The scope of your translation effort defines how you install and use the gettext functions in your code.

Application Localization

For application-wide translations, it would be acceptable to install a function like ungettext() globally using the __builtins__ namespace because you have control over the top-level of the application’s code.

import gettext
gettext.install('gettext_example', 'locale', unicode=True, names=['ngettext'])

print _('This message is in the script.')

The install() function binds gettext() to the name _() in the __builtins__ namespace. It also adds ngettext() and other functions listed in names. If unicode is true, the Unicode versions of the functions are used instead of the default ASCII versions.

Module Localization

For a library, or individual module, modifying __builtins__ is not a good idea because you don’t know what conflicts you might introduce with an application global value. You can import or re-bind the names of translation functions by hand at the top of your module.

import gettext
t = gettext.translation('gettext_example', 'locale', fallback=True)
_ = t.ugettext
ngettext = t.ungettext

print _('This message is in the script.')

See also

gettext: The standard library documentation for this module.
GNU gettext: The message catalog formats, API, etc. for this module are all based on the original gettext package from GNU. The catalog file formats are compatible, and the command line scripts have similar options (if not identical). The GNU gettext manual has a detailed description of the file formats and describes GNU versions of the tools for working with them.
Internationalizing Python: A paper by Martin von Löwis about techniques for internationalization of Python applications.
Django Internationalization: Another good source of information on using gettext, including real-life examples.

PyMOTW Home

The canonical version of this article

installing GNU gettext for use with Python on OS X

2009-06-14T08:39:00.001-04:00

I've been working on my blog post about Python's gettext module for the past couple of mornings, and ran into a snag. The documentation claims that the Python source distribution includes all the tools you'll need, but when I got to the point where I wanted to write examples of internationalizing plural strings, pygettext.py wasn't working.

It worked great for extracting individual string messages up to that point, but refused to extract messages wrapped in the ungettext() call, even when I used what seemed to be the appropriate command line options. I ended up installing the GNU gettext tools and using xgettext instead. Installation took me longer than I expected, so I'm documenting the process I went through here.

Fink or MacPorts: Not so much.

Since OS X doesn't ship with a version of gettext by default, and there doesn't seem to be one in the Xcode package (Apple has their own internationalization tools), I needed to find a copy elsewhere.

Ages ago I had installed fink as an "easy" way to grab copies of these sorts of utilities. However, it seems that somewhere along the line the version of fink I have stopped working (probably due to upgrading to 10.5, I haven't tried using fink or FinkCommander directly in some time). After wiping and reinstalling, I was pleased to find a slightly old version (0.14.5) of gettext installed as part of the default set of packages. Unfortunately, xgettext wasn't included in the package at all.

Next, I grabbed a copy of MacPorts, a competitor to fink. Although I've been warned off of MacPorts by a few people I trust, others have had no problems with it. Installation was fairly easy, and as with fink it installed everything to its own directory tree (under /opt/local). Once I had the port program installed, the next step was to run:

$ sudo port install gettext

It downloaded several dependencies, patched the source, compiled everything, and installed it. Voila! Well, not so much.

Even though the most current version of gettext (0.17) was installed, and the documentation clearly described the included Python language support, the binary refused to recognize any language other than C.

Scratch that.

Compiling from Source: Partial Success

Since MacPorts had to download the package and compile it anyway, I decided to go ahead and do that on my own. I was a little wary because I wasn't exactly sure what port was doing in its "patching" step, but I thought I would give it a try anyway. I snagged the most recent tarball from the gettext site and ran the usual

$ ./configure
$ make

That gave me a binary for xgettext inside the source directory, and testing it against my Python source yielded the results I wanted. A simple .pot file was extracted with the original message strings and placeholders for singular and plural translations.

Next, I thought I'd get clever and install the results into the virtualenv I use for working on PyMOTW. Re-configuring with my --prefix set to $VIRTUAL_ENV, rebuilding, then running make install copied the binaries and a bunch of associated data files right where I expected them to be. And the binary only recognized the C language.

After a little more fighting, I did manage to get it working by installing with a prefix of $VIRTUAL_ENV/gettext and adding $VIRTUAL_ENV/gettext/bin to my PATH. I'm not sure if the problem was solved by clearing out an older, bad version of xgettext from elsewhere in my path or if using $VIRTUAL_ENV as the prefix somehow confused the install script.

Conclusion: I think I understand why internationalization is frequently the last feature dealt with in a project.

Book Review: The Economics of Iterative Development

2009-06-11T13:00:00.001-04:00

The Economics of Iterative Software Development, by Walker Royce, Kurt Bittner, and Mike Perrow, covers techniques for achieving more predictable results with development projects.

Disclaimer: I received a review copy of this book through the PyATL Book Club.

The goal of the book is to encourage adoption of iterative development processes, rather than the old-fashioned waterfall model (does anyone really still use that model?). In the authors' view, iterative processes, where one builds a rough version of a product and continually refine it over time, yield better results in terms of predictable schedule and meeting real requirements. This isn't a new idea, and they make reference to agile processes such as XP and Scrum, along with RUP. But the book isn't necessarily a guide to a specific set of processes. It's framed as more of an argument in favor of the entire class of techniques represented by modern development methodologies.

The authors clearly have experience with development processes and evaluating and improving performance of teams. Royce is a VP from IBM's Rational Services group and a contributor to RUP. Bettner is a CTO at Ivar Jacobsen Consulting and also contributed to RUP as well as jazz.net. Perrow is a writer within the Rational group at IBM. Their experience shows in the authoritative tone the book takes when presenting best practices.

"day-to-day decisions are dominated by value judgements, cost trade-offs, ..."

They start from the premise that software development is less of an engineering practice and more like a creative endeavor such as producing a film. Only 1/3 of movies deliver on time and within budget, for example. The comparison resonates with me since I have always viewed development work as more creative than mechanical manufacturing or engineering, although I was never quite sure other types of engineering were as non-creative as they are portrayed.

The more I think about it though, the more I am inclined to see managing software development as managing invention, which is even farther from typical engineering than movie production. With software, no two products or projects are exactly the same, so the "best practices" we learn have to be adapted for every situation.

They go on to describe a generalized history of approaches to software development processes. In the '60's and '70's the attitude was "craftsmanship", with lots of customization of tools and processes. In the '80's and '90's the trend was towards an engineering approach, but it still had a lot of innovation with new technology and techniques. Recent techniques have paid more attention to risk, taking advantage of automation.

Write code. Less of it. Mostly high-level languages.

The authors a couple of primary ways to reduce risk in development. The first is using propose component-based and service-oriented architectures. By isolating parts of the implementation from one another and connecting them through established interfaces, you can iterate over different parts improving as needed. There is also an emphasis on reducing the amount of "human generated" source code, either through high-level languages, off-the-shelf components, or code generation (unsurprisingly, they specifically mention UML-based code generators).

This was about the point in the book where I realized that it was missing the material needed to back up the assertions and claims being made. With a name like "The Economics of Iterative Software Development," I expected to find more statistics and supporting research material than is provided. I don't disagree with any of the authors' conclusions (indeed, they're hardly new insights for anyone who has read a couple of books on agile methodologies). The problem is if I was trying to use this book to convince someone who did not already accept the premise, I wouldn't have any basis for an argument.

This lack of background was particularly evident in chapter 7, where they talk about ways to "accelerate cultural change" to iterative development by choosing a high-profile project instead of easing into it with a pilot project. Their rationale is that the people assigned to work on high-profile projects are typically the better performers already, and if you get their buy-in, they will make the change work because of their dedication. The trick is getting the buy-in in the first place, of course.

I found the book interesting, well written, and worth reading. It doesn't quite stand on its own, though, if you're looking for ammunition to change your boss' mind about process. If you have already decided to go with an iterative process, it will reinforce your decision and provide guidance to make it work (particularly in the appendix). But it didn't live up to my expectations, based on the title.

Updated: Check out my notes for this book on readernaut.com.

Using Readernaut for taking notes on books

2009-06-11T07:45:00.001-04:00

Over the past couple of months I've been working on reviving the PyATL book club. We received our first batch of books from Addison-Wesley/Pearson Education in time to distribute them at our meeting last month. I'll be posting my review soon, but before I do that I wanted to share my experiences with Nathan Borror's excellent http://readernaut.com/.

The site is easy to use. Once you have an account, just use the search field to find a book, either already in the Readernaut catalog or at Amazon. Add it to your shelf, then record your progress and notes as you read. In true web 2.0 fashion, you can connect with other readers and follow their progress (indeed, I may be coming late to the party on this one, because after I signed up I found a bunch of other Pythonistas already using the site to post notes and reviews about the books they're reading).

After a bit of experimentation, I'm hooked. I used the site to take notes about my first book club book, and am finding the results very helpful while preparing the summary for my review. I can record notes, quotes, and remarks as separate types of comments. All include page numbers, so as I build the review I don't even have to go hunting through the book for references.

Besides posting my own material, I've found a couple of interesting-looking titles among the books others are reading. As though I needed help finding more books to read.

If you take your reading seriously, check out http://readernaut.com/ as a convenient way to keep up with your notes.

new project: sphinxcontrib-paverutils

2009-06-05T08:50:00.001-04:00

Kevin Dangoor's Paver includes basic integration for Sphinx, the excellent document production toolkit from Georg Brandl. As I have written before, however, the default integration didn't quite meet my needs for producing different forms of output from the same inputs.

Georg has opened the sphinxcontrib repository on BitBucket for developers who want to collaborate on providing unofficial extensions to Sphinx, so I decided to go ahead and package up the alternate integration I use and release it in case someone else finds it helpful. The result is sphinxcontrib.paverutils.

python-authors mailing list

2009-06-03T07:10:00.001-04:00

At PyCon this year, a group of authors and editors met to start building the writing sub-community for people interested in Python-related topics appearing online and in print. One of the suggestions that came out of the meeting was to establish a new SIG mailing list, hosted on python.org. I'm pleased to announce that the python-authors list has been established and is ready to host traffic!

If you have any experience or interest in writing, editing, or reviewing technical writing such as blog posts, magazine articles, or books, this list is for you. We'll be trading tips, looking for collaborators for new projects, and just generally talking shop.

I hope you'll join us!

Updated 5 June 2009: Unfortunately I've had to set the list subscription to moderated to avoid spammers. The list itself is still unmoderated, and I will try to approve requests for new subscriptions as quickly as possible.

Data persistence tools in the Python Standard Library

2009-05-31T18:46:00.001-04:00

This is the first in a new sub-series of Python Module of the Week posts with a feature-based introduction to modules in the standard library, organized by what your goal might be. Each article may include cross-references to several modules from different parts of the library, and show how they relate to one another. Feedback is welcome, as usual.

Data Persistence and Exchange¶

Python provides several modules for storing data. There are basically two aspects to persistence: converting the in-memory object back and forth into a format for saving it, and working with the storage of the converted data.

Serializing Objects¶

Python includes two modules capable of converting objects into a transmittable or storable format (serializing): pickle and json. It is most common to use pickle, since there is a fast C implementation and it is integrated with some of the other standard library modules that actually store the serialized data, such as shelve. Web-based applications may want to examine json, however, since it integrates better with some of the existing web service storage applications.

Storing Serialized Objects¶

Once the in-memory object is converted to a storable format, the next step is to decide how to store the data. A simple flat-file with serialized objects written one after the other works for data that does not need to be indexed in any way. But Python includes a collection of modules for storing key-value pairs in a simple database using one of the DBM format variants.

The simplest interface to take advantage of the DBM format is provided by shelve. Simply open the shelve file, and access it through a dictionary-like API. Objects saved to the shelve are automatically pickled and saved without any extra work on your part.

One drawback of shelve is that with the default interface you can’t guarantee which DBM format will be used. That won’t matter if your application doesn’t need to share the database files between hosts with different libraries, but if that is needed you can use one of the classes in the module to ensure a specific format is selected (Specific Shelf Types).

If you’re going to be passing a lot of data around via JSON anyway, using json and anydbm can provide another persistence mechanism. Since the DBM database keys and values must be strings, however, the objects won’t be automatically re-created when you access the value in the database.

Relational Databases¶

The excellent sqlite3 in-process relational database is available with most Python distributions. It stores its database in memory or in a local file, and all access is from within the same process, so there is no network lag. The compact nature of sqlite3 makes it especially well suited for embedding in desktop applications or development versions of web apps.

All access to the database is through the Python DBI 2.0 API, by default, as no object relational mapper (ORM) is included. The most popular general purpose ORM is SQLAlchemy, but others such as Django’s native ORM layer also support SQLite. SQLAlchemy is easy to install and set up, but if your objects aren’t very complicated and you are worried about overhead, you may want to use the DBI interface directly.

Data Exchange Through Standard Formats¶

Although not usually considered a true persistence format csv, or comma-separated-value, files can be an effective way to migrate data between applications. Most spreadsheet programs and databases support both export and import using CSV, so dumping data to a CSV file is frequently the simplest way to move data out of your application and into an analysis tool.

Japanese translation of PyMOTW

2009-05-25T11:38:00.001-04:00

I am very pleased to announce that Tetsuya Morimoto is creating a Japanese translation of the Python Module of the Week series.

Tetsuya has used Python for 1.5 years. He has experience at a Linux Distributor using Python with yum, anaconda, and rpm-tools while building RPM packages. Now, he uses it to make useful tools for himself, and is interested in application frameworks such as Django, mercurial and wxPython. Tetsuya is a member of Python Japan User's Group and Python Code Reading.

Thanks, Tetsuya!

PyMOTW: json

2009-05-24T13:57:00.001-04:00

json – JavaScript Object Notation Serializer

Purpose:	Encode Python objects as JSON strings, and decode JSON strings into Python objects.
Python Version:	2.6

The json module provides an API similar to pickle for converting in-memory Python objects to a serialized representation known as “JavaScript Object Notation”. Unlike pickle, JSON has the benefit of having implementations in many languages (especially JavaScript), making it suitable for inter-application communication. JSON is probably most widely used for communicating between the web server and client in an AJAX application, but is not limited to that problem domain.

Encoding and Decoding Simple Data Types

The encoder understands Python’s native types by default (int, float, list, tuple, dict).

import json

data = [ { 'a':'A', 'b':(2, 4), 'c':3.0 } ]
print 'DATA:', repr(data)

data_string = json.dumps(data)
print 'JSON:', data_string

Values are encoded in a manner very similar to Python’s repr() output.

$ python json_simple_types.py
DATA: [{'a': 'A', 'c': 3.0, 'b': (2, 4)}]
JSON: [{"a": "A", "c": 3.0, "b": [2, 4]}]

Encoding, then re-decoding may not give exactly the same type of object.

import json

data = [ { 'a':'A', 'b':(2, 4), 'c':3.0 } ]
data_string = json.dumps(data)
print 'ENCODED:', data_string

decoded = json.loads(data_string)
print 'DECODED:', decoded

print 'ORIGINAL:', type(data[0]['b'])
print 'DECODED :', type(decoded[0]['b'])

For example, tuples are converted to JSON lists.

$ python json_simple_types_decode.py
ENCODED: [{"a": "A", "c": 3.0, "b": [2, 4]}]
DECODED: [{u'a': u'A', u'c': 3.0, u'b': [2, 4]}]
ORIGINAL: <type 'tuple'>
DECODED : <type 'list'>

Human-consumable vs. Compact Output

Another benefit of JSON over pickle is that the results are human-readable. The dumps() function accepts several arguments to make the output even nicer. For example, sort_keys tells the encoder to output the keys of a dictionary in sorted, instead of random, order.

import json

data = [ { 'a':'A', 'b':(2, 4), 'c':3.0 } ]
print 'DATA:', repr(data)

unsorted = json.dumps(data)
print 'JSON:', json.dumps(data)
print 'SORT:', json.dumps(data, sort_keys=True)

first = json.dumps(data, sort_keys=True)
second = json.dumps(data, sort_keys=True)

print 'UNSORTED MATCH:', unsorted == first
print 'SORTED MATCH  :', first == second

Sorting makes it easier to scan the results by eye, and also makes it possible to compare JSON output in tests.

$ python json_sort_keys.py
DATA: [{'a': 'A', 'c': 3.0, 'b': (2, 4)}]
JSON: [{"a": "A", "c": 3.0, "b": [2, 4]}]
SORT: [{"a": "A", "b": [2, 4], "c": 3.0}]
UNSORTED MATCH: False
SORTED MATCH  : True

For highly-nested data structures, you will want to specify a value for indent, so the output is formatted nicely as well.

import json

data = [ { 'a':'A', 'b':(2, 4), 'c':3.0 } ]
print 'DATA:', repr(data)

print 'NORMAL:', json.dumps(data, sort_keys=True)
print 'INDENT:', json.dumps(data, sort_keys=True, indent=2)

When indent is a non-negative integer, the output more closely resembles that of pprint, with leading spaces for each level of the data structure matching the indent level.

$ python json_indent.py
DATA: [{'a': 'A', 'c': 3.0, 'b': (2, 4)}]
NORMAL: [{"a": "A", "b": [2, 4], "c": 3.0}]
INDENT: [
  {
    "a": "A",
    "b": [
      2,
      4
    ],
    "c": 3.0
  }
]

Verbose output like this increases the number of bytes needed to transmit the same amount of data, however, so it isn’t the sort of thing you necessarily want to use in a production environment. In fact, you may want to adjust the settings for separating data in the encoded output to make it even more compact than the default.

import json

data = [ { 'a':'A', 'b':(2, 4), 'c':3.0 } ]
print 'DATA:', repr(data)
print 'repr(data)             :', len(repr(data))
print 'dumps(data)            :', len(json.dumps(data))
print 'dumps(data, indent=2)  :', len(json.dumps(data, indent=2))
print 'dumps(data, separators):', len(json.dumps(data, separators=(',',':')))

The separators argument to dumps() should be a tuple containing the strings to separate items in a list and keys from values in a dictionary. The default is (', ', ': '). By removing the whitespace, we can produce a more compact output.

$ python json_compact_encoding.py
DATA: [{'a': 'A', 'c': 3.0, 'b': (2, 4)}]
repr(data)             : 35
dumps(data)            : 35
dumps(data, indent=2)  : 76
dumps(data, separators): 29

Encoding Dictionaries

The JSON format expects the keys to a dictionary to be strings. If you have other types as keys in your dictionary, trying to encode the object will produce a TypeError. One way to work around that limitation is to skip over non-string keys using the skipkeys argument:

import json

data = [ { 'a':'A', 'b':(2, 4), 'c':3.0, ('d',):'D tuple' } ]

print 'First attempt'
try:
    print json.dumps(data)
except TypeError, err:
    print 'ERROR:', err

print
print 'Second attempt'
print json.dumps(data, skipkeys=True)

Rather than raising an exception, the non-string key is simply ignored.

$ python json_skipkeys.py
First attempt
ERROR: key ('d',) is not a string

Second attempt
[{"a": "A", "c": 3.0, "b": [2, 4]}]

Working with Your Own Types

All of the examples so far have used Pythons built-in types because those are supported by json natively. It isn’t uncommon, of course, to have your own types that you want to be able to encode as well. There are two ways to do that.

First, we’ll need a class to encode:

class MyObj(object):
    def __init__(self, s):
        self.s = s
    def __repr__(self):
        return '<MyObj(%s)>' % self.s

The simple way of encoding a MyObj instance is to define a function to convert an unknown type to a known type. You don’t have to do the encoding yourself, just convert one object to another.

import json
import json_myobj

obj = json_myobj.MyObj('instance value goes here')

print 'First attempt'
try:
    print json.dumps(obj)
except TypeError, err:
    print 'ERROR:', err

def convert_to_builtin_type(obj):
    print 'default(', repr(obj), ')'
    # Convert objects to a dictionary of their representation
    d = { '__class__':obj.__class__.__name__, 
          '__module__':obj.__module__,
          }
    d.update(obj.__dict__)
    return d

print
print 'With default'
print json.dumps(obj, default=convert_to_builtin_type)

In convert_to_builtin_type(), instances of classes not recognized by json are converted to dictionaries with enough information to re-create the object if a program has access to the Python modules necessary.

$ python json_dump_default.py
First attempt
ERROR: <MyObj(instance value goes here)> is not JSON serializable

With default
default( <MyObj(instance value goes here)> )
{"s": "instance value goes here", "__module__": "json_myobj", "__class__": "MyObj"}

To decode the results and create a MyObj instance, we need to tie in to the decoder so we can import the class from the module and create the instance. For that, we use the object_hook argument to loads().

The object_hook is called for each dictionary decoded from the incoming data stream, giving us a chance to convert the dictionary to another type of object. The hook function should return the object it wants the calling application to receive instead of the dictionary.

import json

def dict_to_object(d):
    if '__class__' in d:
        class_name = d.pop('__class__')
        module_name = d.pop('__module__')
        module = __import__(module_name)
        print 'MODULE:', module
        class_ = getattr(module, class_name)
        print 'CLASS:', class_
        args = dict( (key.encode('ascii'), value) for key, value in d.items())
        print 'INSTANCE ARGS:', args
        inst = class_(**args)
    else:
        inst = d
    return inst

encoded_object = '[{"s": "instance value goes here", "__module__": "json_myobj", "__class__": "MyObj"}]'

myobj_instance = json.loads(encoded_object, object_hook=dict_to_object)
print myobj_instance

Since json converts string values to unicode objects, we need to re-encode them as ASCII strings before using them as keyword arguments to the class constructor.

$ python json_load_object_hook.py
MODULE: <module 'json_myobj' from '/Users/dhellmann/Documents/PyMOTW/src/PyMOTW/json/json_myobj.pyc'>
CLASS: <class 'json_myobj.MyObj'>
INSTANCE ARGS: {'s': u'instance value goes here'}
[<MyObj(instance value goes here)>]

Similar hooks are available for the built-in types integers (parse_int), floating point numbers (parse_float), and constants (parse_constant).

Encoder and Decoder Classes

Besides the convenience functions we have already examined, the json module provides classes for encoding and decoding. By using the classes directly, you have access to extra APIs and can create subclasses to customize their behavior.

The JSONEncoder provides an iterable interface for producing “chunks” of encoded data, making it easier for you to write to files or network sockets without having to represent an entire data structure in memory.

import json

encoder = json.JSONEncoder()
data = [ { 'a':'A', 'b':(2, 4), 'c':3.0 } ]

for part in encoder.iterencode(data):
    print 'PART:', part

As you can see, the output is generated in logical units, rather than being based on any size value.

$ python json_encoder_iterable.py
PART: [
PART: {
PART: "a"
PART: :
PART: "A"
PART: ,
PART: "c"
PART: :
PART: 3.0
PART: ,
PART: "b"
PART: :
PART: [
PART: 2
PART: ,
PART: 4
PART: ]
PART: }
PART: ]

The encode() method is basically equivalent to ''.join(encoder.iterencode()), with some extra error checking up front.

To encode arbitrary objects, we can override the default() method with an implementation similar to what we used above in convert_to_builtin_type().

import json
import json_myobj

class MyEncoder(json.JSONEncoder):
    
    def default(self, obj):
        print 'default(', repr(obj), ')'
        # Convert objects to a dictionary of their representation
        d = { '__class__':obj.__class__.__name__, 
              '__module__':obj.__module__,
              }
        d.update(obj.__dict__)
        return d

obj = json_myobj.MyObj('internal data')
print obj
print MyEncoder().encode(obj)

The output is the same as the previous implementation.

$ python json_encoder_default.py
<MyObj(internal data)>
default( <MyObj(internal data)> )
{"s": "internal data", "__module__": "json_myobj", "__class__": "MyObj"}

Decoding text, then converting the dictionary into an object takes a little more work to set up than our previous implementation, but not much.

import json

class MyDecoder(json.JSONDecoder):
    
    def __init__(self):
        json.JSONDecoder.__init__(self, object_hook=self.dict_to_object)

    def dict_to_object(self, d):
        if '__class__' in d:
            class_name = d.pop('__class__')
            module_name = d.pop('__module__')
            module = __import__(module_name)
            print 'MODULE:', module
            class_ = getattr(module, class_name)
            print 'CLASS:', class_
            args = dict( (key.encode('ascii'), value) for key, value in d.items())
            print 'INSTANCE ARGS:', args
            inst = class_(**args)
        else:
            inst = d
        return inst

encoded_object = '[{"s": "instance value goes here", "__module__": "json_myobj", "__class__": "MyObj"}]'

myobj_instance = MyDecoder().decode(encoded_object)
print myobj_instance

And the output is the same as the earlier example.

$ python json_decoder_object_hook.py
MODULE: <module 'json_myobj' from '/Users/dhellmann/Documents/PyMOTW/src/PyMOTW/json/json_myobj.pyc'>
CLASS: <class 'json_myobj.MyObj'>
INSTANCE ARGS: {'s': u'instance value goes here'}
[<MyObj(instance value goes here)>]

Working with Streams and Files

In all of the examples so far, we have assumed that we could (and should) hold the encoded version of the entire data structure in memory at one time. With large data structures it may be preferable to write the encoding directly to a file-like object. The convenience functions load() and dump() accept references to a file-like object to use for reading or writing.

import json
import tempfile

data = [ { 'a':'A', 'b':(2, 4), 'c':3.0 } ]

f = tempfile.NamedTemporaryFile(mode='w+')
json.dump(data, f)
f.flush()

print open(f.name, 'r').read()

A socket would work in much the same way as the normal file handle used here.

$ python json_dump_file.py
[{"a": "A", "c": 3.0, "b": [2, 4]}]

Although it isn’t optimized to read only part of the data at a time, the load() function still offers the benefit of encapsulating the logic of generating objects from stream input.

import json
import tempfile

f = tempfile.NamedTemporaryFile(mode='w+')
f.write('[{"a": "A", "c": 3.0, "b": [2, 4]}]')
f.flush()
f.seek(0)

print json.load(f)

$ python json_load_file.py
[{u'a': u'A', u'c': 3.0, u'b': [2, 4]}]

Mixed Data Streams

The JSONDecoder includes the raw_decode() method for decoding a data structure followed by more data, such as JSON data with trailing text. The return value is the object created by decoding the input data, and an index into that data indicating where decoding left off.

import json

decoder = json.JSONDecoder()
def get_decoded_and_remainder(input_data):
    obj, end = decoder.raw_decode(input_data)
    remaining = input_data[end:]
    return (obj, end, remaining)

encoded_object = '[{"a": "A", "c": 3.0, "b": [2, 4]}]'
extra_text = 'This text is not JSON.'

print 'JSON first:'
obj, end, remaining = get_decoded_and_remainder(' '.join([encoded_object, extra_text]))
print 'Object              :', obj
print 'End of parsed input :', end
print 'Remaining text      :', repr(remaining)

print
print 'JSON embedded:'
try:
    obj, end, remaining = get_decoded_and_remainder(
        ' '.join([extra_text, encoded_object, extra_text])
        )
except ValueError, err:
    print 'ERROR:', err

    

Unfortunately, this only works if the object appears at the beginning of the input.

$ python json_mixed_data.py
JSON first:
Object              : [{u'a': u'A', u'c': 3.0, u'b': [2, 4]}]
End of parsed input : 35
Remaining text      : ' This text is not JSON.'

JSON embedded:
ERROR: No JSON object could be decoded

See also

json: The standard library documentation for this module.
http://json.org/: JavaScript Object Notation (JSON)
http://code.google.com/p/simplejson/: simplejson, from Bob Ippolito, et al, is the externally maintained development version of the json library included with Python 2.6 and Python 3.0. It maintains backwards compatibility with Python 2.4 and Python 2.5.

PyMOTW Home

The canonical version of this article

The Case for Working With Your Hands - NYTimes.com

2009-05-24T11:36:00.001-04:00

The Case for Working With Your Hands - NYTimes.com

Some diagnostic situations contain a lot of variables. Any given symptom may have several possible causes, and further, these causes may interact with one another and therefore be difficult to isolate. In deciding how to proceed, there often comes a point where you have to step back and get a larger gestalt. Have a cigarette and walk around the lift. The gap between theory and practice stretches out in front of you, and this is where it gets interesting. What you need now is the kind of judgment that arises only from experience; hunches rather than rules. For me, at least, there is more real thinking going on in the bike shop than there was in the think tank.

Although I like this article, and understand the source of Mr. Crawford's frustration with corporate culture, I have to take some exception with the categorization of all "knowledge work" as less honest than trade work. The quote above, for example, could just as easily apply to software as motorcycles.

I've spent the last several weeks at work completely refactoring a major aspect of our application because we had finally reached the end of the long blind alley the existing design lead us to follow (it lasted us 7 years, so it was more like a blind turnpike). As I worked my way back out onto the main road and found the right direction to take, I broke and re-fixed several ancillary features, wandered off path myself a few times, and spent a lot of time pondering the variables involved and how they interact.

The resulting design is proving to be more malleable and simpler to test, and as a result of both of those aspects it is vastly easier to maintain. That's a big win for us, since even though I didn't add any customer-visible features to it in this release, by making the code easier to modify I was able to find and fix a major race condition. The redesign also lays the groundwork for future planned enhancements.

That's all by way of saying that working with your hands is honorable, but so can be working with your mind. Just because companies like the abstracting service described in the article do a half-assed job of their work, that doesn't automatically mean all companies do.

Implementing MapReduce with multiprocessing

2009-04-28T08:13:00.001-04:00

Implementing MapReduce with multiprocessing

While looking for example applications for Python's multiiprocessing module to use in this week's PyMOTW, someone suggested implementing MapReduce. Below is the simple implementation I came up with (the source is included in the PyMOTW tarball as of version 1.89).

The Pool class can be used to create a simple single-server MapReduce implementation. Although it does not give the full benefits of distributed processing, it does illustrate how easy it is to break some problems down into distributable units of work.

SimpleMapReduce

In MapReduce, input data is broken down into chunks for processing by different worker instances. Each chunk of input data is mapped to an intermediate state using a simple transformation. The intermediate data is then collected together and partitioned based on a key value so that all of the related values are together. Finally, the partitioned data is reduced to a result set.

import collections
import multiprocessing

class SimpleMapReduce(object):
    
    def __init__(self, map_func, reduce_func, num_workers=None):
        """
        map_func

          Function to map inputs to intermediate data. Takes as argument one input value and
          returns a tuple with the key and a value to be reduced.
        
        reduce_func
          Function to reduce partitioned version of intermediate data to final output. Takes
          as argument a key as produced by map_func and a sequence of the values associated
          with that key.
         
        num_workers
          The number of workers to create in the pool. Defaults to the number of CPUs
          available on the current host.
        """
        self.map_func = map_func
        self.reduce_func = reduce_func
        self.pool = multiprocessing.Pool(num_workers)
    
    def partition(self, mapped_values):
        """Organize the mapped values by their key.
        Returns a dictionary mapping each key to a sequence of values.
        """
        partitioned_data = collections.defaultdict(list)
        for sublist in mapped_values:
            for key, value in sublist:
                partitioned_data[key].append(value)
        return partitioned_data
    
    def __call__(self, inputs, chunksize=1):
        """Process the inputs through the map and reduce functions given.
        
        inputs
          An iterable containing the input data to be processed.
        
        chunksize=1
          The portion of the input data to hand to each worker.  This
          can be used to tune performance during the mapping phase.
        """
        mapped_values = self.pool.map(self.map_func, inputs, chunksize=chunksize)
        partitioned_data = self.partition(mapped_values)
        reduced_values = self.pool.map(self.reduce_func, partitioned_data.items())
        return reduced_values

Counting Words in Files

The following example script uses SimpleMapReduce to counts the “words” in the reStructuredText source for this article, ignoring some of the markup.

import multiprocessing
import string

from multiprocessing_mapreduce import SimpleMapReduce

def file_to_words(filename):
    """Read a file and return a sequence of (word, occurances) values.
    """
    STOP_WORDS = set([
        'a', 'an', 'and', 'are', 'as', 'be', 'for', 'if', 'in', 
        'is', 'it', 'of', 'or', 'py', 'rst', 'the', 'to', 'with',
        ])
    TR = string.maketrans(string.punctuation, ' ' * len(string.punctuation))

    print multiprocessing.current_process().name, 'reading', filename
    output = []

    with open(filename, 'rt') as f:
        for line in f:
            if line.lstrip().startswith('..'): # Skip rst comment lines
                continue
            line = line.translate(TR) # Strip punctuation
            for word in line.split():
                word = word.lower()
                if word.isalpha() and word not in STOP_WORDS:
                    output.append( (word, 1) )
    return output


def count_words(item):
    """Convert the partitioned data for a word to a
    tuple containing the word and the number of occurances.
    """
    word, occurances = item
    return (word, sum(occurances))


if __name__ == '__main__':
    import operator
    import glob

    input_files = glob.glob('*.rst')
    
    mapper = SimpleMapReduce(file_to_words, count_words)
    word_counts = mapper(input_files)
    word_counts.sort(key=operator.itemgetter(1))
    word_counts.reverse()
    
    print '\nTOP 20 WORDS BY FREQUENCY\n'
    top20 = word_counts[:20]
    longest = max(len(word) for word, count in top20)
    for word, count in top20:
        print '%-*s: %5s' % (longest+1, word, count)

Each input filename is converted to a sequence of (word, 1) pairs by file_to_words. The data is partitioned by SimpleMapReduce.partition() using the word as the key, so the partitioned data consists of a key and a sequence of 1 values representing the number of occurrences of the word. The reduction phase converts that to a pair of (word, count) values by calling count_words for each element of the partitioned data set.

$ python multiprocessing_wordcount.py
PoolWorker-1 reading communication.rst
PoolWorker-1 reading index.rst
PoolWorker-1 reading mapreduce.rst
PoolWorker-2 reading basics.rst

TOP 20 WORDS BY FREQUENCY

process         :    74
multiprocessing :    42
worker          :    35
after           :    31
running         :    30
start           :    29
python          :    28
processes       :    27
literal         :    26
header          :    26
pymotw          :    26
end             :    26
daemon          :    23
now             :    22
consumer        :    20
starting        :    18
exiting         :    17
event           :    16
that            :    16
by              :    15

See also

MapReduce - Wikipedia: Overview of MapReduce on Wikipedia.
MapReduce: Simplified Data Processing on Large Clusters: Google Labs presentation and paper on MapReduce.

Special thanks to Jesse Noller for helping review this information.

PyMOTW Home

PyMOTW: multiprocessing, part 2

2009-04-28T08:03:00.001-04:00

Communication between processes with multiprocessing

This is part 2 of coverage of the multiprocessing module. If you missed part one, you may want to start there.

Passing Messages to Processes

As with threads, a common use pattern for multiple processes is to divide a job up among several workers to run in parallel. A simple way to do that with multiprocessing is to use Queues to pass messages back and forth. Any pickle-able object can pass through a multiprocessing Queue.

import multiprocessing

class MyFancyClass(object):
    
    def __init__(self, name):
        self.name = name
    
    def do_something(self):
        proc_name = multiprocessing.current_process().name
        print 'Doing something fancy in %s for %s!' % (proc_name, self.name)


def worker(q):
    obj = q.get()
    obj.do_something()


if __name__ == '__main__':
    queue = multiprocessing.Queue()

    p = multiprocessing.Process(target=worker, args=(queue,))
    p.start()
    
    queue.put(MyFancyClass('Fancy Dan'))
    
    # Wait for the worker to finish
    queue.close()
    queue.join_thread()
    p.join()
    

This short example only passes a single message to a single worker, then the main process waits for the worker to finish.

$ python multiprocessing_queue.py
Doing something fancy in Process-1 for Fancy Dan!

A more complex example shows how to manage several workers consuming data from the queue and passing results back to the parent process. The poison pill technique is used to stop the workers. After setting up the real tasks, the main program adds one “stop” value per worker to the job queue. When a worker encounters the special value, it breaks out of its processing loop.

import multiprocessing
import time

class Consumer(multiprocessing.Process):
    
    def __init__(self, task_queue, result_queue):
        multiprocessing.Process.__init__(self)
        self.task_queue = task_queue
        self.result_queue = result_queue

    def run(self):
        proc_name = self.name
        while True:
            next_task = self.task_queue.get()
            if next_task is None:
                # Poison pill means we should exit
                print '%s: Exiting' % proc_name
                break
            print '%s: %s' % (proc_name, next_task)
            answer = next_task()
            self.result_queue.put(answer)
        return


class Task(object):
    def __init__(self, a, b):
        self.a = a
        self.b = b
    def __call__(self):
        time.sleep(0.1) # pretend to take some time to do our work
        return '%s * %s = %s' % (self.a, self.b, self.a * self.b)
    def __str__(self):
        return '%s * %s' % (self.a, self.b)


if __name__ == '__main__':
    # Establish communication queues
    tasks = multiprocessing.Queue()
    results = multiprocessing.Queue()
    
    # Start consumers
    num_consumers = multiprocessing.cpu_count() * 2
    print 'Creating %d consumers' % num_consumers
    consumers = [ Consumer(tasks, results)
                  for i in xrange(num_consumers) ]
    for w in consumers:
        w.start()
    
    # Enqueue jobs
    num_jobs = 10
    for i in xrange(num_jobs):
        tasks.put(Task(i, i))
    
    # Add a poison pill for each consumer
    for i in xrange(num_consumers):
        tasks.put(None)
    
    # Start printing results
    while num_jobs:
        result = results.get()
        print 'Result:', result
        num_jobs -= 1
        

Although the jobs enter the queue in order, since their execution is parallelized there is no guarantee about the order they will be completed.

$ python multiprocessing_producer_consumer.py
Creating 4 consumers
Consumer-4: 3 * 3
Consumer-4: 7 * 7
Consumer-4: Exiting
Consumer-2: 1 * 1
Consumer-2: 6 * 6
Consumer-2: Exiting
Consumer-3: 0 * 0
Consumer-3: 5 * 5
Consumer-3: 8 * 8
Consumer-3: Exiting
Consumer-1: 2 * 2
Consumer-1: 4 * 4
Consumer-1: 9 * 9
Consumer-1: Exiting
Result: 0 * 0 = 0
Result: 1 * 1 = 1
Result: 2 * 2 = 4
Result: 3 * 3 = 9
Result: 5 * 5 = 25
Result: 4 * 4 = 16
Result: 6 * 6 = 36
Result: 7 * 7 = 49
Result: 8 * 8 = 64
Result: 9 * 9 = 81

Signaling between Processes with Event objects

Events provide a simple way to communicate state information between processes. An event can be toggled between set and unset states. Users of the event object can wait for it to change from unset to set, using an optional timeout value.

import multiprocessing
import time

def wait_for_event(e):
    """Wait for the event to be set before doing anything"""
    print 'wait_for_event: starting'
    e.wait()
    print 'wait_for_event: e.is_set()->', e.is_set()

def wait_for_event_timeout(e, t):
    """Wait t seconds and then timeout"""
    print 'wait_for_event_timeout: starting'
    e.wait(t)
    print 'wait_for_event_timeout: e.is_set()->', e.is_set()


if __name__ == '__main__':
    e = multiprocessing.Event()
    w1 = multiprocessing.Process(name='block', 
                                 target=wait_for_event,
                                 args=(e,))
    w1.start()

    w2 = multiprocessing.Process(name='non-block', 
                                 target=wait_for_event_timeout, 
                                 args=(e, 2))
    w2.start()

    print 'main: waiting before calling Event.set()'
    time.sleep(3)
    e.set()
    print 'main: event is set'

When wait() times out it returns without an error. The caller is responsible for checking the state of the event using is_set().

$ python multiprocessing_event.py
wait_for_event_timeout: starting
wait_for_event_timeout: e.is_set()-> False
wait_for_event: starting
wait_for_event: e.is_set()-> True
main: waiting before calling Event.set()
main: event is set

Controlling access to resources with Lock

In situations when a single resource needs to be shared between multiple processes, a Lock can be used to avoid conflicting accesses.

import multiprocessing
import sys

def worker_with(lock, stream):
    with lock:
        stream.write('Lock acquired via with\n')
        
def worker_no_with(lock, stream):
    lock.acquire()
    try:
        stream.write('Lock acquired directly\n')
    finally:
        lock.release()

lock = multiprocessing.Lock()
w = multiprocessing.Process(target=worker_with, args=(lock, sys.stdout))
nw = multiprocessing.Process(target=worker_no_with, args=(lock, sys.stdout))

w.start()
nw.start()

w.join()
nw.join()

In this example, the messages printed to stdout may be jumbled together if the two processes do not synchronize their access of the output stream with the lock.

$ python multiprocessing_lock.py
Lock acquired via with
Lock acquired directly

Synchronizing threads with a Condition object

Condition objects let you synchronize parts of a workflow so that some run in parallel but others run sequentially, even if they are in separate processes.

import multiprocessing
import time

def stage_1(cond):
    """perform first stage of work, then notify stage_2 to continue"""
    name = multiprocessing.current_process().name
    print 'Starting', name
    with cond:
        print '%s done and ready for stage 2' % name
        cond.notify_all()

def stage_2(cond):
    """wait for the condition telling us stage_1 is done"""
    name = multiprocessing.current_process().name
    print 'Starting', name
    with cond:
        cond.wait()
        print '%s running' % name

if __name__ == '__main__':
    condition = multiprocessing.Condition()
    s1 = multiprocessing.Process(name='s1', target=stage_1, args=(condition,))
    s2_clients = [
        multiprocessing.Process(name='stage_2[%d]' % i, target=stage_2, args=(condition,))
        for i in range(1, 3)
        ]

    for c in s2_clients:
        c.start()
        time.sleep(1)
    s1.start()

    s1.join()
    for c in s2_clients:
        c.join()

In this example, two process run stage two of a job in parallel once the first stage is done.

$ python multiprocessing_condition.py
Starting s1
s1 done and ready for stage 2
Starting stage_2[1]
stage_2[1] running
Starting stage_2[2]
stage_2[2] running

Controlling concurrent access to resources with a Semaphore

Sometimes it is useful to allow more than one worker access to a resource at a time,
while still limiting the overall number. For example, a connection pool might
support a fixed number of simultaneous connections, or a network application
might support a fixed number of concurrent downloads. A Semaphore is one way
to manage those connections.

import random
import multiprocessing
import time

class ActivePool(object):
    def __init__(self):
        super(ActivePool, self).__init__()
        self.mgr = multiprocessing.Manager()
        self.active = self.mgr.list()
        self.lock = multiprocessing.Lock()
    def makeActive(self, name):
        with self.lock:
            self.active.append(name)
    def makeInactive(self, name):
        with self.lock:
            self.active.remove(name)
    def __str__(self):
        with self.lock:
            return str(self.active)

def worker(s, pool):
    name = multiprocessing.current_process().name
    with s:
        pool.makeActive(name)
        print 'Now running: %s' % str(pool)
        time.sleep(random.random())
        pool.makeInactive(name)

if __name__ == '__main__':
    pool = ActivePool()
    s = multiprocessing.Semaphore(3)
    jobs = [
        multiprocessing.Process(target=worker, name=str(i), args=(s, pool))
        for i in range(10)
        ]

    for j in jobs:
        j.start()

    for j in jobs:
        j.join()
        print 'Now running: %s' % str(pool)

In this example, the ActivePool class simply serves as a convenient way to
track which process are running at a given moment. A real resource pool
would probably allocate a connection or some other value to the newly active
process, and reclaim the value when the task is done. Here, it is just used to
hold the names of the active processes to show that only 3 are running
concurrently.

$ python multiprocessing_semaphore.py
Now running: ['3', '2', '0']
Now running: ['3', '2', '0']
Now running: ['0', '1', '5']
Now running: ['2', '0', '1']
Now running: ['0', '1', '4']
Now running: ['3', '2', '0']
Now running: ['0', '7', '6']
Now running: ['0', '4', '7']
Now running: ['7', '6', '8']
Now running: ['7', '8', '9']
Now running: ['7', '6', '8']
Now running: ['7', '6', '8']
Now running: ['7', '6', '8']
Now running: ['7', '6', '8']
Now running: ['7', '6', '8']
Now running: ['7', '6', '8']
Now running: ['7', '8', '9']
Now running: ['8', '9']
Now running: ['9']
Now running: []

Managers

In the previous example, the list of active processes is maintained centrally in the ActivePool instance via a special type of list object created by a Manager. The Manager is responsible for coordinating shared information state between all of its users. By creating the list through the manager, the list is updated in all processes when anyone modifies it. In addition to lists, dictionaries are also supported.

import multiprocessing

def worker(d, key, value):
    d[key] = value

if __name__ == '__main__':
    mgr = multiprocessing.Manager()
    d = mgr.dict()
    jobs = [ multiprocessing.Process(target=worker, args=(d, i, i*2))
             for i in range(10) 
             ]
    for j in jobs:
        j.start()
    for j in jobs:
        j.join()
    print 'Results:', d

$ python multiprocessing_manager_dict.py
Results: {0: 0, 1: 2, 2: 4, 3: 6, 4: 8, 5: 10, 6: 12, 7: 14, 8: 16, 9: 18}

Namespaces

In addition to dictionaries and lists, a Manager can create a shared Namespace. Any named value added to the Namespace is visible across all of the clients.

import multiprocessing

def producer(ns, event):
    ns.value = 'This is the value'
    event.set()

def consumer(ns, event):
    try:
        value = ns.value
    except Exception, err:
        print 'Before event, consumer got:', str(err)
    event.wait()
    print 'After event, consumer got:', ns.value

if __name__ == '__main__':
    mgr = multiprocessing.Manager()
    namespace = mgr.Namespace()
    event = multiprocessing.Event()
    p = multiprocessing.Process(target=producer, args=(namespace, event))
    c = multiprocessing.Process(target=consumer, args=(namespace, event))
    
    c.start()
    p.start()
    
    c.join()
    p.join()

$ python multiprocessing_namespaces.py
Before event, consumer got: 'Namespace' object has no attribute 'value'
After event, consumer got: This is the value

It is important to know that updates to mutable values in the namespace are not propagated.

import multiprocessing

def producer(ns, event):
    ns.my_list.append('This is the value') # DOES NOT UPDATE GLOBAL VALUE!
    event.set()

def consumer(ns, event):
    print 'Before event, consumer got:', ns.my_list
    event.wait()
    print 'After event, consumer got:', ns.my_list

if __name__ == '__main__':
    mgr = multiprocessing.Manager()
    namespace = mgr.Namespace()
    namespace.my_list = []
    
    event = multiprocessing.Event()
    p = multiprocessing.Process(target=producer, args=(namespace, event))
    c = multiprocessing.Process(target=consumer, args=(namespace, event))
    
    c.start()
    p.start()
    
    c.join()
    p.join()

$ python multiprocessing_namespaces_mutable.py
Before event, consumer got: []
After event, consumer got: []

Pool.map

For simple cases where the work to be done can be broken up and distributed between workers, you do not have to manage the queue and worker processes yourself. The Pool class maintains a fixed number of workers and passes them jobs. The return values are collected and returned as a list. The result is functionally equivalent to the built-in map(), except that individual tasks run in parallel.

import multiprocessing

def do_calculation(data):
    return data * 2

if __name__ == '__main__':
    pool_size = multiprocessing.cpu_count() * 2
    pool = multiprocessing.Pool(processes=pool_size)
    
    inputs = list(range(10))
    print 'Input   :', inputs
    
    builtin_outputs = map(do_calculation, inputs)
    print 'Built-in:', builtin_outputs
    
    pool_outputs = pool.map(do_calculation, inputs)
    print 'Pool    :', pool_outputs

$ python multiprocessing_pool.py
Input   : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Built-in: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
Pool    : [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

PyMOTW Home

PyMOTW: multiprocessing, part 1

2009-04-19T12:52:00.001-04:00

`multiprocessing` Basics

Purpose:	Provides an API for managing processes.
Python Version:	2.6

The multiprocessing module includes a relatively simple API for dividing work up between multiple processes. It is based on the API for threading, and in some cases is a drop-in replacement. Due to the similarity, the first few examples here are modified from the threading examples. Features provided by multiprocessing but not available in threading are covered later.

Process objects

The simplest way to use a sub-process is to instantiate it with a target function
and call start() to let it begin working.

import multiprocessing

def worker():
    """worker function"""
    print 'Worker'
    return

if __name__ == '__main__':
    jobs = []
    for i in range(5):
        p = multiprocessing.Process(target=worker)
        jobs.append(p)
        p.start()

The output includes the word “Worker” printed five times, although it may not be entirely clean depending on the order of execution. A later example illustrates using a lock to ensure that only one worker can print to stdout at a time.

$ python multiprocessing_simple.py
Worker
Worker
Worker
Worker
Worker

It usually more useful to be able to spawn a process with arguments to tell it what
work to do. Unlike with threading, to pass arguments to a multiprocessing Process the argument must be able to be pickled. As a simple example we could pass each
worker a number so the output is a little more interesting in the second
example.

import multiprocessing

def worker(num):
    """thread worker function"""
    print 'Worker:', num
    return

if __name__ == '__main__':
    jobs = []
    for i in range(5):
        p = multiprocessing.Process(target=worker, args=(i,))
        jobs.append(p)
        p.start()

The integer argument is now included in the message printed by each worker:

$ python multiprocessing_simpleargs.py
Worker: 0
Worker: 1
Worker: 2
Worker: 3
Worker: 4

Importable Target Functions

One difference you will notice between the threading and multiprocessing examples is the extra protection for __main__ used here. Due to the way the new processes are started, the child process needs to be able to import the script containing the target function. In these examples I accomplish that by wrapping the main part of the application so it is not run recursively in each child. You could also import the target function from a separate script.

For example, this main program:

import multiprocessing
import multiprocessing_import_worker

if __name__ == '__main__':
    jobs = []
    for i in range(5):
        p = multiprocessing.Process(target=multiprocessing_import_worker.worker)
        jobs.append(p)
        p.start()

uses this worker function, defined in a separate module:

def worker():
    """worker function"""
    print 'Worker'
    return

and produces output like the first example above:

$ python multiprocessing_import_main.py
Worker
Worker
Worker
Worker
Worker

Determining the Current Process

Passing arguments to identify or name the process is cumbersome, and unnecessary.
Each Process instance has a name with a default value that you can change as
the process is created. Naming processes is useful if you have a server
with multiple service children handling different operations.

import multiprocessing
import time

def worker():
    name = multiprocessing.current_process().name
    print name, 'Starting'
    time.sleep(2)
    print name, 'Exiting'

def my_service():
    name = multiprocessing.current_process().name
    print name, 'Starting'
    time.sleep(3)
    print name, 'Exiting'

if __name__ == '__main__':
    service = multiprocessing.Process(name='my_service', target=my_service)
    worker_1 = multiprocessing.Process(name='worker 1', target=worker)
    worker_2 = multiprocessing.Process(target=worker) # use default name

    worker_1.start()
    worker_2.start()
    service.start()

The debug output includes the name of the current process on each line. The
lines with “Process-3” in the name column correspond to the unnamed
process w2.

$ python multiprocessing_names.py
worker 1 Starting
worker 1 Exiting
Process-3 Starting
Process-3 Exiting
my_service Starting
my_service Exiting

Daemon Processes

By default the main program will not exit until all of the children have exited. There are
times when you want to start a background process and let it run without blocking the main
program from exiting. Using daemon processes like this is useful for services where there may
not be an easy way to interrupt the worker or where letting it die in the middle of its work
does not lose or corrupt data (for example, a task that generates “heart beats” for a service
monitoring tool). To mark a process as a daemon, set its daemon attribute with a boolean
value. The default is for processes to not be daemons, so passing True turns the daemon mode
on.

import multiprocessing
import time
import sys

def daemon():
    print 'Starting:', multiprocessing.current_process().name
    time.sleep(2)
    print 'Exiting :', multiprocessing.current_process().name

def non_daemon():
    print 'Starting:', multiprocessing.current_process().name
    print 'Exiting :', multiprocessing.current_process().name

if __name__ == '__main__':
    d = multiprocessing.Process(name='daemon', target=daemon)
    d.daemon = True

    n = multiprocessing.Process(name='non-daemon', target=non_daemon)
    n.daemon = False

    d.start()
    time.sleep(1)
    n.start()

Notice that the output does not include the “Exiting” message from the daemon
process, since all of the non-daemon processes (including the main program) exit
before the daemon process wakes up from its 2 second sleep.

$ python multiprocessing_daemon.py
Starting: non-daemon
Exiting : non-daemon

The daemon process is terminated before the main program exits, to avoid leaving orphaned processes running.

Waiting for Processes

To wait until a process has completed its work and exited, use the join() method.

import multiprocessing
import time
import sys

def daemon():
    print 'Starting:', multiprocessing.current_process().name
    time.sleep(2)
    print 'Exiting :', multiprocessing.current_process().name

def non_daemon():
    print 'Starting:', multiprocessing.current_process().name
    print 'Exiting :', multiprocessing.current_process().name

if __name__ == '__main__':
    d = multiprocessing.Process(name='daemon', target=daemon)
    d.daemon = True

    n = multiprocessing.Process(name='non-daemon', target=non_daemon)
    n.daemon = False

    d.start()
    time.sleep(1)
    n.start()

    d.join()
    n.join()

Since we wait for the daemon to exit using join(), we do see its
“Exiting” message.

$ python multiprocessing_daemon_join.py
Starting: non-daemon
Exiting : non-daemon
Starting: daemon
Exiting : daemon

By default, join() blocks indefinitely. It is also possible to pass a timeout
argument (a float representing the number of seconds to wait for the process to
become inactive). If the process does not complete within the timeout period,
join() returns anyway.

import multiprocessing
import time
import sys

def daemon():
    print 'Starting:', multiprocessing.current_process().name
    time.sleep(2)
    print 'Exiting :', multiprocessing.current_process().name

def non_daemon():
    print 'Starting:', multiprocessing.current_process().name
    print 'Exiting :', multiprocessing.current_process().name

if __name__ == '__main__':
    d = multiprocessing.Process(name='daemon', target=daemon)
    d.daemon = True

    n = multiprocessing.Process(name='non-daemon', target=non_daemon)
    n.daemon = False

    d.start()
    n.start()

    d.join(1)
    print 'd.is_alive()', d.is_alive()
    n.join()

Since the timeout passed is less than the amount of time the daemon
sleeps, the process is still “alive” after join() returns.

$ python multiprocessing_daemon_join_timeout.py
Starting: non-daemon
Exiting : non-daemon
d.is_alive() True

Terminating Processes

Although it is better to use the poison pill method of signaling to a process that it should exit, if a process appears hung or deadlocked it can be useful to be able to kill it forcibly. Calling terminate() on a process object kills the child process.

import multiprocessing
import time

def slow_worker():
    print 'Starting worker'
    time.sleep(0.1)
    print 'Finished worker'

if __name__ == '__main__':
    p = multiprocessing.Process(target=slow_worker)
    print 'BEFORE:', p, p.is_alive()
    
    p.start()
    print 'DURING:', p, p.is_alive()
    
    p.terminate()
    print 'TERMINATED:', p, p.is_alive()

    p.join()
    print 'JOINED:', p, p.is_alive()

Note

It is important to join() the process after terminating it in order to give the background machinery time to update the status of the object to reflect the termination.

$ python multiprocessing_terminate.py
BEFORE: <Process(Process-1, initial)> False
DURING: <Process(Process-1, started)> True
TERMINATED: <Process(Process-1, started)> True
JOINED: <Process(Process-1, stopped[SIGTERM])> False

Process Exit Status

The status code produced when the process exits can be accessed via the exitcode attribute.

For exitcode values

== 0 – no error was produced
> 0 – the process had an error, and exited with that code
< 0 – the process was killed with a signal of -1 * exitcode

import multiprocessing
import sys
import time

def exit_error():
    sys.exit(1)

def exit_ok():
    return

def return_value():
    return 1

def raises():
    raise RuntimeError('There was an error!')

def terminated():
    time.sleep(3)

if __name__ == '__main__':
    jobs = []
    for f in [exit_error, exit_ok, return_value, raises, terminated]:
        print 'Starting process for', f.func_name
        j = multiprocessing.Process(target=f, name=f.func_name)
        jobs.append(j)
        j.start()
        
    jobs[-1].terminate()

    for j in jobs:
        j.join()
        print '%s.exitcode = %s' % (j.name, j.exitcode)

Processes that raise an exception automatically get an exitcode of 1.

$ python multiprocessing_exitcode.py
Starting process for exit_error
Starting process for exit_ok
Starting process for return_value
Starting process for raises
Process raises:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/process.py", line 231, in _bootstrap
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "multiprocessing_exitcode.py", line 24, in raises
    raise RuntimeError('There was an error!')
RuntimeError: There was an error!
Starting process for terminated
exit_error.exitcode = 1
exit_ok.exitcode = 0
return_value.exitcode = 0
raises.exitcode = 1
terminated.exitcode = -15

Logging

When debugging concurrency issues, it can be useful to have access to the internals of the objects provided by multiprocessing. There is a convenient module-level function to enable logging called log_to_stderr(). It sets up a logger object using logging and adds a handler so that log messages are sent to the standard error channel.

import multiprocessing
import logging
import sys

def worker():
    print 'Doing some work'
    sys.stdout.flush()

if __name__ == '__main__':
    multiprocessing.log_to_stderr(logging.DEBUG)
    p = multiprocessing.Process(target=worker)
    p.start()
    p.join()
    

By default the logging level is set to NOTSET so no messages are produced. Pass a different level to initialize the logger to the level of detail you want.

$ python multiprocessing_log_to_stderr.py
[INFO/Process-1] child process calling self.run()
Doing some work
[INFO/Process-1] process shutting down
[DEBUG/Process-1] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-1] running the remaining "atexit" finalizers
[INFO/Process-1] process exiting with exitcode 0
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers

To manipulate the logger directly (change its level setting or add handlers), use get_logger().

import multiprocessing
import logging
import sys

def worker():
    print 'Doing some work'
    sys.stdout.flush()

if __name__ == '__main__':
    multiprocessing.log_to_stderr()
    logger = multiprocessing.get_logger()
    logger.setLevel(logging.INFO)
    p = multiprocessing.Process(target=worker)
    p.start()
    p.join()

The logger can also be configured through the logging configuration file API, using the name multiprocessing.

$ python multiprocessing_get_logger.py
[INFO/Process-1] child process calling self.run()
Doing some work
[INFO/Process-1] process shutting down
[INFO/Process-1] process exiting with exitcode 0
[INFO/MainProcess] process shutting down

Subclassing Process

Although the simplest way to start a job in a separate process is to use Process and pass a target function, it is also possible to use a custom subclass. The derived class should override run() to do its work.

import multiprocessing

class Worker(multiprocessing.Process):

    def run(self):
        print 'In %s' % self.name
        return

if __name__ == '__main__':
    jobs = []
    for i in range(5):
        p = Worker()
        jobs.append(p)
        p.start()
    for j in jobs:
        j.join()

$ python multiprocessing_subclass.py
In Worker-1
In Worker-2
In Worker-3
In Worker-4
In Worker-5

See also

multiprocessing: The standard library documentation for this module.
threading: High-level API for working with threads.

PyMOTW Home

Configuring Firefox so its Tab/Window Behavior isn't Annoying

2009-04-06T10:01:00.001-04:00

I toggled a setting in Firefox a couple days ago and its tab/window behavior changed in a way that was exceptionally annoying. It took me a while to to find the now-hidden settings (who removes options only from the UI?) to reset the behavior, so I'm recording it here for next time.

I want it to create a new window when an external app tells it to open a URL, and to use a tab when I click on a link that would open a new window from a page I'm already viewing (such as search results). That prevents apps like Quicksilver from opening tabs in the "last active" window that happens to be on another Spaces desktop, but also doesn't clutter my screen with zillions of new windows as I browse.

FF 3.0.8 for Mac OS X has an option to tell it whether to open new tabs or windows by default:

It used to have separate options for what to do when an external application asked for a URL to be opened vs. when you clicked on a link on a page. I guess someone decided that was confusing. This Cnet article describes how to turn it on:

The options for this behavior are shown at the top of the Tabbed Browsing panel in the Preferences window. If you ever feel the need to set this preference via About:config, look for browser.link.open_external. Setting this preference to 1 opens the link in the current tab and window, 2 opens it in a new window, and 3 opens it in a new tab in the front window.

Along the way I found another semi-useful setting via Lifehacker that opens a new tab when you use the search box.

Type about:config into the address bar, and then put the following into the filter box: browser.search.openintab. Double-click the value to change it to true.

Now every search comes up in its own tab and I don't lose any open pages.

PyMOTW: pipes

2009-04-05T13:19:00.001-04:00

pipes – Unix shell command pipeline templates

Purpose:	Create repeatable Unix shell command pipelines.
Python Version:	Python 1.4

The pipes module implements a class to create arbitrarily complex Unix command pipelines. Inputs and outputs of the commands can be chained together as with the shell | operator, even if the individual commands need to write to or read from files instead of stdin/stdout.

Passing Standard I/O Through a Pipe

A very simple example, passing standard input through a pipe and receiving the results in a file looks like this:

import pipes
import tempfile

# Establish a very simple pipeline using stdio
p = pipes.Template()
p.append('cat -', '--')
p.debug(True)

# Pass some text through the pipeline,
# saving the output to a temporary file.
t = tempfile.NamedTemporaryFile(mode='r')
f = p.open(t.name, 'w')
try:
    f.write('Some text')
finally:
    f.close()

# Rewind and read the text written
# to the temporary file
t.seek(0)
print t.read()
t.close()

The pipeline Template is created and then a single command, cat - is added. The command reads standard input and writes it to standard output, without modification. The second argument to append() encodes the input and output sources for the command in two characters (input, then output). Using - means the command uses standard I/O. Using f means the command needs to read from a file (as may be the case with an image processing pipeline).

The debug() method toggles debugging output on and off. When debugging is enabled, the commands being run are printed and the shell is given set -x so it runs verbosely.

After the pipeline is set up, a NamedTemporaryFile is created to give the pipeline somewhere to write its output. A file must always be specified as argument to open(), whether reading or writing.

$ python pipes_simple_write.py
+ cat -
cat - >/var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpeaa2Zz
Some text

Reading from a pipeline works basically the same way, with a few changes to the arguments. For our example, we need to set up the contents of the input file before opening the pipline. Then we can pass that filename as input to open().

import pipes
import tempfile

# Establish a very simple pipeline using stdio
p = pipes.Template()
p.append('cat -', '--')
p.debug(True)

# Establish an input file
t = tempfile.NamedTemporaryFile(mode='w')
t.write('Some text')
t.flush()

# Pass some text through the pipeline,
# saving the output to a temporary file.
f = p.open(t.name, 'r')
try:
    contents = f.read()
finally:
    f.close()

print contents

We can read the results from the pipeline directly.

$ python pipes_simple_read.py
+ cat -
cat - </var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpiu1dGJ
Some text

Using Files Instead of Streams

Some commands need to work on files on the filesystem instead of input streams. Commands that process a large amount of data might perform better in this mode, since they will not block on the next command reading their output. Anything that works on non-stream-based data requires this capability as well (e.g., databases or other binary file manipulation tools). To support this mode of operation, append() lets you specify a kind of f, and the pipeline code will create the needed temporary files. The filenames are passed to the shell as $IN and $OUT, so those variable names need to appear in your command string.

import pipes
import tempfile

p = pipes.Template()
p.append('cat $IN > $OUT', 'ff')
p.debug(True)

t = tempfile.NamedTemporaryFile('r')
f = p.open(t.name, 'w')
try:
    f.write('Some text')
finally:
    f.close()

t.seek(0)
print t.read()
t.close()

As you see, several intermediate temporary files are created to hold the input and output of the step.

$ python pipes_file_kind.py
+ trap 'rm -f /var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmp13RgCk; exit' 1 2 3 13 14 15
+ cat
+ IN=/var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmp13RgCk
+ OUT=/var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpvHbUr6
+ cat /var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmp13RgCk
+ rm -f /var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmp13RgCk
trap 'rm -f /var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmp13RgCk; exit' 1 2 3 13 14 15
cat >/var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmp13RgCk
IN=/var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmp13RgCk; OUT=/var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpvHbUr6; cat $IN > $OUT
rm -f /var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmp13RgCk
Some text

And of course the input and output kind values can be mixed so that different steps of the pipeline use files or standard I/O as needed.

import pipes
import tempfile

p = pipes.Template()
p.append('cat $IN', 'f-')
p.append('cat - > $OUT', '-f')
p.debug(True)

t = tempfile.NamedTemporaryFile('r')
f = p.open(t.name, 'w')
try:
    f.write('Some text')
finally:
    f.close()

t.seek(0)
print t.read()
t.close()

The trap statements visible in the output here ensure that the temporary files created by the pipeline are cleaned up even a task fails in the middle or the shell is killed.

$ python pipes_mixed_kinds.py
+ trap 'rm -f /var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpsYkNeR; exit' 1 2 3 13 14 15
+ cat
+ IN=/var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpsYkNeR
+ cat /var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpsYkNeR
+ OUT=/var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmp3bffm3
+ cat -
+ rm -f /var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpsYkNeR
trap 'rm -f /var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpsYkNeR; exit' 1 2 3 13 14 15
cat >/var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpsYkNeR
IN=/var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpsYkNeR; cat $IN |
{ OUT=/var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmp3bffm3; cat - > $OUT; }
rm -f /var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpsYkNeR
Some text

A More Complex Example

All of the examples up to this point have been fairly trivial. They were constructed to illustrate how to use pipes.Template() without depending on deep knowledge of shell scripting in general. This example is more complex, and shows how several commands can be combined to manipulate data before bringing it into Python.

My virtualenvwrapper script includes a shell function for listing all of the virtual environments you have created. The function is used for tab-completion and can be used directly to list the environments, in case you forget a name. The heart of that function is a small pipeline that looks in $WORKON_HOME for directories that look like virtual environments (i.e., they have an activate script). That pipeline is:

(cd "$WORKON_HOME"; for f in */bin/activate; do echo $f; done) \
    | sed 's|^\./||' \
    | sed 's|/bin/activate||' \
    | sort

Implemented using pipes, the pipeline looks like:

import pipes
import tempfile

p = pipes.Template()
p.append('cd "$WORKON_HOME"; for f in */bin/activate; do echo $f; done', '--')
p.append(r"sed 's|^\./||'", '--')
p.append("sed 's|/bin/activate||'", '--')
p.append('sort', '--')

t = tempfile.NamedTemporaryFile('r')

f = p.open(t.name, 'r')
try:
    sandboxes = [ l.strip() for l in f.readlines() ]
finally:
    f.close()

print 'SANDBOXES:', sandboxes

Since each sandbox name is written to a separate line, parsing the output is easy:

$ python pipes_multistep.py
SANDBOXES: ['CastSampler', 'blogbackup', 'docket', 'doughellmann', 'nose', 'personal', 'pymag', 'pymotw', 'pymotw25', 'pymotw26']

Passing Files Through Pipelines

If the input to your pipeline already exists in a file on disk, there’s no need to read it into Python simply to pass it to the pipeline. You can use the copy() method to pass the file directly through the pipeline and create an output file for reading.

import pipes
import tempfile

p = pipes.Template()
p.debug(True)
p.append('grep -n tortor $IN', 'f-')

t = tempfile.NamedTemporaryFile('r')

p.copy('lorem.txt', t.name)

t.seek(0)
print t.read()
t.close()

$ python pipes_copy.py
+ IN=lorem.txt
+ grep -n tortor lorem.txt
IN=lorem.txt; grep -n tortor $IN >/var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmpibxIEG
3:elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
6:lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
11:eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet

Cloning Templates

Once you have a pipeline template, you may want to use it multiple times or create variants without re-constructing the entire object. The clone() method makes both of these operations easy. This example constructs a simple word-counter pipeline, then prepends commands to a couple of clones to make it look for different words.

import pipes
import tempfile

count_word_substring = pipes.Template()
#count_word_substring.debug(True)
count_word_substring.append('grep -f - /usr/share/dict/words', '--')
count_word_substring.append('wc -l', '--')

count_py = count_word_substring.clone()
count_py.prepend('echo "py"', '--')
f = count_py.open('/dev/null', 'r')
try:
    print '  "py": %5s' % f.read().strip()
finally:
    f.close()

count_perl = count_word_substring.clone()
count_perl.prepend('echo "perl"', '--')
f = count_perl.open('/dev/null', 'r')
try:
    print '"perl": %5s' % f.read().strip()
finally:
    f.close()

By prepending a custom command to each clone, we can create separate pipelines that perform the same basic function with small variations.

$ python pipes_clone.py
  "py":  1379
"perl":    71

See also

pipes: The standard library documentation for this module.
tempfile: The tempfile module includes classes for managing temporary files.
subprocess: The subprocess module also supports chaining the inputs and outputs of processes together.

PyMOTW Home

virtualenvwrapper updates

2009-03-24T08:18:00.001-04:00

The latest version available for download is 1.12. New features include pre and post operation hooks for operations involving virtualenvs (so you can do things like install your favorite common packages every time you make a new virtualenv), navigation functions for working inside the environment, and a host of fixes and optimizations. Check the release log on PyPI for a complete history of the updates.

In the ~2 weeks since I've uploaded virtualenvwrapper to bitbucket, I've received more patches than in all the previous time I maintained the script. Score 1 for DVCS!

[Updated: Changed "bitkeeper" to "bitbucket" - Sorry Jesper!]

PyMOTW source now available on BitBucket.org

2009-03-14T11:01:00.001-04:00

Earlier this week I uploaded all of the history for PyMOTW to BitBucket.org.

BitBucket uses Mercurial as its DVCS, and after a couple of hiccups with the initial conversion I've had pretty good luck. I think I have the workflow down far managing local changes vs. updates pushed to the central repository.

I still need to work out how (and whether) to use branches. I have a sense that I can use local clones in the way I used to use branches: starting an article that was going to take several sessions, without committing the changes to the trunk. What are the typical practices for hg branching? Do people use them?

Today's PyMOTW post is the first post since the change-over. If you encounter any problems, please report them as an issue via the BitBucket issue tracker.

PyMOTW: asynchat

2009-03-14T10:51:00.001-04:00

asynchat – Asynchronous protocol handler

Purpose:	Asynchronous network communication protocol handler
Python Version:	1.5.2 and later

The asynchat module builds on asyncore to make it easier to implement protocols based on passing messages back and forth between server and client. The async_chat class is an asyncore.dispatcher subclass that receives data and looks for a message terminator. Your subclass only needs to specify what to do when data comes in and how to respond once the terminator is found. Outgoing data is queued for transmission via FIFO objects managed by async_chat.

Message Terminators

Incoming messages are broken up based on terminators, controlled for each instance via set_terminator(). There are three possible configurations:

If a string argument is passed to set_terminator(), the message is considered complete when that string appears in the input data.
If a numeric argument is passed, the message is considered complete when that many bytes have been read.
If None is passed, message termination is not managed by async_chat.

The EchoServer example below uses both a simple string terminator and a message length terminator, depending on the context of the incoming data. The HTTP request handler example in the standard library documentation offers another example of how to change the terminator based on the context to differentiate between HTTP headers and the HTTP POST request body.

Server and Handler

To make it easier to understand how asynchat is different from asyncore, the examples here duplicate the functionality of the EchoServer example from the asyncore discussion. The same basic structure is needed: a server object to accept connections, handler objects to deal with communication with each client, and client objects to initiate the conversation.

The EchoServer needed to work with asynchat is basically the same as the one created for the asyncore-based example, with fewer logging calls because they are less interesting this time around:

import asyncore
import logging
import socket

from asynchat_echo_handler import EchoHandler

class EchoServer(asyncore.dispatcher):
    """Receives connections and establishes handlers for each client.
    """
    
    def __init__(self, address):
        asyncore.dispatcher.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.bind(address)
        self.address = self.socket.getsockname()
        self.listen(1)
        return

    def handle_accept(self):
        # Called when a client connects to our socket
        client_info = self.accept()
        EchoHandler(sock=client_info[0])
        # We only want to deal with one client at a time,
        # so close as soon as we set up the handler.
        # Normally you would not do this and the server
        # would run forever or until it received instructions
        # to stop.
        self.handle_close()
        return
    
    def handle_close(self):
        self.close()

The EchoHandler is based on asynchat.async_chat instead of the asyncore.dispatcher this time around. It operates at a slightly higher level of abstraction, so reading and writing are handled automatically. All we need to do is tell the handler:

what to do with incoming data (by overriding handle_incoming_data())
how to recognize the end of an incoming message (via set_terminator())
what to do when a complete message is received (in found_terminator())
what data to send (using push())

In the example application, we have 2 operating modes. We are either waiting for a command of the form ECHO length\n, or we are waiting for the data to be echoed. We toggle back and forth between the two modes by setting an instance variable process_data to the method to be invoked when the terminator is found and then changing the terminator as appropriate.

import asynchat
import logging


class EchoHandler(asynchat.async_chat):
    """Handles echoing messages from a single client.
    """
    
    def __init__(self, sock):
        self.received_data = []
        self.logger = logging.getLogger('EchoHandler%s' % str(sock.getsockname()))
        asynchat.async_chat.__init__(self, sock)
        # Start looking for the ECHO command
        self.process_data = self._process_command
        self.set_terminator('\n')
        return

    def collect_incoming_data(self, data):
        """Read an incoming message from the client and put it into our outgoing queue."""
        self.logger.debug('collect_incoming_data() -> (%d)\n"""%s"""', len(data), data)
        self.received_data.append(data)

    def found_terminator(self):
        """The end of a command or message has been seen."""
        self.logger.debug('found_terminator()')
        self.process_data()
    
    def _process_command(self):        
        """We have the full ECHO command"""
        command = ''.join(self.received_data)
        self.logger.debug('_process_command() "%s"', command)
        command_verb, command_arg = command.strip().split(' ')
        expected_data_len = int(command_arg)
        self.set_terminator(expected_data_len)
        self.process_data = self._process_message
        self.received_data = []
    
    def _process_message(self):
        """We have read the entire message to be sent back to the client"""
        to_echo = ''.join(self.received_data)
        self.logger.debug('_process_message() echoing\n"""%s"""', to_echo)
        self.push(to_echo)
        # Disconnect after sending the entire response
        # since we only want to do one thing at a time
        self.close_when_done()

Once the complete command is found, we switch to message-processing mode and wait for the complete set of text to be received. When all of the data is available, we push it onto the outgoing channel and set up the handler to be closed once the data is sent.

Client

The client works in much the same way as the handler. As with the asyncore implementation, the message to be sent is an argument to the client’s constructor. When the socket connection is established, handle_connect() is called so we can send the command and message data.

The command is pushed directly, but a special “producer” class is used for the message text. The producer is polled for data to send out over the network. When the producer returns an empty string, it is assumed to be empty and writing stops.

The client expects just the message data in response, so it sets an integer terminator and collects data in a list until the entire message has been received.

import asynchat
import logging
import socket


class EchoClient(asynchat.async_chat):
    """Sends messages to the server and receives responses.
    """
    
    def __init__(self, host, port, message):
        self.message = message
        self.received_data = []
        self.logger = logging.getLogger('EchoClient')
        asynchat.async_chat.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.logger.debug('connecting to %s', (host, port))
        self.connect((host, port))
        return
        
    def handle_connect(self):
        self.logger.debug('handle_connect()')
        # Send the command
        self.push('ECHO %d\n' % len(self.message))
        # Send the data
        self.push_with_producer(EchoProducer(self.message))
        # We expect the data to come back as-is, 
        # so set a length-based terminator
        self.set_terminator(len(self.message))
    
    def collect_incoming_data(self, data):
        """Read an incoming message from the client and put it into our outgoing queue."""
        self.logger.debug('collect_incoming_data() -> (%d)\n"""%s"""', len(data), data)
        self.received_data.append(data)

    def found_terminator(self):
        self.logger.debug('found_terminator()')
        received_message = ''.join(self.received_data)
        if received_message == self.message:
            self.logger.debug('RECEIVED COPY OF MESSAGE')
        else:
            self.logger.debug('ERROR IN TRANSMISSION')
            self.logger.debug('EXPECTED "%s"', self.message)
            self.logger.debug('RECEIVED "%s"', received_message)
        return

class EchoProducer(asynchat.simple_producer):

    logger = logging.getLogger('EchoProducer')

    def more(self):
        response = asynchat.simple_producer.more(self)
        self.logger.debug('more() -> (%s)\n"""%s"""', len(response), response)
        return response

Putting It All Together

The main program for this example sets up the client and server in the same asyncore main loop.

import asyncore
import logging
import socket

from asynchat_echo_server import EchoServer
from asynchat_echo_client import EchoClient

logging.basicConfig(level=logging.DEBUG,
                    format='%(name)s: %(message)s',
                    )

address = ('localhost', 0) # let the kernel give us a port
server = EchoServer(address)
ip, port = server.address # find out what port we were given

message_data = open('lorem.txt', 'r').read() * 2
client = EchoClient(ip, port, message=message_data)

asyncore.loop()

Normally you would have them in separate processes, of course, but this makes it easier to show the combined output.

$ python asynchat_echo_main.py
EchoClient: connecting to ('127.0.0.1', 49919)
EchoClient: handle_connect()
EchoProducer: more() -> (512)
"""Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra f"""
EchoProducer: more() -> (512)
"""ringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iac"""
EchoHandler('127.0.0.1', 49919): collect_incoming_data() -> (9)
"""ECHO 1474"""
EchoHandler('127.0.0.1', 49919): found_terminator()
EchoHandler('127.0.0.1', 49919): _process_command() "ECHO 1474"
EchoHandler('127.0.0.1', 49919): collect_incoming_data() -> (1024)
"""Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra fringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iac"""
EchoProducer: more() -> (450)
"""ulis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra fringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"""
EchoHandler('127.0.0.1', 49919): collect_incoming_data() -> (450)
"""ulis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra fringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"""
EchoHandler('127.0.0.1', 49919): found_terminator()
EchoHandler('127.0.0.1', 49919): _process_message() echoing
"""Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra fringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra fringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"""
EchoProducer: more() -> (0)
""""""
EchoClient: collect_incoming_data() -> (512)
"""Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra f"""
EchoClient: collect_incoming_data() -> (512)
"""ringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iac"""
EchoClient: collect_incoming_data() -> (450)
"""ulis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra fringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"""
EchoClient: found_terminator()
EchoClient: RECEIVED COPY OF MESSAGE

See also

asynchat: The standard library documentation for this module.
asyncore: The asyncore module implements an lower-level asynchronous I/O event loop.

PyMOTW Home

Doug Hellmann

Book Review: IronPython in Action

New command line interface to PyMOTW

PyMOTW: abc - Abstract Base Classes

abc – Abstract Base Classes

Why use Abstract Base Classes?

How ABCs Work

Registering a Concrete Class

Implementation Through Subclassing

Incomplete Implementations

Concrete Methods in ABCs

Abstract Properties

Collection Types

Book Review: Hello, World!

PyMOTW: pyclbr

pyclbr – Python class browser support

Scanning for Classes

Scanning for Functions

PyMOTW: robotparser

robotparser – Internet spider access control

robots.txt

Simple Example

Long-lived Spiders

Exception Handling Techniques

PyMOTW: gettext

gettext – Message Catalogs

Translation Workflow Overview

Creating Message Catalogs from Source Code

Finding Message Catalogs at Runtime

Plural Values

Application vs. Module Localization

Application Localization

Module Localization

installing GNU gettext for use with Python on OS X

Book Review: The Economics of Iterative Development

Using Readernaut for taking notes on books

new project: sphinxcontrib-paverutils

python-authors mailing list

Data persistence tools in the Python Standard Library

Data Persistence and Exchange¶

Serializing Objects¶

Storing Serialized Objects¶

Relational Databases¶

Data Exchange Through Standard Formats¶

Japanese translation of PyMOTW

PyMOTW: json

json – JavaScript Object Notation Serializer

Encoding and Decoding Simple Data Types

Human-consumable vs. Compact Output

Encoding Dictionaries

Working with Your Own Types

Encoder and Decoder Classes

Working with Streams and Files

Mixed Data Streams

The Case for Working With Your Hands - NYTimes.com

Implementing MapReduce with multiprocessing

Implementing MapReduce with multiprocessing

SimpleMapReduce

Counting Words in Files

PyMOTW: multiprocessing, part 2

Communication between processes with multiprocessing

Passing Messages to Processes

Signaling between Processes with Event objects

Controlling access to resources with Lock

Synchronizing threads with a Condition object

Controlling concurrent access to resources with a Semaphore

Managers

Namespaces

Pool.map

PyMOTW: multiprocessing, part 1

multiprocessing Basics

Process objects

Importable Target Functions

Determining the Current Process

Daemon Processes

Waiting for Processes

Terminating Processes

Process Exit Status

Logging

Subclassing Process

`multiprocessing` Basics