akalias

New Wave: PyJamas?

2009-05-30T06:33:00.000-07:00

The Wave demo is very impressive. The thing that stood out the most to me was their usage of Google Web Toolkit. This is based on Java, a language typically ridiculed by the dynamic language crowd. It must have something going for it.

One of the head wave guys mentioned `the amazing tools` surrounding Java (IDEs, automatic refactoring helpers). A lot of this is only possible due to its static nature.

For this numbskull, used to writing little scripts generally only about 500 lines python is OK. Lately I have been starting to break up my scripts into more than one file. It's actually really painful making changes when it effects multiple files. I'm a `sketcher` so I like to change things a lot.

So.. Java has `amazing` refactoring capabilities. The only thing I don't really like about it from what I have seen is the intense verbosity and visual noise. You'd think all that would be a lot of weight and mud underneath your boots.

Perhaps not. With snippets, auto-completion and refactoring helpers available in a good IDE you may even be able to develop faster in Java than Python. You are working at a lower level, but a level that your stupid servant knows well enough to be able to help you. "No you fucking idiot! It's a MetaDuperPolymorpher class! Go get me the fucking arg spec will you!... Ah forget it. I'll do it myself."

Maybe programmers need to go beyond plain-text editors and use markup for type annotations and whatever other metadata the computers need. It *IS* 2009 We could and should have our cake and eat it too; a nice syntax for humans AND all the metadata a compiler could want.

We need rich-*source* editors /languages. I wonder if in 2035 people will still be using plain text wrapped with an 80 column width?

To bridge the gap maybe we could integrate debuggers/shells right into our editors. Why couldn't you edit a `live` program with all the introspective data that would provide? Sure you'd have to be careful about side effects but if you had a nice interface to the `debugger` you could launch into the program at any point.

I have an editor plugin that allows me to push/pull lines from IPython, and pull auto-completion. So my exploratory missions form the basis of my scripts. Sure beats tedious copy/pasting in between. Sometimes I'll push the whole file through and then edit functions in my editor, pushing the changes and then test them out in the shell. Who the hell wants to edit multi-line functions inside IPython? Really I'd like a REPL inside my editor and *smart* access to it's history and type introspection.

With the latest version of Wing-IDE, it actually pulls the source-completion data from the debugged process when you are debugging. Set a breakpoint, and start editing `live` (kind of) I really like this work-flow. The line between repl shell and editor blurs. An *Integrated* Development Environment.

Sprox

2009-05-29T06:25:00.000-07:00

An interesting CRUD library based on sqlalchemy / formencode / toscawidgets. It actually generates the html. Looks quite handy if you wanted fully generated forms.

Sprox

I'm not sure if it can do nested model forms or if it can that the functionality is decoupled from the html generation and available for use. It seems like the author has put a lot of thought into it. I will definitely investigate it and ToscaWidgets further. I haven't really had much experience with auto-generated forms. I get the feeling that they are hard to customize but it's likely worth auto-generating when possible.

There doesn't seem to be m?any projects focused on (semi or fully) automatically creating list/pagination filter forms. This is quite commonly required in admin sections of a site.

I can't even remember if Django does.

sqaencode 0.1

2009-05-29T01:55:00.000-07:00

I have been messing around a bit and now pretty much have a working lib for encoding nested models into forms and back and automatically generating formencode Schema from sqlalchemy models.

E:\python_repos\sqaencode>nosetests --with-coverage --cover-package=sqaencode
................
Name                   Stmts   Exec  Cover   Missing
----------------------------------------------------
sqaencode                  6      6   100%
sqaencode.constants       10     10   100%
sqaencode.decode          47     45    95%   126, 161
sqaencode.encode          28     26    92%   90-91
sqaencode.util            12     12   100%
sqaencode.validators      46     42    91%   30, 64, 66, 129
----------------------------------------------------
TOTAL                    149    141    94%
----------------------------------------------------------------------
Ran 16 tests in 0.962s

OK

I created a ModelSchema class, subclassing formencode.Schema and giving it an inline __metaclass__ inheriting from formencode.declarative.DeclarativeMeta. I over-rode the __repr__ to output an (almost) eval()able representation.

class ModelSchema(Schema):
    class __metaclass__(DeclarativeMeta):
        def __repr__(cls):
            model = cls.__model__.__name__
            base = [ "class %(model)sSchema(model_schema(%(model)s)):" % dict (
                              model = model)  ]

            for arg in SCHEMA_ARGS:
                base.append('    %-20s = %r' % (arg, getattr(cls, arg)))

            base.append('')
            for key, validator in sorted(cls.fields.items()):
                if key.startswith('_'): continue

                args = non_default_validator_args(validator)
                base.append('    %-20s = %s(%s)' % (
                    key, type(validator).__name__, args) )

            return '\n'.join(base)

Using the model_schema factory to create a ModelSchema and then getting a repr:

In [1]: from sqaencode import model_schema

In [2]: model_schema(Product)
Out[2]:
class ProductSchema(model_schema(Product)):
    ignore_key_missing   = True
    allow_extra_fields   = True
    pre_validators       = []

    active               = Bool()
    amount               = UnicodeString()
    colour               = UnicodeString()
    description          = UnicodeString()
    featured             = Bool()
    id                   = Int()
    image                = UnicodeString()
    image_thumb          = UnicodeString()
    image_zoom           = UnicodeString()
    keywords             = UnicodeString()
    material             = UnicodeString()
    name                 = UnicodeString()
    ordernum             = Int()
    sku                  = UnicodeString()
    views                = Int()

The __metaclass__ __repr__ hack serves a dual purpose. a) for debugging and b) as templating. By printing the repr you can use that as a template for customizing a model schema. You actually inherit from a dynamically generated class ( a function call taking a model and optional arguments). I wasn't even too sure you could do that in python. It's nice I don't have to create a new meta class mechanism and can stick with existing formencode semantics.

Note the `ignore_key_missing` flag that is by default set to True. I think when using this I will just define which fields to validate purely by virtue of what is included in the html. eg If there is no `sku` field in the form then it will not be validated.

What if I *didn't* want to globally ignore missing keys and wanted to manually declare which to ignore? Formencode has an in-built sub-classing mechanism whereby if you declare `some_key = None` then some_key will not be validated at all.

To declare a Product model_schema with plural Colors inline:

class ProductSchema(model_schema(Product, nested=True)):
    colors = model_schema(Color, plural=True)

With Python 2.6 you can do inline customisable declarations of relations:

class ProductSchema(model_schema(Product, nested=True)):
    @sqaencode.plural
    class colors(model_schema(Color)):
        some_field = NonDefault()

I'm thinking about creating a mechanism whereby I subclass sqlalchemy.types.* for metadata purposes to further drive automatic schema generation. A cool thing about Django's tight integration is the high level data types. Url, Email etc You declare higher level properties to what are essentially stored as VARCHAR types in the database. It's not *just* a string.

SqlAlchemy, while really great, (rightly) doesn't try and abstract beyond basic SQL. There is nothing stopping an end user however doing something like this:

1  class Url(Unicode):   pass
2  class Email(Unicode): pass
3  
4  higher_level_table = Table ( 'higher_level', metadata,
5      Column('url', Url(32)),
6      Column('email', Email(32)),
7      Column('id', Integer(), primary_key=True, autoincrement=True, nullable=False),
8  )
9

If Url and Email were imported column types from sqaencode.types then you could add them to the sqalchemy => formencode type mapping and they would be picked up by model_schema()

What's on the todo?

Options for automatically generating child Schemas.
Setting the length on String/UnicodeString validators automatically to the max length of the corresponding column.
Automatically create unique column validators

Reading

2009-05-27T22:20:00.000-07:00

A few days ago I scribbled an entry into my TODO.txt:

Learn about Python packaging, namespaces etc, best practices. setup.py etc etc

I started reading a lot and watching whatever screen casts I could. One useful book was Expert Python Programming. It is light on actual python and heavy on the soft skills in using open source python tools to refine your work flow. That is exactly what I was looking for. It is only about 300 pages so I skimmed through most of it in a day.

The Agile Development Tools in Python series by Christopher Perkins, while pretty lighton gave some nice quick overviews of some very useful tools.

Between these two sources I got a good overview of:

setuptools
distutils

virtualenv
paster

nosetest
coverage

docutils
reST
sphinx

virtualenv

virtualenv will set up an isolated version of python, where applications can remain blissfuly ignorant of the rapid change in the outside world. If you have an application that needs a particular version of a lib and another requiring a different one then this is the tool for the job.

It will set up a folder structure with (std) Lib and site-packages folders and a scripts directory. One of the scripts is of course the virtual python interpreter. It also has a tremendously useful `activate` script. This will put the Scripts (bin on *nix) folder on PATH and the lib folders on PYTHONPATH. It also modifies your console prompt to display the name of the current virtualenv.

(myenv) C:\myenv

I haven't used it much as of yet but I can imagine that it could get pretty confusing once you had a tonne of them so that is a nice and thoughtful addition.

The net result being that if you `activate` an environment and then run `python some_python.py` from an arbitrary directory it will run it using the virtualenv. Likewise for any of the scripts in your (*nix: bin, win: Scripts) folder. There is no need for explicitly referencing them. When you are done you just `deactivate` the environment with corresponding script.

setuptools is installed by default into each new virtualenv and easy_install is included in the scripts folder. You use easy_install to, funnily enough, easily install the packages you require.

virtualenv seems of great utility, however I imagine it could get pretty wasteful in terms of network usage. You don't really want to be downloading 10 packages EVERY time you start some new project. It would be handy if all of your local virtualenv environments used some sort of global local cache when using easy_install.

If you needed for example cherrypy and you already had the latest version on your hide drive some where it would just pull the egg from there. Otherwise it would first pull the package into the cache from pypi.

It would be good if this all happened transparently and each virtualenv's easy_install respected some global distutils.cfg setting for cache location. Where do eggs lay about usually? In a Nest. I'll probably happily find that this has been sorted out.

Even if you had a cache, saving yourself some bandwidth, if you just copied an egg into each environment that would still be disk wastage. This seems like a good use for symbolic linking. It would be silly to reinvent that in python. Although, I'm not too sure how well Windows supports something like that. *nix is definitely superior in that respect.

paver

Another interesting util is one called paver that sort of ties together virtualenv and distutils/setuptools to allow a more `pythonic` zc.buildout. Buildout uses declarative ini files for everything and is apparently harder to hack if you want to do anything out of the ordinary. (I wouldn't know for sure if these are valid criticisms )

From what I have read of buildout vs paver/virtualenv I think I will invest in the latter. It just seems to appeal more to my personal sense of aesthetics.

paster

Paster is a hodgepodge macgyver swiss knife created by Ian Bicking. It can launch WSGI applications, launch missiles, sink ships and god knows what else. This utility, `eclectic` in the words of Ian, is basically the kitchen sink.

Of interest to me is the project template creation commands. It will create a folder with setup.py, README.txt etc and run you through a command line based wizard to set it all up, asking version number, author name and the like. These project templates can give a bit of an insight into how other people structure their projects.

Where do the tests go? Inside the distribution proper? Housed inside the same folder as the setup.py? I'm somewhat leaning towards having the tests included in the public package. ie package.tests.fixtures I think it makes it easy then to import fixtures in doctested examples documentation.

distutils / setuptools

With regard to structuring projects, if there is one thing I can hope to take away from all this, then it is the `setup.py develop` command.

This distutils (or is it setuptools?) command will put your under development package on sys.path so you don't have to run `setup.py install` every time you make changes. Quite handy. Of course you could have a virtualenv where you have a package in `develop` mode and another where you have ran a full `setup.py install` at a certain version.

nosetests

Nosetests has a really great coverage plugin. It will run your tests, and you can specify which package/module you want it to dump a list of line ranges that haven't been `covered` by your test code. This makes it super easy to tell which code paths still needs testing( or culling! ). I'm assuming this is only really of use for pure python code. If so, that is another tick next to dynamic `interpreted` languages.

sphinx

I also experimented with docutils/ reST / Sphinx for documentation. Sphinx is the spiffy new reST based documentation system developed originally for Python proper. It has a built-in indexer and javascript search client and seemingly many other great features. For this reason it seems a LOT of python projects are using it now. I imagine the fact that it's quite easy on the eye out of the box doesn't hurt take-up much either.

A new project

I started on creating a setup.py enabled, nose/doc tested, sphinx documented project. It *really* slows down the whole process. I suppose it's a lot to try and learn all at once.

I'm still not really sold on test first prototyping. TDD of something you have a little bit of experience with, sure. TDD/documenting something when you are in new terrain and you KNOW that you are going to mess up and have to redux anyway, just seems wasteful.

I probably just don't `get it` yet. `Slow is fast` has generally held true in a lot of other areas of my experience.

What else to read up on? I need to learn more about the standard library logging package.

Souvenir

2009-05-22T22:39:00.001-07:00

prison color blue
it's a uniform of choice
count yourself lucky
that you don't write the software

Some lines from the great Neil Finn's song `Souvenir`. Count yourself lucky indeed.

FormEncode + SqlAlchemy = SQAEncode

2009-05-22T20:13:00.000-07:00

The way I do admin model CRUD at the moment is have a models.py file containing all my sqlalchemy tables/classes and their respective formencode.Schema/s in a forms.py file. The problem is that I end up having to repeat all the fields and to me that just stinks.

It's not a big deal as far as typing them out goes as I will typically declare a table / model in sqlalchemy and then use a homebrewed scaffolding function to generate derived code for the Schema and html for the actual form. When actually editing a model form I use genshi's HTMLFormFiller filter to set the blank forms values.

This approach really is keeping in the spirit of the formencode library. The author's philosophy is that after a while configuring an `autogenerated` form from a model by declaring options in code is just as much work as declaring in html.

Configuration includes black/white listing certain fields, whether they are allowed to be empty (null/None) and also logical/presentational ordering and grouping of individual fields.

formalchemy on the other hand will take a model and automatically generate a form with the values already set. It can not though do nested models. ie A parent model with an arbitrary amount of inline children to CRUD in the same form. The author considers this `almost always bad design` It only allows what I term `relating` the parent/root model to existing models.

While it *may* sometimes save you from doing any `configuration` whatsoever in the case where the form validation schema reflects perfectly the model, in my (admittedly meagre) experience this would be fairly rare.

The authors of formalchemy recommend the use of CSS to customize the appearance of the auto generated forms. For more extensive customization you can override each fields renderer (<input type = 'text'> vs <textarea> etc) or the global form generation function.

Having not really used formalchemy that much I can't say with certainty just how unweildy it gets trying to customize a forms options / looks. I'd guess that you would end up having to do just as much `configuration`, ie repeating of fields / `work`

Looking at a lot of my formencode Schemas in relation to their underlying model, it seems they are declared in a `white list` manner. The schemas (and html) simply don't declare fields that aren't `public`. The underlying type map is reasonably consistent. A sqlalchemy Bool becomes a formencode Bool, a Unicode maps to a UnicodeString etc. It's typically only the options that are changed.

I have an emybronic idea to auto generate formencode validators from an sqlalchemy model, mapping validators to column/relation types. I imagine `configuring` this would look something like the following.

1  class ProductSchema(ModelSchema): 
2      _model = Product
3      
4      long_description = options(not_empty = True)
5      date_created = None
6      name = UniqueName()

The ModelSchema would use some type of __metaclass__. Any `private` field declared as None would be blacklisted from the underlying derived schema. Above `options` would just be an alias for dict. A dict would be used to configure the auto mapped validator. `not_empty = True` would override the not_empty argument mapped from the respective columns nullable property. UniqueName above would completely override from the UnicodeString. (In fact any column with unique = True could probably have an auto generated `unique field` validator attached)

I'm also currently working on some sqlalchemy formencode integration functions that will take basic one table models and their relations and encode them into nested dictionaries / primary key lists. Also, the other way round, taking a nested dict (as from a formencode.NestedVariables pre validated Schema) and creating/updating/deleting an object tree/graph.

SqlAlchemy mappers makes it quite easy to introspect classes for relations so the object_graph func signature just looks like:

1  object_graph(nd_dict, root_model)

This all works fine in basic unit test land AND for one model forms. The problem I'm having is ONETOMANY child objects in a form. Imagine an Invoice form with an inline InvoiceItems table with each row representing an individual item. On a new invoice there would be 3 `blank` rows. How would you know which ones to ignore? Which ones to delete?

I want to set a `_keep` flag for each item. The _keep flag will be reflected as a checkbox in the form and if unchecked will mean that the child item should be ignored in the validation process.

I could leave the child tables ( invoice items in the concrete sense) empty and use javascript to add a [+] button to add new children. That to me seems like a `bent over` concession.

The `object_graph` function which CRUDS model[s] from a nestable dict is currently decoupled from the validation process and knows nothing of form errors. This might make it hard. Before that I was successfully using `child_crud` hooks in my CRUD controllers for this purpose.

I'll have to sit down and work out which cases I'll need to accommodate. I want to be able to update/delete existing models, create new models and ignore cases where the child CRUD form partials have not been edited.

How to get the formencode.ForEach validator to ignore items without messing up the errors index? These problems were all solved using a child_crud hook in the controllers but trying to abstract and decouple things is making it a lot harder.

It will be a nut worth cracking anyway.

PyParsing + SqlAlchemy = Basic Search Engine

2009-05-11T06:55:00.001-07:00

Today I learned about writing recursive descent parsers using the PyParsing library. I managed to cobble together an sqlalchemy expression builder for a basic search engine.

 1  #################################### IMPORTS ###################################
 2  
 3  # PyParsing
 4  from pyparsing import ( CaselessLiteral, Literal, Word, alphas, quotedString,
 5                          removeQuotes, operatorPrecedence, ParseException,
 6                          stringEnd, opAssoc ) 
 7  
 8  # SqlAlchemy
 9  from sqlalchemy import and_, not_, or_
10  
11  ################################## LIKE ESCAPE #################################
12  
13  LIKE_ESCAPE = r'\\'
14  
15  def like_escape(s):
16      return '%' + ( s.replace('\\', '\\\\')
17                      .replace('%', '\\%')
18                      .replace('_', '\\_') ) + '%'
19  
20  ############################### REUSABLE ACTIONS ###############################
21  
22  class UnaryOperation(object):
23      def __init__(self, t):
24          self.op, self.a = t[0]
25  
26      def __repr__(self):
27          return "%s:(%s)" % (self.name, str(self.a))    
28  
29      def express(self):
30          return self.operator[0](self.a.express())
31  
32  class BinaryOperation(object):
33      def __init__(self, t):
34          self.op = t[0][1]
35          self.operands = t[0][0::2]
36  
37      def __repr__(self):
38          return "%s:(%s)" % ( self.name,  
39                               ",".join(str(oper) for oper in self.operands) )    
40  
41      def express(self):
42          return self.operator[0](*( oper.express() for oper in self.operands ))    
43  
44  class SearchAnd(BinaryOperation):
45      name = 'AND'
46      operator = [and_]
47  
48  class SearchOr(BinaryOperation):
49      name = 'OR'
50      operator = [or_]
51  
52  class SearchNot(UnaryOperation):
53      name = 'NOT'
54      operator = [not_]
55  
56  ############################### REUSABLE GRAMMARS ##############################
57  
58  AND = CaselessLiteral("and") | Literal('+')
59  OR  = CaselessLiteral("or")  | Literal('|')
60  NOT = CaselessLiteral("not") | Literal('!')
61  
62  searchTermMaster =  (
63      Word(alphas) | quotedString.copy().setParseAction( removeQuotes ) )
64  
65  ########################## THREAD SAFE PARSER FACTORY ##########################
66  
67  def like_parser(model, fields=[]):
68      class SearchTerm(object):
69          def __init__(self, tokens):
70              self.term = tokens[0]
71  
72          def express(self):
73              return or_ (
74                  *( getattr(model, field).like( like_escape(self.term),
75                                                 escape = LIKE_ESCAPE) 
76                     for field in fields )
77              )
78  
79          def __repr__(self):
80              return self.term
81      
82      searchTerm = searchTermMaster.copy().setParseAction(SearchTerm)
83  
84      searchExpr = operatorPrecedence( searchTerm,
85             [ (NOT, 1, opAssoc.RIGHT, SearchNot),
86               (AND, 2, opAssoc.LEFT,  SearchAnd),
87               (OR,  2, opAssoc.LEFT,  SearchOr) ] )
88  
89      return searchExpr + stringEnd
90  
91  ########################### SEARCH FIELDS LIKE HELPER ##########################
92  
93  def search_fields_like(s, model, fields):
94      if isinstance(fields, basestring): fields = [fields]
95      parser = like_parser(model, fields)
96      return parser.parseString(s)[0].express()
97  
98  ################################################################################
99

ListFilter Prototype

2009-04-03T03:59:00.000-07:00

Multiple space terminated regular expressions for list filtering.

The examples presented are somewhat hare-brained. a) Cause I'm hare-brained b) I'm really tired at the moment. Note the dramatic pauses :) Can.....you.....see....what....I'm.........do....ing You can imagine the usefulness though.

The filter is a prototype implementation of a filtering syntax for a QuickOpen Panel specifically designed for `quick` (try not to laugh) *multiple item* selection. The idea is to allow you to just keep typing in to refine your search. NOT this OR that. NOT that. `Open All In New Window` etc.

It is implemented using the editors actual text buffer API.

My editors current QuickOpen panel (while it does support multiple selection using ctrl-enter) works a lot like the FireFox url history search; space terminated tokens that each list item must match else will be filtered from the list. This works great for selecting one item but what if you want to open up multiple items at a time?

Plain regex searches are too unwieldy and lack the speed of entry desired.

The filtering is a regex extension of the current `all space terminated word chunks in any order`. It maintains most of the benefits and generally works as before for non regex characters. If you just need to type in some alphanumerics then it should be just as quick. It's essentially multiple regexes instead of multiple substring matches.

OPERATORS:

' ' AND operator
! NOT operator
| OR operator

jQuery = $;

2009-04-02T18:32:00.000-07:00

I have been making a foray into JavaScript recently for work and having had good experiences with `jQuery Lightbox` decided to use it for the basis of a job I was doing.

It is essentially a carpet gallery website with collections of colors (1:M). The designer wanted it to be `AJAX` ( a term that seems to have been hijacked to mean any page updated without slow browser refresh )

In the middle of the page is a large image. Above it are next/prev collection links and tiny `swatches` (thumbnails) containing links to each color for the current selection.

To either side of the image are next/prev color in collection links and upon mouseover a window will appear showing a magnified area following the cursor which is changed to a crosshair. The cursor will change via css styling to `cursor:wait` whilst waiting for the zoom image to load.

Upon changing color (via swatch, next/prev collection/color ) an animated gif will show while the chosen color's image is loading.

At first I was keeping a counter of the current colors position in the collections array of colors (clicking on the little thumbnails would take the title attribute, slug it and use that for the color, updating the current position index by with an $.inArray(color, colors) )

The filenames to load was, and are, a function of current collection and color.

The problem was having the state in an internal counter didn't really work for `open in new tab` or for sending links. "Hey check out this carpet... no the red one... Did you type in that url properly? Just paste it in."

I did some googling for `ajax urls` and stumbled upon a technique that sounded useful. That being polling the hash location for changes and then setting page state as a function of the hash. They said the ideal `polling` rate was 100ms. Sounded pretty hacky but at least the urls worked.

I searched for and found a plugin for jQuery that allows you to set event handlers for when the window hash changes. It uses polling but is responsive (42 ms) and works on IE 6.

I therefore just set a callback to update all the links and the image upon hash changing. I split the hash on '--' to find the current collection and color

The great thing about it is that the event is fired on page load so it goes through the `change color` routine. Updates all the links, shows the loading animated gif etc.

You can send links to people, open in next tab from any of the links etc

jQuery and its plugins made everything really straight forward. The only head scratching was getting the magnifying glass to work. None of the plugins worked out of the box for images that changed src. They worked mainly for a `static` gallery.

CTags 2: TDD

2009-04-02T17:55:00.000-07:00

This is part two of CTags. See part 1

Test Driven Development

Adherents religiously write tests first for *every* function they write.

Personally, I think a tonne of unit tests while prototyping is a waste of time for `simple` stuff with only one programmer working. It tends to feel like walking in mud. I prefer higher level tests that while they may not pinpoint the exact cause of failure won't double refactoring efforts. Especially when prototyping. Spend ages testing that `The Wrong Way` works? No thanks... Prototype, then rewrite with tests.

Sometimes though, `TDD` really is indispensable while exploring. Especially when working on stuff that is pushing the limits of your understanding. If I start encountering bugs I usually see it as a sign I need to start writing tests. Rather than using transient print statements to debug I'll write some unit tests.

The TagFile class below (now commented) is an example of when I found testing while prototyping invaluable.

  1  #################################### IMPORTS ###################################
  2  
  3  from __future__ import with_statement
  4  
  5  import os
  6  import bisect
  7  import mmap
  8  import unittest
  9  
 10  ################################### CONSTANTS ##################################
 11  
 12  
 13  """
 14  The tags in a `tags` file are listed one per line formatted as so:
 15  
 16      tag_name<TAB>file_name<TAB>ex_cmd;"<TAB>extension_fields
 17  
 18  """
 19  
 20  # symbolic constants for column indexes
 21  SYMBOL = 0
 22  FILENAME = 1
 23  
 24  ################################################################################
 25  
 26  class TagFile(object):
 27      def __init__(self, p, column):
 28          """ 
 29          
 30          Instantiate a new TagFile
 31  
 32          @p            path to `tags` file
 33          @column       which column to read
 34  
 35          """
 36  
 37          self.p = p
 38          self.column = column
 39  
 40      def __getitem__(self, index):
 41          "self.fh is the mmap opened by get"
 42          # Seek to a certain byte index
 43          self.fh.seek(index)
 44  
 45          # The position is likely to be halfway through a line so read up to
 46          # the first new line and throw away the `junk`
 47          self.fh.readline()
 48  
 49          # Note that it's actually returning the column from the line *after* the
 50          # line region containing the index.
 51          
 52          return self.fh.readline().split('\t')[self.column]
 53  
 54      def __len__(self):
 55          # bisect.bisect_left search must know how large the file is
 56          return os.stat(self.p).st_size
 57  
 58      def get(self, *tags):
 59          """
 60          
 61          Get all lines for one or more tags
 62          
 63          """
 64  
 65          with open(self.p, 'r+') as fh:
 66              # mmap is not needed but delivers performance increase
 67              self.fh = mmap.mmap(fh.fileno(), 0)
 68  
 69              for tag in tags:
 70                  # As __getitem__ returns the colum from the line region *after*
 71                  # that containing the index pt then bisect( alias of
 72                  # bisect_right ) will give the wrong index.
 73  
 74                  b4 = bisect.bisect_left(self, tag)
 75  
 76                  # Move the file to the position found
 77                  fh.seek(b4)
 78  
 79                  # Iterate over file. There may be more than one tag line to get
 80                  # per symbol/filename
 81                  for l in fh:
 82                      # Compare search vs line at column
 83                      comp = cmp(l.split('\t')[self.column], tag)
 84  
 85                      # This handles the case of being `left of left` due to 
 86                      # __getitem__ index being left of symbol it returns
 87                      # ie wait until catch up
 88                      if    comp == -1:  continue
 89                      # If line is greater then have moved on to next symbol
 90                      elif  comp:  break
 91  
 92                      # Found tag!
 93                      yield l
 94  
 95              # close mmap
 96              self.fh.close()
 97  
 98  ##################################### TESTS ####################################
 99  
100  class CTagsTest(unittest.TestCase):
101      def test_tags_files(self):
102          """
103  
104          This test basically iterates over each line in the tags file creating
105          a list of lines for each unique symbol it finds. It then compares this
106          list to that returned by the TagFile binary search. 
107  
108          """
109  
110          # Successfully passed test on 10MB+ tags file
111          # tags = r"C:\Python25\Lib\tags"
112          
113          tags = 'tags'
114          tag_file = TagFile(tags, SYMBOL)
115  
116          with open(tags, 'r') as fh:
117              latest = ''
118              lines  = []
119  
120              for l in fh:
121                  symbol = l.split('\t')[SYMBOL]
122  
123                  if symbol != latest:
124  
125                      if latest:
126                          tags = list(tag_file.get(latest))
127                          self.assertEqual(lines, tags)
128  
129                          lines = []
130  
131                      latest = symbol
132  
133                  lines += [l]
134  
135  if __name__ == '__main__':
136      unittest.main()
137  
138  ################################################################################

CTags 1: Captain Bisect And His Rag (c)Tag Crew

2009-03-29T20:21:00.000-07:00

CTags

Ctags generates an index (or tag) file of language objects found in source files
that allows these items to be quickly and easily located by a text editor or
other utility. A tag signifies a language object for which an index entry is
available (or, alternatively, the index entry created for that object).

Python has a philosophy of `Batteries Included` (and `Designer Straight Jacket`)
and its standard library has many useful modules to allow you to zoom out and
fly. Say what? What pixie dust have you have been snorting!?

The higher level you are the higher level you can go. If your stuck in the
trenches of detail it's hard to see emergent patterns to exploit which can help
you simplify code. Refactoring and simplifying is generally an iterative process
of simplifying, getting a higher level view and simplifying again.

Taking a bottom up approach in this helps as you know what building blocks you
need to create along the way. Don't want to be *too* simplistic. How can you do
things `top down` if you aren't `on top`?
Maybe you have flown over many times already and are just implementing an old
innovation from a time honoured map. That's exactly the spirit of prototyping.
Sending in the scouts to survey the terrain and report back.

However, those troops get mighty mutinous really quick if you don't take the
time to at least make a spot before you send em out like a swarm of jellyfish
(especially when they are being fired at)

Strategy? Fly over, send in the scouts and maintain communication.

What the hell is all that got to do with CTags? Not much I admit. But we does
love to babble.

Python std libraries are your `A Team` of special operatives that can do the
work you tell em without constant supervision and no need to worry about the
messy details. Go nuke PHP! "Yes Sir!"

One of these crackshot ninjas is Captain `bisect` with his special move
`bisect_left`.

He's a great leader because he's absolutely fastidious in keeping his company of
troops in perfect sorted order at all times.

Some of his duties involves training new recruits and he'll do a trick where
upon meeting them all for the first time he will get them to silently line up in
alphabetical order while he has his back turned. (we'll say alphabetical by
name, but well, these guys *are* macho army men so..)

>>> scum = "omewEDjyFapAdxslfhbgBcnCtkqzuvri"

>>> len(scum) == 32
True

>>> men = ''.join(sorted(scum))
>>> men
'ABCDEFabcdefghijklmnopqrstuvwxyz'

( Isn't it funny how the ones that try and make them selves seem big are in fact the smallest? )

" I bet I can find the position in the line of any one of you maggots after at most 5 guesses "

A, Yeah right, he thinks. "Find q sir"

bisect, "man 16 are you a lesser man than q?" k, "Yes"
bisect, "man 24 are you a lesser man than q?" s, "No"
bisect, "man 20 are you a lesser man than q?" o, "Yes"
bisect, "man 22 are you a lesser man than q?" q, "No" (smiles)
bisect, "man 21 are you a lesser man than q?" p, "Yes"

bisect, "man 22 you are q!"

troops, "Bravo Sir!"

( bisect is a a bit of a kookoo and uses 0 based indexing. Lucky the scum neverseem to get confused )

How does he do it!?

He starts in the middle (32 / 2 = 16) and compares what he was searching for with what he finds there. What he finds, k, is less than q so he knows he can rule out all other men before k as they too would be less than q.

This only worked for him because all the men are in order.

He then subdivides again. The man at position 16, k, is less than q so his area will start at
17 and extend in the opposite direction ending (as before) at 32 (the greatest man). He then looks for his next midpoint with an eye to ruling out another half of the remaining men.

( He repeats this simple process until his start point is no longer less than the
end point )

((17 + 32) // 2 = 24)
men[24] (s) is not lt q so his end point becomes 24

((17 + 24) // 2 = 20)
men[20] (o) is less than q so his start point becomes 21

((21 + 24) // 2 = 22)
men[22] (q) is not less than q :) so his end point becomes 22

((21 + 22 // 2 = 21))
men[21] (p) is less than q so his start point becomes 22

Start is 22 and end is 22 so he has a match!

With 64 men (double 32) he would only take 1 more guess; As each guess rules
out half and half is the inversion of double.

What about 128 (or some arbitrary number?) How many times can you half 128
before you get one (the right `one`).

Or in other words what to the power of two makes 128?

32 log 2 = 5
64 log 2 = 6
128 log 2 = 7

import math
math.log(128, 2)

Captain bisect's talent scales exceedingly well and in fact he's ready to put
forth his talent to whatever use come up with.

import bisect
bisect.bisect_left(some_sequence, search)

WOAH What a verbose explanation! Too much detail! Couldn't you just have said
use `bisect.bisect_left` to index left most occurence of any item in a sequence.

Yeah, that's kind of the point. Python is chock full of handy high level
abstractions you can trust to do the job without worrying about the details.
You are already zooming.

"Ok you admit your are babbling but what the heck is this got to do with
CTags?"

To paraphrase, Ctags generates an index (in sorted order) of symbols to
be quickly and easily located. Sounds like a job for Captain bisect. He is
actually made of this stern stuff called `C` that makes him faster than anything
you could genetically engineer in your laboratories.

Python also has this useful thing going called duck typing. bisect will work
on any class that exposes a __getitem__ method.

But what if you wanted Captain bisect to search a 50 MB ctags file? You can't
sub class a file object... can you? In any case each index would just return
a character wouldn't it?

A sneak preview of how to use bisect and mmap to binary search CTags `tags` files. Explanation to follow.

 1  #################################### IMPORTS ###################################
 2  
 3  from __future__ import with_statement
 4  
 5  import os
 6  import bisect
 7  import mmap
 8  import unittest
 9  
10  ################################### CONSTANTS ##################################
11  
12  # CSV Column in tag file
13  SYMBOL = 0
14  FILENAME = 1
15  
16  ################################################################################
17  
18  class TagFile(object):
19      def __init__(self, p, column):
20          self.p = p
21          self.column = column
22  
23      def __getitem__(self, index):
24          self.fh.seek(index)
25          self.fh.readline()
26          return self.fh.readline().split('\t')[self.column]
27  
28      def __len__(self):
29          return os.stat(self.p).st_size
30  
31      def get(self, *tags):
32          with open(self.p, 'r+') as fh:
33              self.fh = mmap.mmap(fh.fileno(), 0)
34  
35              for tag in tags:
36                  b4 = bisect.bisect_left(self, tag)
37                  fh.seek(b4)
38  
39                  for l in fh:
40                      comp = cmp(l.split('\t')[self.column], tag)
41  
42                      if    comp == -1:    continue
43                      elif  comp:          break
44  
45                      yield l
46  
47              self.fh.close()
48  
49  ##################################### TESTS ####################################
50  
51  class CTagsTest(unittest.TestCase):
52      def test_tags_files(self):
53          tags = r"tags"
54          tag_file = TagFile(tags, SYMBOL)
55  
56          with open(tags, 'r') as fh:
57              latest = ''
58              lines  = []
59  
60              for l in fh:
61                  symbol = l.split('\t')[SYMBOL]
62  
63                  if symbol != latest:
64  
65                      if latest:
66                          tags = list(tag_file.get(latest))
67                          self.assertEqual(lines, tags)
68  
69                          lines = []
70  
71                      latest = symbol
72  
73                  lines += [l]
74  
75  if __name__ == '__main__':
76      unittest.main()
77  
78  ################################################################################

GSOC OverView

2008-08-28T18:41:00.001-07:00

My GSOC project was all about testing for PyGame;

I wrote lots of tests; Almost every module in PyGame now has at least one test

Test modules can now be isolated in subprocesses; one segfault no longer brings down the whole test suite

Can now test for speed regressions; important for real time software such as games

PyGame Automated Build Page extended
- Shows / Collects more info
- Runs tests in subprocesses

Test Stubbing Utility: A Testing "Todo List"

Optional Interactive Tests / Test Tagging

For writing the tests I wrote a small utility that inspects the PyGame package and finds all the untested callables (functions, properties) and creates test stubs, including documentation for each so you don't have to leave the editor. The stubber knows which functions have already been tested by using a naming scheme for all of the tests. Essentially, "test_$callable__$comment", namespaced by having TestCase[s] per Class and a test module per module.

In this way I could create stubs for each module, essentially a TODO list, and cycle through all the modules looking for tests that were easy to write. The functions in PyGame are many and greatly varied, each requiring somewhat specialised knowledge to test. I wasn't able to write tests for all them but hopefully the test stubbing utility will help enable some testing sprints. I intend to develop a testing website where people can submit bugs/tests in the form of a unittest.

PyGame has a somewhat unique set of requirements compared to most python libraries in that most of the framework is actually written in C. C code when it goes awry can do some very strange things. We had a test runner running all of the tests in one single process so if one failed hard it would bring down the whole suite. This can be a bit of a pain so I developed a test runner that isolates each module in a subprocess.

Some of the tests in PyGame have requirements that make them unsuitable for running as part of the main test suite. For example some require a CDRom, a JoyStick, take way too long or need interaction with a human. With the test runner script I extended unittest with the ability to exclude certain tests by tags. The tags can be module, class or individual test level and are inheritable/ over-ridable.

Another extension to the test runner was the ability to randomize the run ordering of tests, so along with the test results the seed is printed out. If there are failures you can seed the randomizer with the failure inducing seed. We also wanted to be able to record the timings of each individual test so we could make comparisons between revisions / platforms. I again extended the test runner with that ability.

I worked with Brian Fisher to extend the PyGame automated build page to record the test results in a ZODB and utilize the new test runner to run tests in subprocesses. We will be able to use this information for detecting speed regressions amongst other things.

Johnny, Kick A Hole Right In The Sky

2008-08-23T08:25:00.000-07:00

Johnny, Kick a hole right in the sky! Won't some body testify? Poke a lion in it's eye!

I bought pygame-testify.net today, and set up a python/cgi based form that takes a zip and enumerates the results + adds the (safe evaled) test results dict to a ZODB.

I found a multi-part python snippet for POST[ing] of test results.

The test/build page is starting to come together.

I am using htpasswd for security.

todo_xxxxxxx

2008-08-02T16:22:00.000-07:00

I recently altered the "fail incomplete tests" mechanism we use in the pygame test runner. Before we were doing assertions on test_utils.test_not_implemented(). This would check a module level variable test_utils.fail_incomplete_tests, which we would set as desired depending on whether we wanted to fail incomplete tests.

This was a fairly non-invasive technique but as I was already hijacking the test loading mechanism for filtering tests by tags, I realized I could alter the TestLoader class to pick up tests starting with the prefix "todo_" as well as "test_". I would call TestCase.fail directly which would only run if picking up todo_ tests.

This of course meant altering all the stubs. I pondered briefly doing a mass search and replace, completely automating it but I don't really trust that for tests.

For the test stubs I have been including the documentation so it's really easy to walk through a test file writing tests without having to leave the editor. I was just using inspect.getdoc to get the __doc__ string.

It seems the documentation included in the .doc files is different to that contained in the __doc__ for each function. The __doc__ seems to be the function signature and a very brief, usually one sentence description. The .doc files contains a lot more detailed descriptions that can be very useful when writing tests.

I quickly added a docs_as_dict() function to makeref.py, then added it to the stub generator. The stub generator will add both the __doc__ and the .doc file documentation to each stub.

I went through semi-manually updating all the unfilled out stubs for each test file with the more complete docs and the new todo_xxxxx test naming. It took about an hour but I feel more confident than if I had just grep'd it.

Everything is pretty much now in place for the test site I wanted to create.

Test Timing
Test Tagging
Isolated Tests

import test.unittest as unittest

2008-07-18T21:14:00.000-07:00

I split the test runner further, now into three files, with all the monkey business in unittest_patch.

The patching is done by a patch() function taking an optparse options object as the solo argument, which drives the decisions behind which parts of unittest are patched.

With the features we wanted I had to override some methods in a quite drastic way. I even needed to override TestCase.run, a many many line method. The only way I could do this was to basically copy/paste, alter and monkey-patch in. This meant sometimes calling private members.

Unfortunately, the author of unittest had decided somewhere between python 2.4 and 2.5 that he would rename all the private members from the double underscore preceding __name_mangling convention to a single underscore _caution.

As my mentor said (or something like it), "using an underscore is a warning, that said member is an implementation detail not an interface".

What to do? We now include a 2.5 version of unittest in the test directory. Apparenly pygame has come full circle; it was included way back in the day before PyUnit was part of the standard library.

All of our individual test files, typically $module_test.py, all import an unpatched unittest and run unittest.main() to make the module "conveniently executable". Only when running the complete suite is unittest enhanced with extra functionality.

While I was in there tinkering with the internals, recording timings of individual tests I moved the redirect std(err|out) per module to per test. I then patched the TextTestRunner to dump stderr/stdout on error.

def printErrorList(self, flavour, errors):
    for test, err in ((e[0], e[1]) for e in errors):
        self.stream.writeln(self.separator1)
        self.stream.writeln("%s: %s" % (flavour, test))
        self.stream.writeln(self.separator2)
        self.stream.writeln("%s" % err)

        # DUMP REDIRECTED STDERR / STDOUT ON ERROR / FAILURE
        if self.show_redirected_on_errors:
            stderr, stdout = map(self.tests[test].get, ('stderr','stdout'))
            if stderr: self.stream.writeln("STDERR:\n%s" % stderr)
            if stdout: self.stream.writeln("STDOUT:\n%s" % stdout)

It would be relatively easy to add in support for show locals() etc.

Redesign

2008-07-15T18:12:00.000-07:00

I decided to (had to) redesign the test runner, this time cutting more directly to the root of matters, overriding select methods of unittest classes.

Before, in subprocess mode, I was calling the individual test modules, which would in turn run unittest.main() with all the attendant pains of cmd line options conflicting and output parsing. (we have to add profiling, exclusion by tags etc). One major design change I made was to unify the single / subprocess modes to use one test runner, (test_runner.py).

In it, along with a lot of utility functions, is defined a run_test() function. It takes a list of modules and an options object as arguments. It compiles a dictionary of the test results and on completion either returns the dict or in subprocess mode pretty prints it to stdout. (This is then eval'd for an all_results.update(result))

RESULTS_TEMPLATE = {
    'output'     :  '',         # unittest.TextTestRunner output
    'stderr'     :  '',         # stderr outpout 
    'stdout'     :  '',         # stdout output
    'num_tests'  :   0,         # taken directly from the unittest results object
    'failures'   :  [],         # ditto
    'errors'     :  [],         # ditto
}

In single process mode run_tests.py just imports from test_runner.py run_test() function and passes it the optparse options object and list of modules to search for tests.

Both run_tests.py and test_runner.py, share the same optparse cmd line parser options. In subprocess mode, run_tests.py calls test_runner.py with essentialy the same sys.argv it was initiated with. if __main__ it runs the run_test() function on a list of [args[0]]. Now all the extra functionality and cmd line parsing is all in one place.

There were quite a few extra little changes that have made it not perfect but a lot better. Adding exclution by tagging functionality took 10 minutes, most of the time being spent on picking a format.

|Tags:display|

Adding profiling decorators or whatever other functionality is desired will also be a lot easier now.

Comedy Of Errors

2008-07-10T18:09:00.000-07:00

** Build Page / Testing **
==========================

As reported earlier, in reaction to the crashing tests rendering the build page ineffective, I have been working on creating a script to isolate test modules in subprocesses. The approach I took, was to compile the results of each isolated test into the same form as the old test runner. A quick hack, or so I hoped.

I realised that subprocess out of the box has no cross platform non-blocking calls, so you can't timeout on hung tests. I had to find a recipe for this which unfortunately required win32 extensions. Not really a big deal but still time spent and dependencies.

So what we have is a test runner parsing the results of a unittest text report, meant for human consumption, which is then in turn parsed by the automated build pages regexes. This seems pretty ridiculous, especially as the form is not exactly machine friendly. I could have (should have?) hacked into the build page code and modularised the test parsing code there, sharing between the test runner and build page.

But then if you are going to do that why not just replace the TextTestRunner class with something completely customised for the job? Replace unittest bit by bit in an adhoc as-needed fashion? Slowly building a framework? I didn't want to. I'm not really supposed to be and that was the psychology in play.

Another? foolish design decision I made, based on a shallow visual aesthetic of less LOC, was to parse unittest results in a way that only worked when there was no "test noise". What do I mean by test noise? print statments left in source code. C extensions that don't respect sys.stderr, sys.stdout redirection/supression.

See below exhibit A, a specimen from a sunny day of testing.

...............
---------------------------------------------------------------------
Ran 15 tests in 1.234s

As the tests are running unittest prints to a stream, by default stderr, but it can be any file-like object of choice, either a dot an E or an F, mapping to pass, error or fail. I used a simple regex ^[.EF]*$ to find any "dots" in the return output. If there were any, I would take a slice from the length of the dots. From there I would take the first of a split at the "Ran xxx tests" boundary, defined as '%s\nRan' % (70 * '-'). In between the DOTS and the RAN_TEST_DIV (thus named) would lay the failures.

To piece it all together as if it was the output of one run I would "join the dots" and join the failures. Then at the end count the total length of DOTS (., E, F combined), E)rrors and F)ailures. Voila. Worked a charm.

What the hell was I thinking? The whole point of the exercise was to create a reliable test runner. I suppose I thought I was. I wrote a few tests for some spectacularly unimaginative cases. I compared output of single process mode and subprocess mode running some fake test suites, zero assertions, all passing, some failures, some errors. The subprocess mode was character for character perfect in its mime artistry. In fact it was for this easy, pull apart, bind together, compare automated testing that I did it in the first place.

All was simple and peaceful, until I finally got a linux test box working again. (my laptop fan died) I used ssh to log in and run the tests from my friends windows machine. Of course one of the tests that required initiating the display failed.

single process mode: 504 tests, FAIL (failures=1)
subprocess mode: 495 tests OK

What the hell was going on? With horror I realized what I had done. Something was wreaking havoc with the fragile little regex. On failure a huge amount of debugging output was put out by one of the SDL functions interupting the DOTS. I thought about rewriting it using some more substantive regular expressions. I tossed up between doing that and redirecting sys.(stdout|stderr) and passing a StringIO to unittest for test results. I figured by doing that I would be able to keep the comparison tests I had in place, and for that matter the same degree of mimicry. I opted for redirecting std(err|out). I imagined other uses for this at the same time, none all that compelling upon reflection and only useful if implemented in another manner. (only show stdout/stderr on failure of test, can leave print statement debugging in there, I did global redirection)

Of course to do that I needed to create a command line option for each individual test module to call from the "master" script in subprocess mode. Because unittest.main() is running with it's own getopt parsing, you can't just add an option and check sys.argv or use optparse. You have to do either and then clear those options from sys.argv which would otherwise cause unittest to error. So more fun hamfisting around with unittest. I realised that I would need to do that at some stage for profiling cmd line options so there was another push in that direction. All the time wondering whether I should just completely override the parseArgs method.

I replaced the test_utils.get_fail_incomplete_option();unittest.main() in each module with a test_utils.get_command_line_options(). unittest.main() always calls sys.exit() on completion of tests so I had to subclass it, overriding one of it's methods. I did this because after catching the unittest result stream to a StringIO, I would restore stderr and write the results to it.

I added in some test cases, print_stdout and print_stderr, comparing the results (I of course had to put a redirect mode onto single process mode for purposes of testing). Everything was OK again, until I ran it again on the the linux box through ssh.

495 tests OK. (should have been 504 with one failure)

Damn it! So it seems that some stderr, stdout is not redirected. I imagine it's mostly C extensions (or system calls) and the like that would do this but then that is pygame all over. Briefly I pondered printing results back out on stdout, and just PrayingTM that any such noise would always be stderr.

So what did I do? What any fool, already invested would do. I decided to markup the results, with lines like.

<!-- UNITTEST_RESULTS_START_HERE --!>

I created 3 sections using 2 divisors. The first is all the noise output, anything not respecting redirection. The second is the unittest results and the last is the multiplexed results, what you would see if running the script in a shell. I overrode the write method on a StringIO collecting unittest results and made it also write to (a previously redirected) stdout. using subprocess.Popen(...., stdout=subprocess.PIPE, stderr = subprocess.STDOUT) everything is muxed together. I then wrote a function that regex splits the 3, keeping the results for compiling DOTS. It's a long way from Kansas though isn't it Toto.

What a PITA? That's not even the half of it. I ended up having to rewrite all the command lines I was passing to subprocess.Popen from string template to lists so it would work cross platform. Also, the way subprocess multiplexes stderr and stdout when you use the same file object for both is inconsistent cross platform. What you would see is not neccessarily what you get. On windows it would suffice to just "print compiled_test_results", but on linux had there was need to print >> sys.stderr.

All in all, a lot of tipsy toeing around unittest. I really made a complete tangled webby mess of the whole job. A black comedy of errors. I'm not sure whether to remove the stderr/stdout redirection and replace the regexes with something less fragile. It's already been too much of a hole, sucking in time. I would have to update the run_tests__tests also.

What would I do differently looking back? What would I do if I had no constraints? Unfortunately, probably two very different questions.

** What I would do differently? **
==================================

This much I do know, the build page and the test runner script require intersecting functionality. They both parse the results of a unittest TextTestRunner output to gather statistics on test results. I could have modularised this parsing functionality, sharing between the two of them. This really begs the question though, why parse something designed for human consumption at all? Why not pass a customised test runner class into unittest?

Still there is the problem of communication across process boundaries, solved by using an asynchronous extension class of subprocess.Popen. Would you log the result of each processes output to a file using something like xml? Or maybe, pickling the results and then joining them back together? You could even have a client / server architecture, using sockets to transfer pickled test results as native python objects back to the server to piece together.

As well as the requirement for isolation of tests, we are wanting to add profiling functionality and tagging to split tests into different groups.

killer redux

2008-07-08T19:58:00.000-07:00

I was laboring under the bastard conception that when using subprocess.Popen(), shell=True is required for a subprocess executable to have access to the environment variables. Where the hell did I get that idea? Stupid unquestioned assumption that almost gave birth to a lasting bug.

For the test runner I was using system calls to taskkill or pskill for process controll under windows. The idea was to try executing each and if one was on the %PATH% the return code would not be one of err. If this was the case then the search was over and a Popen wrapper of (taskkill|pskill) would suffice as an os.kill().

This worked fine and dandy except that on windows98, there would be no error code if either of the task killers weren't on the path. It would define a useless os.kill.

Lenard, the windows maintainer of PyGame questioned why use a hacky wrapper of pskill or one of it's ilk, when if there was already a reliance on pywin32, why not use win32api.TerminateProcess?

That works fine but does not kill process trees, something I thought was a requirement due to using shell = True as a Popen constructor argument. Using shell = 1 calls cmd.exe etc which in turn calls the subprocess of choice.

Realizing that there was only need to kill one process, and that it would also avoid problems with differing return codes on older versions of windows, TerminateProcess was given the job.

Long live TerminateProcess.

dot points on build page extensions

2008-07-04T23:28:00.000-07:00

I have been thinking about making some extensions to the build page.

Raw Data

Keep raw_data to process at any time. No need to discount old data collected from buggy analysis.

Profiling

Use function wrappers, that log profiling of each test and multiple calls.

-p|--profile command line mode

Tests

Use subprocess mode by default for run_tests.py

Web interface for ticketing off tests

Build information

Post compiler version

Post complete Setup file

Post complete build output

Post complete test output

Python sys.path

Environment variables

As much as possible, unprocessed for archives

Machine information

Processor speed

CDRom availability

etc, etc.

Breaking up tests

Should the tests fail if a machine doesn't have a CD drive (assuming stubs were filled out) for example?

Should tests that require Numeric or NumPy fail if neither available?

There are some classes of tests that it seems to make sense to split apart from the main "base" group of tests. What should be the "base" group of tests to automate with the run_tests.py test runner?

What about tests that require human verification? For the build page a "base" group of tests should be specified.

What should be the requirements for machines sending results to the build page? Numeric, Numpy? win32 extensions on windows? A CD rom drive? 32 bit color display?

test_not_implemented()

2008-07-03T21:10:00.000-07:00

def test_get_arraytypes(self):

    # __doc__ (as of 2008-06-25) for pygame.sndarray.get_arraytypes:

      # pygame.sndarray.get_arraytypes (): return tuple
      # 
      # Gets the array system types currently supported.
      # 
      # Checks, which array system types are available and returns them as a
      # tuple of strings. The values of the tuple can be used directly in
      # the use_arraytype () method.
      # 
      # If no supported array system could be found, None will be returned.

    self.assert_(test_not_implemented())

test_not_implemented() will fail if any test suite is run with a "(-i|--incomplete)" command line option.

As mentioned in previous posts, I developed a unittest stub generator that will output stubs for any untested units. It is supported by a naming scheme for the tests. The stubber will inspect the xxxx_test.py modules and based upon the names of the unittest.TestCase's and their children test_xxxx methods will determine what is already tested.

For each public callable there is a corresponding test named test_$callable_name. Comments or descriptions will be appended to this separated by a double underscore.

test_quit__returns_None_if_not_already_init

What if there is a module.quit and a module.class.quit ? Each class has it's own TestCase (and thus namespace) named $classTypeTest. This is typically the case anyway with setUp()'s specific to the class tested.

def get_callables(obj, if_of = None, check_where_defined=False):
    publics = (getattr(obj, x) for x in dir(obj) if is_public(x))
    callables = (x for x in publics if callable(x) or isgetsetdescriptor(x))

    if check_where_defined:
        callables = (c for c in callables if ( 'pygame' in c.__module__ or
                    ('__builtin__' == c.__module__ and isclass(c)) )
                    and REAL_HOMES.get(c, 0) in (0, obj))

    if if_of:
        callables = (x for x in callables if if_of(x)) # isclass, ismethod etc

    return set(callables)

The script uses inspection to find all testables in pygame but there were a few complications, for example getter/setter properties and the fact that some objects need to be instantiated before inspection reveals their innards. Also, filtering out non-pygame callables and after that callables that appeared in more than one module.

eg pygame.rect.Rect led a double life as pygame.sprite.Rect. Just check the __module__ attribute ?

In [4]: pygame.sprite.Rect.__module__
Out[4]: 'pygame'

The workaround was to make a mapping of object to the place where it was defined. There were only 9 of these.

REAL_HOMES = {
    pygame.rect.Rect         : pygame.rect,
    pygame.mask.from_surface : pygame.mask,
    pygame.time.get_ticks    : pygame.time,
    .....

On some of the classes the __module__ attribute was __builtin__ so I needed put an exception for them in the filtering out of non pygame callables.

In [7]: pygame.cdrom.CDType.__module__
Out[7]: '__builtin__'

def module_stubs(module):
    stubs = {}
    all_callables = get_callables(module, check_where_defined = True) - IGNORES
    classes = set (
        c for c in all_callables if isclass(c) or c in MUST_INSTANTIATE
    )

    for class_ in classes:
        base_type = class_

        if class_ in MUST_INSTANTIATE:
            class_ = get_instance(class_)

        stubs.update (
            make_stubs(get_callables(class_) - IGNORES, module, base_type)
        )

    stubs.update(make_stubs(all_callables - classes, module))

    return stubs

The stubber finds all modules in the pygame package. For each module it uses inspection to create a set of all the callables minus those set in the IGNORE setting. This is here for any exceptions to the filtering and also for tests that have been grouped under one test name. These objects will not be stubbed.

IGNORES = set([

    pygame.rect.Rect.h,           pygame.rect.Rect.w,
    pygame.rect.Rect.x,           pygame.rect.Rect.y,

    pygame.color.Color.a,         pygame.color.Color.b,
    pygame.color.Color.g,         pygame.color.Color.r,

......

From that it creates a subset of "classes", the criteria being that for each element "inspect.isclass(element)" or that the element is in the manually set MUST_INSTANTIATE dict. This is a mapping of class to helper function, and instantiation args required to return an instance.

MUST_INSTANTIATE = {
    
    # BaseType / Helper               # (Instantiator / Args) / Callable

    pygame.cdrom.CDType            :  (pygame.cdrom.CD, (0,)),
    pygame.mixer.ChannelType       :  (pygame.mixer.Channel, (0,)),
    pygame.time.Clock              :  (pygame.time.Clock, ()),


..

}

Inspecting the xxxxType would reveal no methods, and they needed to be instantiated, but then the object returned contained no other attributes; one example being __name__ needed later for determing the test name. Therefore the xxxxType was sent to the stub generation function as the "parent class" for each callable that was gathered by inspecting the instantiation.

Any callables not in the "classes" set are assumed module level functions and a stub is created for each.

The test stubber is used from the command line:

$ gen_stubs.py --help
Usage:
$ gen_stubs.py ROOT

eg.

$ gen_stubs.py sprite.Sprite

def test_add(self):

    # Doc string for pygame.sprite.Sprite:

    ...


Options:
  -h, --help        show this help message and exit
  -l, --list        list callable names not stubs
  -t, --test_names  list test names not stubs

$ gen_stubs.py pygame -l
pygame.base.error.args,
pygame.bufferproxy.BufferProxy.length,
pygame.bufferproxy.BufferProxy.raw,
pygame.event.Event,
pygame.image.tostring,
pygame.joystick.Joystick,
pygame.key.get_repeat,
pygame.mask.Mask,
pygame.mixer.Channel,
pygame.movie.Movie,
pygame.overlay.overlay.display,
pygame.overlay.overlay.get_hardware,
pygame.overlay.overlay.set_location,
pygame.pixelarray.PixelArray.surface,
pygame.sprite.AbstractGroup.add,
pygame.sprite.AbstractGroup.add_internal,
pygame.sprite.AbstractGroup.clear,
pygame.sprite.AbstractGroup.copy,
pygame.sprite.AbstractGroup.draw,
pygame.sprite.AbstractGroup.empty,
pygame.sprite.AbstractGroup.has_internal,
pygame.sprite.AbstractGroup.remove,
pygame.sprite.AbstractGroup.remove_internal,
pygame.sprite.AbstractGroup.sprites,
pygame.sprite.AbstractGroup.update,
pygame.sprite.collide_rect,

Commas are appended for easy copy/paste into IGNORE list.

gen_stubs.py is an integral part of the plan to make it extremely easy for people to contribute to unittests. One man can only do so much.

subprocessed

2008-06-30T20:05:00.000-07:00

PyGame tests are structured in such a way that for each module in the pygame package (eg pygame.sprite, pygame.color) there is a test/xxxx_test.py file containing corresponding unittests. PyGame has an automated build page that shows build and test results for the latest svn version of PyGame on a variety of platforms and versions of python. It uses regular expressions to parse the results of the test runner script.

The test runner script compiles tests from each of the xxxx_test.py files and runs them in a single process. Advantage: speed, disadvantage: instability. PyGame uses a lot of c code, and where there is c code there is potential for strange errors.
As one example, there was a test for the ability to save OpenGl surfaces which would segfault on windows. This would stop the test runner half way through, leaving it's output in a form the automated build page could not decipher.

"Build Successful, Invalid Test Results"

Other issues with running all tests in one process is the need to restore a "fresh" state for tests that rely on it. Conflicts can cause the test runner script to crash completely. On the other hand, some obscure bugs have been uncovered due to them.

Besides writing tests for individual units I have recently been working on adding a subprocess mode to the python test runner script. It processes the output of each module's test script and outputs the results in the same form as the single process mode.

There is a library called subunit that uses os.fork() to run unittest suites in subprocesses, that seemed like it would have been a perfect candidate for the job. Unfortunately windows doesn't have the fork system call so it was not an option. Windows python does not even provide os.kill().

What good is running all tests in subprocesses if one of them hangs and python is using a blocking call to retrieve it's output?
As I was going to the trouble of making a subprocess mode, I realized I should deal with this possibility. Unfortunately the python subprocess module doesn't ship with async calls but I found a recipe on the ActiveState Python CookBook site.

On windows it relies on win32pipe and win32file from the pywin32 package. I worked around the lack of os.kill on windows by using sytem calls to "taskkill" or "pskill". If a wayward test suite running in a subprocess doesn't finish up in a specified allowance of time then it will be os.kill'd.

COMPLETE_FAILURE_TEMPLATE = """
======================================================================
ERROR: all_tests_for (%s.AllTestCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test\%s.py", line 1, in all_tests_for

subprocess completely failed with return code of %s

cmd: %s

return (abbrv):
%s

"""  # Leave that last empty line else build page regex won't match

Running each test suite in a subprocess is a huge performance hit. I think for the automated build page the performance hit won't really effect the experience as it's all running headlessly from cron jobs. Nevertheless, I added the ability to run subprocessed tests simultaneously in multiple threads. Also, the single process mode is still available as is running module specific tests suites.

I wrote some tests comparing the output of (single|sub)process modes running a group of fake test suites, some all OK, some with errors and failures.

$ run_tests__test.py
all_ok suite OK
failures1 suite OK

2/2 passes

-h for help

$ run_tests__test.py -h

-v, to output diffs even on success
-u, to output diffs of unnormalized tests

The standard library module difflib is very good, and extremely well documented.

Other than the obvious differences such as timing which are normalized before comparison, all is OK :)

Testing

2008-06-25T18:48:00.000-07:00

Just a quick note,

Still waiting on my fan, it's getting shipped in from Amurricah.

I wrote a run_tests_sub.py the other day that uses subprocess to run each xxxx_test.py in the trunk/test directory.

It will run with an optional threads paramater: -t num_threads

$ run_tests_sub.py -t 4

Apparently this runs faster on mult-core.

It should output results similar to run_tests.py though may need tweaking to get it run transparently in place of run_tests.py for the build page.

Speaking of the build page, Rene and I have had a few ideas for a combined build / test web app that collected builds and test statistics ( profiling / passes etc). Also, a means to distribute the writing of tests. Many hands make light work.

If it was possible to be assigned a stub of a test to fill out and then post it back painlessly we could quite quickly increase the coverage of our tests. If twenty people filled out 1 test a week, then over a month that would be 80 extra unit tests.

ATM there are "FAILED (failures=232)", unimplemented tests and possibly that many again that haven't been stubbed out waiting to be written.

$ run_tests.py -i

Will show tests that need fleshing out.

Aha!

2008-06-20T22:41:00.000-07:00

I realised why the change from CONSTANT = (expr) to CONSTANT = [expr] fixed the bug in the color_test.py

A generator expression is only good for one iteration and after that it will act as an empty sequence. I thought it would be a reusable lazily evaluated simily of a list comp. Turns out I was dead wrong.

I went over the stub generator recently and it's pretty much in it's finalized form as far as the naming scheme is concerned.

Can't wait to get my own computer back in action.

Fan

2008-06-18T03:26:00.000-07:00

Damn fan on my laptop packed it in. I will have to convince my friend to let me install linux on his windows box while I wait for a replacement. Can't get a windows build of development pygame at the moment due to failing tests... or can I? Temporarily disable the failing tests and let the build farm run?

What a pain in the arse.

Happenings

2008-06-12T21:05:00.000-07:00

I have been writing unittests using the naming scheme (see below) keeping to it as much as possible.

There have been a few modifications but that's fine as long as I am consistent. I haven't yet written the part of the test stub generator that filters from the generated tests any tests for units that have already been written. I am letting the writing of tests dictate the naming schemes evolution.

Will post some more thoughts on the naming scheme in days to come. Also thoughts on one to one test names.

Thoughts on speed of test suites, isolating "dangerous" tests that can crash the whole test suite.