optilude

Improving Plone's third party products story

2006-11-06T13:30:00.000Z

We've had a lot of discussions recently about quality management of plone.org/products - how do people who don't spend their days on plone-user and #plone work out what products are worth betting on, where they are going and whether they'll stay viable in the future?

A lot of really great ideas came out of the session we held at the Plone Conference. We've taken the ones that seemed most workable and turned them into a todo-list for the PloneSoftwareCenter.

To deliver these ideas, we need volunteers - Archetypes programmers, documentation writers, product reviewers and project manager types.

If you want to help out, please check out the OpenPlans collaboration site - add your name and thoughts to the wiki, join the mailing list, discuss, commit code!

See: http://www.openplans.org/projects/plone-products

The mile high stack

2006-10-06T12:26:00.000+01:00

Since getting back into doing some Java/J2EE web development, I'm constantly struck by the sheer number of mega-frameworks that I feel I probably should know. It's overwhelming, it gives me a nervous twitch, like I could be investing lots of time in some framework that turns out to be inferior because I have no way of comprehending what else is out there.

How do Java web app developers ever get anything done?

Here's a list of frameworks and APIs that I've come across in my current project, all of which could have some relevance to my project:

Java Servlet API
JSP syntax
JSTL
JSF
MyFaces/Tomahawk JSF components
Oracle ADF JSF components
Java Facelets
Hibernate
iBATIS
EJB 2
EJB 3
JBoss SEAM
Spring
Struts
WebWork
SiteMesh
Tapestry
AppFuse
Log4J
JUnit
JMock
JDBC
DBUnit
Cactus
Maven
Cargo
Ant
Tomcat
JBoss
Eclipse
MyEclipse

Oh ... and I need to determine which ones of these can work together, and which of my tools will tolerate which technologies. Aaaaah!

To be clear, my application is basically just forms that drive SQL queries. I don't even care so much about security. However, a goal was to try and use best-practice patterns and technologies, to ensure the application could scale up in the future.

Now, I'm pretty good developer, I know Java (the language) well and I learn quickly. But several of the pieces above would take me a week or more (each) to learn properly. Obviously some are easier than others, and many are directly competing. But how do I know what to choose? How do I know where to invest my time?

I looked for books, but it seems that many of them are out of date by the time they hit the printers. In the blogsphere, there are people who seem to be able to pick and choose between all of these. Power to them, but there is not a great consensus on what is best this week.

If there is some canonical book that covers an end-to-end process with best practices - please, I'd like to know. :)

But I started thinking about how this works in the Plone community. Im sure new users feel a bit overwhelemed, and that we could do things to make it easier for them. However, based on own experiences from when I first played with Plone, it wasn't nearly so bad:

Plone is there, you can touch it and feel it and play with things to see how they break.
Python is terse and readable.
Although Java kicks our ass on API docs, it's easier to follow the actual code since the .py files are all there and legible.
I learn best by example. Like most people, I started with a few copies of some page templates and later some Archetypes classes. That meant I could get results very quickly.
Plone let me worry about my application and not think about security, user management, URL dispatch etc.

Of course, I'm comparing apples to oranges here. Plone is a CMS and a platform for CMS-like systems. J2EE is much wider in scope. However, my application is not so unique - I need forms, page flow logic and database queries. Maybe one day some CRUD logic, and a few pages of text.

Obviously, this is all in the terriotory of that infamous Better Web App video, and the cliche exodus of Java coders to Ruby-on-Rails. Fair enough, but I'd like to think that it's not impossible to be productive and feel in control of a Java-based solution. This is often a question of skills (people learn Java in school), of prior investment (people have been trained in Java already), of back-end systems (that already run in Java), and of trust (for business-critical components that may feasibly need a web front-end). I refuse to believe that just because Java is strongly typed and compiled, this can't be done right. So most likely I just haven't discovered that framework yet.

However, there is another argument - Java is great for the middle tier, but less productive for the front end. Perhaps we should put more time into finding a bearable way of defining service layers that would let you build your user management and GUI in Plone or Django or Rails and let actual business processes take place in Java beans or EJBs. At least I think that separation would make a lot of decision makers more comfortable - Hibernate, EJB, Oracle (or Postgres) on the back-end, sexy GUI app on the front end that is infinitely more dispsable and lower risk, and is more likely to be what users want if it is written in a framework that encourages agile and rapid development. It may even help to ensure that the middle/back-end business objects are completely decoupled from the GUI and facilitate SOA.

Pipe dreams. Anyone got a good Java book to recommend?

Ignorance and Java Server Faces

2006-10-04T10:55:00.000+01:00

I'm not quite sure why it's called Java Server Faces - perhaps beacuse it makes my server a two-faced liar - but they're here. Specifically, we're using JSF with Tomcat and Apache MyFaces and whatever versions of the JSF API that Maven decided to pull down today (I think we're on 1.1).

Now, I don't have the time to read a few books on the subject, so I'm feeling a bit ignorant and inadequate. That feeling, however, is amplified many-fold by the fact that I am unable to step through most of the code that's not behaving the way I expected. Perhaps I could get there by downloading big tarballs and configuring source attachments in Eclipse, but I worry this may be a big black hole and I'm reluctant. The balance of JAR files being managed in my IDE and deployed in various cases is already quite precarious, so I don't want to upset it.

The second thing I miss desperately is pdb's (the python debugger) interactive prompt. Some things are definitely not working the way I expect. I can get some way with the Eclipse debugger and watch expressions, but it's cumbersome and slow.

Thirdly, my productivity is hampered by the constant re-compiles. Every time I change something, even a template, I'm recompiling and re-deploying to my local Tomcat instance. It's possible that I could find a way to edit templates in-place in the exploded .war file in Tomcat, or hot-replace them at least, but the only way I knew to do that was with MyEclipse and it doesn't like my Java Facelets templates.

This is not made better by the fact that we have three dependent projects. I have to a dance in Eclipse and Maven that involves 'mvn clean install' on a dependent project, cleaning code in Eclipse, and then re-deploying the project. I just lost nearly an hour pulling my hair out writing banal tests when it turned out that Tomcat wasn't executing the code that was being shown in my debugger. Meanwhile, this blog post has formed whilst I'm waiting for things to re-compile. Each time, my distraction period grows a bit longer, and my productivity dips a bit further.

Lastly (for now) there is way, way too much magic going on with JSF components. As far as I can tell, every form button is actually managed by a generated snippet of inline JavaScript. This (or something else) is managing some state that persists across Tomcat restarts, which sometimes is persisting the component tree (i.e. the tree of JSF components, which is formed by XML tags in various nanespaces, interleaved in non-obvious ways with the DOM of page itself). Sometimes, I get bizarre errors because some remnant of the page from half an hour ago is still being processed. Sometimes, not reliably reproduceable. The only way I've found to reliably test my code is to close down my browser completely and then re-open it after each re-compile/server re-start. Joy.

Further, sometimes the state appears to be saved but it's not - when I click a certain button which has a client-side handler (which is hooked up to a method on a bean by way of some magic expressions) but which aims to stay at the current page after processing, the UI shows form controls with values still entered, but they are magically lost when I click another form button causing an empty form to be submitted. Oh - and sometimes the first command link (think an <a> tag, except it's actually handled with some inline JavaScript on an onclick event and bound into the JSF dispatch machinery) that I click immediately after the application is started leads to a blank page being rendered (something about a phase of the component tree rendering not being ready, I read somewhere), and I have to re-load it to see anything.

I'm sure I'm just ignorant and that there are better ways of going about all of this, but I've spent as much time as I can afford with books and Google, and now I just need to get some work done. I'm patching together working practices that feel inferior and clumsy, but basically work. Luckily, this application is small, so there's not that much to do. But it doesn't feel particularly elegant. (/me wants a view and a few adapters)

Please refrain from pointing and laughing, but if you have some useful tips, I'm all ears.

Programming aesthetics

2006-09-25T16:19:00.000+01:00

I really should know better than to post this. Really. Please, don't start a big flame war. But, I peer over the fence from time to time skim through some tutorials and demos on Ruby and Ruby-on-Rails in particular, and one of the things that puts me off it (aside from the sunk cost of my Python knowledge and the fact that my favourite open source CMS project uses Python), is that ... it's so ... ugly.

I have an in-built aversion to blocks that end with an end keyword. Call it prejudice. I once learnt Visual Basic. Maybe that's it, but there are also lots of strange symbols flying about. $`? < for inherits-from? Functions without braces (oh, that's just crazy...).

If someone has seen the beauty in Ruby, please enlighten me. I genuinely believe that code asthetics are important in motivating programmers, ensuring consistency, catching mistakes and ultimately producing better software.

Of course this is totally subjective. But some good examplse of elegance and aesthetics would be nice.

Tool or language

2006-09-24T18:12:00.000+01:00

One of the interesting fallouts from the recent discussions about Nuxeo moving CPS over to Java in the future, is whether relying on tools like IDEs is legitimate, and indeed whether this is just excuse-making for the deficiencies of a language.

I'd argue that it doesn't matter. Whether you call it a language or a tool is largely moot. We don't edit Python in ed, we use vim or Emacs or TextMate (sensible people use the latter). Some languages are complemented well by tools, others have less need for them, and some just don't have very good tools available. The important metric is how efficient you are within your working environment, taken as a whole to include your language, your framework and your full toolchain.

Therefore, the learning curve = the language + the tool. If getting the tools set up is going to take you a long time, you lose productivity. Eclipse is fairly easy going. Some of the plug-ins are a pain to get and install, others are a breeze. But move down one level, and it can become a pain to manage builds with Maven (which is very powerful but can also be very complex when you step outside the box a little) or Ant (which is not that complicated, but requires a fair bit of manual work). I know companies that invest two weeks of a team's time just to get the build environment set up and configured right, which surely is wrong.

And secondly - when the tool does the wrong thing, you probably still need to understand what's going on. That complexity that the tool was hiding is rarely hidden completely, forever, because at some point your tool doesn't yet support the thing you need to do. And in many cases, the tool then expects to manage your source code in its own way and either stomps on your changes or breaks utterly.

Good tool design, it seems, is not much easier than good language design. There are still some things I miss in my current, very minimal Python set-up (consisting of Python, Terminal.app, TextMate, Firefox, FireBug), such as code-completion and support for refactoring by finding and modifying code that needs changing because I renamed a function or moved it up or down the inheritance hierachy. But in most cases, Python is high-level enough that the tools I need are in the language itself, saving me from having to learn a whole new tool.

And in the meantime, I just love doing this:

$ cd Products
$ mate CMFPlone CMFCore CMFDefault ATContentTypes

This opens TextMate with all those directories in a project, and I can grep (Cmd+Shift+F), edit, read, svn (Ctrl+Shift+A), add and remove files and do everything else I need. The alternative is to start Eclipse, find the right workspace, and add the particular folder to a project (or open an existing one). Thanks, but by the time Eclipse has loaded, I'm already onto my next task. :)

Nuxeo 5 - J2EE returns

2006-09-21T18:07:00.000+01:00

I'm going to argue out of ignorance. I am not nearly as close to the Nuxeo guys as I am to many other Zope people, and I don't have very much direct experience with CPS at all. However, I want to speculate a bit about the recent announcement that Nuxeo 5 (which is basically the evolution from CPS 4) will be 100% J2EE, in light of some realities that are abundantly clear at my existing employer.

CPS always felt a bit reclusive in the Zope world. Florent, Leonnart, Jean-Marc and Tariq (I probably spelled half of those wrong) were visible in the Zope 2, CMF and Five spheres, and I really like them and their work. However, CPS itself was (is) one of things that we often thought, "oh, it's really powerful and good" but never really used. Maybe it's because I don't speak French. Maybe it's because I'm addicted to the Plone community. But CPS was (is) Nuxeo's baby and business model. Which is perfectly, perfectly fine (and it's still fully open source, as will Nuxeo 5 be). Florent and Leonnart in particular contributed tons to CMF, Five and other parts of the stack. But it's these contributions we'll miss (presuming they stop, which I hope they don't), not the CPS-specific Zope technology that we never much re-used (or got around to looking at, perhaps).

I've been doing some J2EE work lately. There are things that are quite good about it, and there are things that are not. The Nuxeo 5 FAQ says that Java can be pleasurable with the right tool support. Maybe, but I can count on all my hands and toes the number of hours I've lost to trying to get the build system to play ball, configuring various tools and plug-ins, and debugging strange errors in non-readable code. I'm sure it'd be much better if I worked with Java day in, day out, got tons of training and had it all laid out for me by a standard build that some soul had spent months setting up and testing. At my employer, that'll never be the reality.

I have some particular beef with JSF (Java Server Faces). It may be the best Java's got for web UI, but it's still painfully slow and often unpredictable (I see some component tree state caches that don't invalidate properly, for example). It does not play well with JSP at all. Java Facelets help, and even have METAL-style macros, but don't even think of showing a complex JSF-based UI to a web designer. It's hard to know where the line between HTML and JSF components go, and JSF components still involve writing HTML to an output stream with Java code (they say they're working on it). I'm sure I just don't understand it, but I have seen supposedly cutting-edge JSF work (in Alfresco), and it's still awkard (single templates having a myriad of responsibilities).

The reason that I don't understand it because the stack is a mile high. JSF builds on JSP builds on servlets builds on streams builds on ... EJBs are just scary. Hibernate is quite good, actually, but I found it hard to debug. I'm quite quick at picking up new technologies, but the amount of technologies I have to understand to be productive is so large. I really wanted to use Spring, but that was another book to read. Again, if I did it day in day out and could affort to take a month on a training course or two, I'd probably do fine, but it's not the reality.

Will Nuxeo's programmers be happier with Java? Perhaps so, it may fit their brains well. They may find that it pays back in terms of stability, predictability and scalability (I'm somewhat sceptical of those arguments). They will certainly find it easier to hire new, talented developers.

But I can't help suspect that this has a lot to do with commercial realities. Alfresco has created a lot of noise off a product that's very, very impressive on paper and reasonably impressive on screen. It doesn't have anywhere near Plone's customisation/application-building story, for example. But if you read the spec list, it includes things like JSR170 and JSR168. Java standards (supposedly) for content storage and portlets, respectively. It talks to Lucene and Hibernate and MyFaces and MySQL, it uses Spring. These are things that are often singled out as "best-of-breed" open source components.

A digression... the Open Source in Java Enterprise world is bit of a parallel universe to the one we know in Zope and Plone. Different kinds of people hang out there. Investment comes from heavy-weights and Sun specification working groups. Apache/Jakarta are kings (and they have a lot of very, very impressive software)... end digession...

But Gartner goes out and says, "demand JSR170 to avoid lock-in". The ZODB makes IT managers quiver (call it an object store, not a database, Matt Hammilton says). Butler says, "demand JSR168 to integrate your applications with your portal server". IBM tell you, maybe it'll even work. We have viewlets and component architectures that make those things seem like last year's fashion, but that isn't what people buying IT look for. And probably for good reason. They shouldn't invest their money in unproven solutions, hard-to-integrate platforms or dying products. It's often a lot of money on the table, and being conservative is often a good thing (whether they always get what the bought is another story).

So, for Nuxeo, I'm sure that their clients will be very much more impressed if the 10 minute PowerPoint sales pitch includes a number of technologies they have heard about. It makes a lot of sense, especially since their stated target is the ECM sphere, and I'm guessing their customers have a lot of similar Java technologies already deployed. And I am really glad that there will be a credible alternative to Alfresco in the Java-based ECM/DMS market.

As far as Plone is concerned, I think our edge will always be in flexibility and agility. I can do things with Plone much faster than I can with any other platform, and I can develop cleaner, better-tested components using Zope 3 technologies than any other platform I've ever seen. At the same time, people like Enfold are doing incredible things with integration, and Plone 3.0 should put us back on the innovation curve with exciting new tools and technologies. Mostly, I'm not worried because the Plone community is so vibrant, with so many cool people and such a lot of friendship. In open source, this is worth more than technology.

But at the same time, I'm very eager to find out what Nuxeo's very talented developers will manage to come up with. Hopefully, I'll have a chance to sit down with them over a drink and get their experiences straight from the horse's mouth. Nuxeo.com/org have a commercial story to tell. I wish them the best of luck.

Five small TTW debugging tricks

2006-09-11T08:41:00.000+01:00

Sometimes doing debugging through-the-web is unavoidable. Templates are a prime example - I just spent the better part of two hours trying to work out why my use of metal:macro was not playing ball with a Five browser view (hint: understand what macro you're actually calling...). Sometimes I also simply don't know enoug about how something works to write a reliable and more easily debuggable test, and some trial-and-error is needed.

A few things can make it more bearable, though:

Customise default_error_message. That is, go to portal_skins/plone_templates in the ZMI and customise this template. Remove the call to metal:use-macro="here/main_template/macros/main".

Why? Because rendering Plone's full 404 template takes a few seconds on my machine. For some reason, this template has always been dog slow. For debugging, all you care about is the error message. By not referencing main_template, you get a plain-text error message, which renders instantly.

While you're at it, change <dd translate="" content="structure err_value"> to >dd translate="" content="err_value">, removing the structure keyword. This gives you a plain-text error message, which will not hide thing slike in when the browser things its HTML.
Add an external method with a pdb break point. e.g. I have Extensions/debug.py, which contains only:

def debug(self):
import pdb; pdb.set_trace()

Then, either in the portal_skins/custom folder or just the root of the Plone site, add an External Method in the ZMI, with name debug-ex, module debug, method debug.

With this in place, you can go to http://localhost:8080/Plone/foo/debug-ex. Make sure you run Zope in the foreground (zopectl fg). Your browser will hang, but on the console that you launched Zope, you will get a pdb session, where self refers to the current context (i.e. foo in the URL above).

Here, you can do whatever you want, such as inspect the object or mutate it. Remember, you can import other code, call protected methods - do pretty much anything you can do on the interpreter shell. Type pp to pretty-print a variable or an expression. Type c and press Enter to get your browser back.
If you're using a Five browser view with a template (either via the template argument to the ZCML directive or via an explicitly invoked ViewPageTemplateFile/ZopeTwoPageTemplateFile, remember that this does not have security assertions the way .pt files in portal_skins does. That is, you can do things like:

tal:content="python:foo._secretVariable.__of__(context).aq_chain"

So long as you're in debug mode, the you can reload the page and see the output of that statement. This is quite useful for trying things out and quick-inspecting variables.
Be ruthless with pdb. Sometimes you are wondering how something works deep inside Zope. So long as you revert your changes afterwards, you can put a pdb statement almost anywhere. For example, whilst debugging a security problem, enable verbose-security in zope.conf and try to put a pdb statement in AccessControl.ImplPython.validate(). You'll have to restart Zope obviously for this to take effect...
Interleave your work. Sometimes, you need to restart Zope. This is slow. :) Learn to do two things at once, e.g. work on improving tests for other parts of the code, writing new interfaces or updating documentation while Zope is restarting, or go help someone with their problems on #plone on IRC :)

Category greed

2006-09-06T10:09:00.000+01:00

I think it's really, really great that people contribute documentation and third-party products to Plone and publish them on plone.org. However, as the reviewer of a lot of the submissions, I have one complaint:

Selecting every conceivable category, audience or section makes your content harder to find, not easier.

80% of the items I review have selected four or five audiences (Oh, you know, this is relevant to site administrators, developers, advanced developers and integrators, customisers, maybe), or two or three product categories (hey, there's code here, so let's call it a code example, and also it could go into the misc category, and really, it's a service so let's put it there too).

Unfortunately, that just makes the categories less useful to everyone.

This is why we have reviewers, but reviewers forget to check the categories (or they make the same mistake) and we end up with cruft. We could change these to a single-select field, but there are cases where we need multiple categories. Perhaps we could introduce a validator that means only reviewers can select more than one category, but this is somewhat clunky.

So please - apply some critical judgement. The categories are there to make it easier to browse the content, as the listings are grouped by them. If every grouping has every item, then we may as well just not have categories.

Maybe we're just not that kind of people

2006-09-04T10:03:00.000+01:00

I always enjoy reading Carlos de la Guardia's posts on the stature of Zope and Zope 3 in the Python community. His top ten list hits it on the head, I think.

One of the big, big criticism against Zope 3 is that it has no website from which it tries to create buzz. I'm not going to repeat that argument, but look at something like the competition and you get the idea.

Then again, Zope has always felt a bit inwardly focused. I don't think its developers feel the need to be validated so strongly by having an influx of people knocking down the doors. They are focused on making the perfect framework, and Zope 3 is in many ways lightyears ahead of the competition. They are people who run small businesses or work for places like Zope Corp or Infrae. They are solving their customer's needs, refactoring ruthlessly and building something they believe in - a framework they love to use, that makes their lives easier (we hope).

But they are not website building people. There is a project going on to build a new webiste, and kudos to people like Martijn Fassen and Philipp von Weitershausen for shepherding this along, but it's been months and I'm guessing it will still be months. We're defining IA and web design still. As far as I know, there's not even a consensus on which technology platform to run it on (I have my pragmatic preferences, of course, but there are politics to be navigated there as well).

In a way, it's a bit of a Catch 22. I think you need a certain type of semi-peripheral enthusiast to build kick-ass websites. Hardcore coders are rarely good web designers or even good at defining information architecture and site structure. More impotantly, they rarely burn to do it, they'd rather be making zope.app a less hostile environment and give us a decent local site customisation story (thank you!).

But the kind of bright young things looking to make a name for themselves, who are happy to be drawn into a holy war over their web framework and throw insults at all those who dear imply it's anything but perfect - the perfect guys to make the advocacy website (with some moderation) - are just not Zope 3 users (at least there aren't many of them). For better or for worse...

Maybe they would be if there was a better website. But I think it's also true that Zope 3 targets a more "serious" end of the market, with a steeper learning curve, but with more power and elegance. Whomever may read this and think I'm slagging off the people who build websites for Django or TurboGears or Rails - wait with hitting that comment link. I think you guys are doing a great job, and the bit of my heart that's sentimental towards Zope envies you. I just think there's a different type of community behind Zope 3 - unfortunately one that has many fewer people truly interested in doing this kind of work.

The other part of the problem is governance. Zope 3 lacks a strong leadership figure to put out edicts about this kind of thing. We have strong leadership figures in the shape of Jim, Stephan, Martijn and Philipp - but these are not people interested in defining how Zope 3 should be marketed, and Jim (who's probably least interested, which is no criticism at all) is probably the only one who has enough legitimacy to be able to say "this is what we must do". Well, Martijn and Philipp are trying, but I'm sure they feel it's a bit of a slow, uphill struggle to gain enough buy-in from the rest of the community to get something done quickly. It is done by committee and consensus, like most things in Zope 3 land, and I'm not quite sure marketing is a job best done that way. The promised land may be the Zope Foundation, but for someone not directly involved with its creation, I'm not seeing much of it.

Of course, I may be wrong, and I really don't wish to undermine the efforts of the people who are trying to rectify the situation (I've even offered to help with the technologies I know well that I think could help).

But, here's a thought - perhaps, the people who care and who would benefit indirectly from Zope 3 gaining more recognition in the wild in terms of more business could pool some resources and pay someone to build the right kind of site?

Summer of Code

2006-09-01T10:45:00.000+01:00

It's been a couple of weeks since the formal end of Google's Summer of Code programme where the Plone Foundation were represented with three students. On the whole, it has been a success, though there are lessons to be learnt.

Right up front, I'd like to say - Thank you Google! It's not every day an organisation spends $3m on handouts to the open source world. Plone certainly will feel the benefits of the work that have come out of this for years to come.

First - the successes. Hanno Schlichting, mentored by Alec Mitchell, has done some great work on our i18n architecture, that benefits not only Plone, but the CMF and Zope (3) communities as well, and produced a bunch of collateral benefits, such as a GenericSetup syntax for local site setup in Zope 3. I don't think anyone really doubted that Hanno would kick ass, being one of our more consciencious and consistently solid contributors, but it's great to see him rewarded and motivated by the programme. Hanno blogged about this progress here.

Markus Fuhrer, mentored by me, also made some giant strides. He was working on PLIP 157 - Content Rules Engine. To be fair to Markus, he was thrown in rather at the deep end, building a "pure" Zope 3 package at the base to model the use case and a first cut at the Plone UI for it. Unlike Hanno, Markus had no previous Plone experience, and much of this was new to his mentor as well. Moreover, the task was to build a very open-ended framework, taking as much complexity as possible away from people wanting to build new "rule elements" (actions that you can string together to form a rule that is executed upon some event) and plug them into the system. That necessarily put a lot of the complexity in the framework itself.

Being an old friend and living in London, we were able to pair-program (with plenty of Falafel and Diet Coke) about once a week. The biggest victory in my eyes is that Markus now has a good understanding of how this architecture works, and it is my sincere hope that he will be able to stay involved with its maintenance and further development. PLIP 157 is currently under review for Plone 3.0. You can check out the review bundle for yourself. I also hope that the plone.contentrules package (which has no Zope 2 or Plone dependencies, and thus should work in pure Zope 3 applications as well) will be of use to the wider Zope community.

The last student we took on was less forunate. He was tasked with PLIP118, refactoring PlonePortlets into a more general and modern (as in, less Archetypes-dependent) solution for managing portlets in Plone. Unfortunately, we never saw any code. I worry that perhaps we scared him away (his task of was similar complexity to Markus'), and he eventually broke communication with us, without producing any code.

In the aftermath, I picked up this work with a view to get it ready for the Plone 3.0 review deadline. A huge thank-you to Philipp von Weitershausen for helping me with many of the Zope 3 details, and to Geir, Dorneles and Helge from PloneSolutions and Christof Hammerlie (who was working with us at the Archipelago Sprint), who not only made the original PlonePortlets but helped with use cases and design. You can check out the review bundle for PLIP 118 if you are interested in how this works. Similarly to plone.contentrules, I'm hopeful that plone.portlets can be useful to the Zope 3 community as a standalone component.

The lessons we learnt

There are many things to learn from the Summer of Code experience, especially if Plone wishes to participate in the programme should Google run it again. Here area few:

Plan early! In the days leading up to the deadline for proposals, there was some chaos. I believe we benefit both ourselves and our students the most by having clear proposals with agreed scope for students to pick, and by having well-prepared potential mentors. Internal debates about what is a reasonable scope for an SoC project and what potential projects are more important than other ones should probably happen before applications have to be rated in earnest.
Don't expect too much! We were initially hoping to get about 10 students. That was completely unrealistic. There are many organisations and a few meta-organisations like Apache and GNU who (deservedly) get many more students. Reputation, previous year's experience and relevance to the world at large seemed to be important factors for Google to determine the allocation. And yes, we are smaller than Drupal, say - we have to adjust our expectations accordingly.
Evaluate proposals carefully! It's very hard to evaluate a student based on the short text they submit. The potential mentors pushing for a particular project may also be blinded somewhat by the desire to see that work done.

Our three highest rated students ended up being one safe bet (Hanno), one student I could vouch for (Markus) and one unknown that we were hoping to get involved in the community. I personally think that was a decent balance. Hanno deserves recognition for the great work he does, Markus ended up producing something very solid; and getting fresh talent involved is an explicit and important goal, even if there is more risk taking on an unknown.

Unfortunately, there were other students marginally below the line (like Tim Hicks, who got as many points as the third student and seemed a victim of chance for being excluded). In fairness to Tim, he probably would have been a safer bet, having already contributed to Plone by way of Quills, Ploneboard and other components. Balancing the risks and benefits of betting on an unknown is important, especially when the number of allocated students is low. Having clear, honest and selfless discussions among the evaluators on the students in question is paramount.
Communicate! One of the problems with the portlets work that got canned was that we never formally kick-started the student. We offered to help, but expected the student to pull more than we'd push. In this case, that didn't happen.

The student ran into trouble (such as trying to install Plone 2.5 on Zope 3.0) that we could have avoided if we were more directly aware of what he was doing. I take some self-criticism for being qutie tough on the student in this case, out of frustration that he had spent a lot of time doing something we could have told him would be a dead end - all because we were out of the loop. Unfortunately, the pattern repeated itself with subsequent problems, and communication did not noticably improve.

Of course, the communication loop has two ends that need to stay active. Mentors are busy, but have a responsibility towards the student. Different students may need different amount of time invested in them. Furthermore, not every student will be as used to the constant email and IRC communication that we all take for granted. We need to be understanding of this, and help the student get over the initial bumps. We need to be helpful and friendly, and have realistic expectations of how much of a learning curve the student has to climb (and not just in terms of the work, but also just the experience of being part of an open source community).

Equally, students have a responsibility to be the more active party. They are being paid a not-insignificant amount of money to do a job (mentors are not paid, mentoring organisations are paid $500 per student, compared to the $4,500 each student receives). It is not something to be taken for granted, and it is expected that the student should be as many hours into this as they would any other summer job. And this is hard - a student (Markus excepted) doesn't have a boss looking over their shoulder when they are blogging instead of working. They have to take significant responsibility for their own planning and be self-managing. They have to be able to ask for help when they are stuck. They have to be able to manage expectations with their mentors and define the scope of what they believe they can acheive. Naturally, some candidates will be better at this than others.

On the whole, Summer of Code has been a very enjoyable experience, and I believe a very positive thing for Plone. We are grateful to Google, and grateful to our students and mentors. Hopefully, we'll get to do it all again next year :)

My new favourite place to put a pdb

2006-08-31T00:39:00.000+01:00

I'm not sure if this is a good thing. In fact, I'm pretty certain it's a bad thing. But sometimes, using Five views and acqusition hurts. The main problem is that Zope 2 (still) depends on acqusition for security, and our code tends to rely on implicit acquistion (acquiring attributes, tools, templates from portal_skins...) to stitch our application together. Zope 3, on the other hand, doesn't have this kind of implicit acquistion-is-everywhere magic. Which is probably a good thing, but it does make for some awkward marriage of the two in Five sometimes.

The basic pattern is, you mix Acquistion.Explicit into your view classes and other objects (or if you have to, Acquistion.Implicit, or sometimes OFS.SimpleItem.SimpleItem... I'm still trying to work out the hard and fast rules, and I'll tell you when I can). You then sometimes have to do some aq-wrapping as well. obj.__of__(aq_inner(self.context)) seems to be a favourite. And quite often you have to do some aq-unwrapping - aq_base() and aq_inner() are your friends.

But it's often quite hard to know where the acquistion chains are coming from, and what they are at various stages of traversal. Today, for example, I discovered that if you have a Zope 3 style container (something that provides IItemMapping) traversal will be handled by zope.app.container.traversal.ItemTraverser. This looks like this:

def publishTraverse(self, request, name):
"""See zope.publisher.interfaces.IPublishTraverse"""
try:
return self.context[name]
except KeyError:
view = queryMultiAdapter((self.context, request), name=name)
if view is not None:
return view

raise NotFound(self.context, name, request)

Great. It looks for the given name that you're traversing to in the container, and returns it if it can, otherwise it tries to look it up as a view, which is what you'd expect.

The problem, though, is that this view never gets acqusition-wrapped. Thus, the acquisition chain is broken at the view, and it has no appropriate security context, for example. This was giving me incredibly non-debuggable security errors.

After some helpful suggestions from Zope 3 + Five guru Philipp von Weitershausen, I decided to put a pdb break point in ZPublisher/BaseRequest.py, specifically the traverseName() function. (This is Zope 2.10, btw, it's different in earlier versions). This gets passed the name ('name') that is being traversed to, and the object ('ob') that is "as far as I've gotten so far". Thus, let's say you're traversing to /plone-site/container/@@myview - it will start out with ob as the Application and name = 'plone-site', then ob as the Plone site and name='container' and so on.

Of course, this is a bit annoying, because if it does render a page, it will traverse to tons of resources like images and stylesheets. Therefore, it may be better to put a conditional break point in there, by doing something like:

if 'myview' in name:
import pdb; pdb.set_trace()

It was here I discovered that the view being looked up was not automatically acquistion-wrapped. My custom traverer for my own type looks like this - and it works!

class PortletAssignmentMappingTraverser(ItemTraverser):
"""A traverser for portlet assignment mappings, that is acqusition-aware
"""
implements(IBrowserPublisher)
adapts(IPortletAssignmentMapping, IBrowserRequest)

def publishTraverse(self, request, name):
ob = ItemTraverser.publishTraverse(self, request, name)
return ob.__of__(self.context)

Now - Philipp has been waving at his plan for world domination - make acquisition less important in cases like this. Dude - we can't wait! :)

In the meantime, I hope we can work out the common patterns and document them, at least.

More lame files - init.py and browser.py

2006-08-29T16:56:00.000+01:00

Since having a blog gives me a license to rant...

Don't put code in __init__.py files. Imports, sys.modules mangling, deprecation warnings - all fine. But real functionality? Blah... I never look at __init__.py files and I wouldn't expect anyone else to do it in my own code. In 994 years they may even become less important.

Secondly - and relatedly, think ahead. If you have only one view now but you may need ten in a week's time, don't create a browser.py for the 'browser' module (which, by Zope 3 convention, is where your browser views and resources tend to live). First of all, you may need a template, which lives best in browser/template.pt. Secondly, most applications or products have several views, and having tons of classes defined in one ever-growing browser.py is a pain. At some point, you'll refactor them into sub-files and get into an import jungle.

Of course, if you're creating something that by definition should have only one view (i.e. a piece of infrastructure), then great - dont' clutter it up with unnecessary directories. But if you're building something at the Plone level, like a set of content types, they'll probably have more than one view, and creating a directory isn't going to kill you.

Oh - and thirdly.... we shouldn't have called the file for the @@plone view plone.py. Why? Because we now want to have namespaces like plone.portlets. That leads to things like this:

IPortletManager = sys.modules['plone.portlets.interfaces'].IPortletManager
IPortletManagerRenderer = sys.modules['plone.portlets.interfaces'].IPortletManagerRenderer

Can you figure out why this wouldn't work?

from plone.portlets.interfaces import IPortletManager

Python 2.5 is rumoured to have relative imports ala ZCML which may make this better, but it won't solve all problems.

Ah, I feel all better now.

adapters.py - a sure sign that you're not using Zope 3 enough

2006-08-29T12:37:00.000+01:00

edit: heh... Freudean slip - I originally titled this 'annotations.py - ....'

There is a kind of worrying (well, perhaps not so worrying, but it does annoy me) precedent being set in a few places:

adapters.py

In Zope 3, adapters are everywhere. Views are adapters (multi-adapters, in fact). Namespace traveral is done with adapters. Regular traversal is done with adapters. And well-designed Zope 3 code tends to use adapters a lot:

One interface for each logical component or piece of functionality
One or more adapters (or utilities) providing the functionality of a particular interface

Of course, some "base" objects, like content objects, are not looked up as adapters and provide some interface directly, but they tend to be the context of adapter lookups for other interfaces.

Usually, though, when something is context-dependent, it tends to be an adapter. You adapt a content object to IAnnotations via an annotations-adapter to be able to annotate it. You adapt an object to ISized to work out its size. You adapt an object to ISlapTheMonkey to slap the monkey attached to this particular context.

... so why would you put all of them in a single file?

That some functionality is provided by an adapter should be a consequence of the design. Logically related functionality should live in the same module (file), and each module should have a clear boundary of responsibility.

I guess my point is - adapters are not special. They are just a way to modularise your code and separate concerns. If they excite you so much that you want to keep them on their own little pedestal in adapters.py, that's great (you've been drinking the kool-aid), but if you keep using them, they'll just become a natural part of your toolset and you'll use them all over the place, where they make sense.

Packages, packages, packages

2006-08-24T16:09:00.000+01:00

It seems I'm being reptititve in my blog topics, but in case you don't read the Zope 3 list:

Zope Corporation has just released a whole slew of packages for Zope 3, many of which could be very useful to Plone. For example:

zc.index -Zope 3 style indexes for RTF, MS Word, HTML etc. (wow...)
zc.preview - In-browser previews for various file formats
zc.notification - Framework for doing e.g. email notifications
zc.selenium - The Zope 3 equivalent of PloneSelenium
zc.vault - A subversion-style repository stored in the ZODB (wow again)
zc.winauth - Authentication against Windows domains
zope.html - FCKEditor Zope 3 widget
zc.set - Persistent Python set implementation

All to be found in svn.zope.org. :)

The things you find...

2006-08-22T10:44:00.000+01:00

I sometimes nip into svn.zope.org looking for a code example. I'm constantly amazed at what I find there. Here are today's discoveries (note that I'm only skimming READMEs at this point, so apologies if I get some of this wrong).

z3c.javascript - Zope 3 resource registrations and registrations with zc.resourcelibrary (if present) for Prototype, MochiKit and friends
z3c.traverser - Plug-in architecture for making custom traveral adapters. Joy! I believe lovely.tag, for example, uses this to get URLs like /tag/foo to work.
z3c.zalchemy - Integration with SQLAlchemy , a lovely RDBMS and ORM tool. Note that Kapil has been working on somehing called Alchemist that also provides some SQLAlchemy integration for Zope 2/3/Plone.
z3c.extfile - An external (large) file handling solution.

There are lots of other good packages in there as well - zc.table, zc.relationship, zc.catalog... Many of these resonate with things we either are doing or want to do in Plone 3.0 and beyond.

Therefore, it's probably wise to add svn.zope.org to the list of places to check before you start building something from scratch that may have been invented before. :)

Local adapters in Zope 2.10

2006-08-21T10:01:00.000+01:00

Further to Hanno's excitement about local components, I thought I'd add something of my own. As Hanno points out, systems like Plone where more than one site (with possibly different configurations) can be in the same Zope instance, and where much of the configuration needs to be done through-the-web, there is often a need for persistent, local components. In CMF, we have tools, which are persistent (in-ZODB) singletons (per site). Their analogue in the Zope 3 world are persistent (local) utilities.

In Zope 2.10 (and 3.3), Jim has refactored the component architecture to make it much, much easier to work with local components. Components are registered with a site manager, of which there is a global one (the deafult one use when you register components in ZCML) and any number of nested local ones attached to one or more sites. Hanno has made the Plone site root a site on the 3.0 bundle, which means that we can attach local components to it.

Imagine you have an object 'obj' that you want to register as a local utility providing an interface 'iface' with the given name 'name'. Here's how you'd register that as a utility on a particular site:

sm = getSiteManager(site)
sm.registerUtility(obj, iface, name)

Hanno has bundled this functionality into a GenericSetup export/import handler here, which means that setting up local utilities should be as simple as an XML file.

After this registration has been made, you can do this:

util = getUtility(iface, name=name)

and you should get back 'obj'.

That is - if you are in the right context. Upon traversal, Zope (and Five, in our case) will know the context and look up the containment hierarchy to find out where the nearest site manager is. If a site manager does not contain a given component registration, it will fall through to its parent, all the way up to the global site manager. Again, Hanno has taken care of this in Plone. In a test, you may need to do something like:

setHooks()
setSite(site)

To simulate what Zope may otherwise do during acquistion.

Now, in plone.portlets I have been doing something similar with local adapters, with no small amount of help from Philipp von Weitershausen. Now, to register a local adapter, you only need to call registerAdapter() on a site manager:

sm.registerAdapter(required=(iface1, iface2,),
provided=iface, name=name, factory=callable)

Here, 'callable' could be a class or a function that produces the adapter providing 'iface'. It will take two parameters, being passed the objects providing iface1 and iface2.

In plone.portlets, however, the pattern is a bit more interesting. The reason for needing a local adapter in the first place is to satisfy zope.contentprovider. This comes with a TALES expression type handler for the provider: expression type. If you write a page template:

tal:replace="structure provider:plone.leftcolumn"

zope.contentprovider will look up a multi-adapter providing IContentProvider from (context, request, view) and call its update() and render() methods.

In plone.portlets, we want to be able to register a content provider to render the left portlet column in place of this expression, which means that an appropriate adapter lookup must be found. Furthermore, the assignment of portlets to a context is a persistent, site-local concept. This implies that not only must the correct IContentProvider local adapter be located, it must also know which "portlet manager" to query for the list of portlets to render.

The solution to this problem is to let the adapter factory (the callable that produces the actual registration) be a persistent object. The implementation of PortletManager has a __call__() method which can produce the appropriate implementation of IContentProvider in the form of a PortletManagerRenderer. When instantiated, this is told which portlet manager it is rendering.

Thus, a portlet manager in /.portlets/left, for example, is registered (using a GenericSetup handler) as the adapter factory for the plone.leftcolumn adapter:

sm.registerAdapter(
required=(Interface, IBrowserRequest, IBrowserView,),
provided=IContentProvider, name='plone.leftcolumn',
factory=site['.portlets']['left'])

In effect, this achieves a link between the (persistent) adapter registration for the plone.leftcolumn content provider adapter and the (persistent) portlet manager storage.

Of course, users of the library shouldn't have to do any more than this in a GenericSetup profile:

Componentising the content menu

2006-08-18T00:52:00.000+01:00

I have an unstated personal vendetta for Plone 3.0 - make the UI more pluggable. There are a few places where people are forced to do very invasive template overrides because Plone doesn't provide any proper way for them to plug in the UI they want. This in turn makes us dependent upon having certain types of customisations in CMFPlone proper, rather than letting them be managed by add-on components. discouraging certain type of innovation in the wider community.

A big part of that will be viewlets - a way of composing snippets into a full page. Alec Mitchell did a ton of great work on making these work in Five, and I'm sure their impact will be felt in very positive way for Plone 3.0.

The other one is metadata - how many people would like to be able to apply a standard (perhaps required) field to every content type without having to sub-class every single one of them? At least for Archetypes, a solution is within reach, thanks to some great work by David 'whit' Morriss, which I adapted and used here. Daniel Nouri, the new Archetypes release manager, is interested in this as well, so with a bit of luck and work, we'll get that one done in time.

The last bit is a template with which I'm rather too familiar - global_contentmenu. This is the code that controls the green bar with the actions, display, add item and state menus.

Take a look at the code. This is a prime example of the stuff you should not do with ZPTs. There is more logic in there than in pretty much any other page template, it's brittle, it's completely untested, and it's hard to maintain. But more than that, if you want to add a new menu or change one of the existing ones, you have to override the entire template. LinguaPlone does this to get a translate menu. BernArticle does it, apparently, for who-knows-what reason. CompositePack apparently uses JavaScript DOM manipulation to get another item in there. Nice.

Now look at the bundle for PLIP142. This does a number of interesting things:

The menu is rendered as a content provider (the Zope 3 component that underpins viewlets) by means of a 'provider:' expression at the bottom of global_contentviews.
This invokes a view component that prepares a ZPT-friendly structure.
The rendering of this data structure is now sane. Compare this template with the old global_contentmenu.
All the logic is implemented in Zope 3 components. These can be tested! There are now 50-odd tests covering the logic that used to be in one template (scary...)

The components in this case use the Browser Menu infrastructure. This consists of one menu called plone.contentmenu which represents the horisontal bar. This has a number of menu sub-menu items representing each of the actions, display, add (aka factories) and workflow menus. Each of the sub-menu items in turn points to another menu that fetches the items to display in the drop-down. All of this is registered in ZCML.

Notice how the sub-menu items are implemented as adapters. This is because they need additional logic beyond what the standard Zope 3 subMenuItem ZCML directive can provide (and also because that directive doesn't seem to work in Five, at least it didn't in Zope 2.9). The implementations are found in menu.py.

It is possible to register your own menus or override existing menus simply by registering new components in any package. For example, you could:

Override the entire menu by providing a more specific adapter or an override for the plone.contentmenu content provider. See the bottom of configure.zcml and the adapts() specification for ContentMenuProvider - anything more specific than Interface could cause a different renderer altogether to be invoked.
Override a specific sub-menu item. Again, a more specific adapter registration could override the sub-menu items. The default items are simply registered for all content types, but you could choose to have a specific implementation of one of the menus for a specific content type.
Add a new sub-menu, either for all contexts or just those providing a given interface. Notice how the sub-menu item implementations all have an order attribute, and that they are ordered with gaps - 10, 20, 30, 40. Add a new sub-menu item and register it as providing IContentMenuItem with an order between any of those numbers, and you can control whether it appears before or after all the existing menus, or in-between two of them.

The work is not quite finished, and there will be a lot of testing to be done, but already the bundle functions for the majority of use cases. Specifically, what remains is:

Make the JavaScript warning on the Delete action work
Fix line breaks in the Display menu
Ensure the selected item in the Display menu is properly marked as such
Test with non-structural folders, folders with a single addable type, and with various combinations of permissions and roles to ensure the new menu behaves like the current one does.

We also need to decide whether existing customisations of global_contentmenu should be allowed to override the default or whether we break all of them. Alec nearly has me convinced that breaking existing customisations is the right thing to do, since any customised version would break any other product that used the "new style" customisation techniques outlined above, and since the existing template is just scary.

However, pending framework team review, I hope to see this in Plone 3.0. I'll miss having days like these trying to figure out how to make the Display menu work under various conditions with some serious TAL voodoo, though...

Lovely new packages

2006-08-16T11:24:00.000+01:00

The pink fluffy people at Lovely Systems have placed some new code in the Zope Subversion repsitory, right under our very noses. Thanks guys!

lovely.tag is a tagging system, which from my cursory look (at both packages) may be somewhat similar to tagger by Whit Morris of OpenPlans.
lovely.ratings is a ratings system for content. Again, there seems to be some overlap with Alec Mitchell's contentratings package.
lovely.remotetask is a framework for executing remote tasks, i.e. on a different host.

The first two, certainly, are aligned with things that we may well want to include in Plone 3.0 and beyond.

As we begin to realise the benefits of the Zope 3 re-use story, we need to be careful to examine what software already exists and how we may leverage it in Plone without having to re-invent too many wheels. Part of that may also be to look at how some of the existing packages that overlap could be consolidated, where this is practical.

Finally, another outlet

2006-08-15T16:02:00.000+01:00

After months of thinking about it, it's finally time to create another excellent source of procrastination. My friends in the Plone community always tell me I write too much in my posts to the mailing lists. Rob Miller and others have rightfully suggested that I may need a blog to vent. Well, here we go.

Expect this blog to contain more musings about Plone than anything else, but hopefully I will also write about my music, my job, and the rest of my life.

Martin