Trained Monkey Hacking Experience

Yes, we can!

2008-02-18T10:36:00.003+01:00

Starting to realize more and more how much different would be Obama's America from the Bushistan we know today.

Link

Happy birthday, Miriam!

2008-02-14T11:26:00.002+01:00

Happy birthday, Mirka!

Pleased

2008-01-15T12:38:00.000+01:00

Your servant is extremely pleased to see Erwin advertize a laptop that is running SUSE Linux Enterprise Desktop 10 and OpenOffice.org Novell Edition. Thanks, dudie!

Frankfurter Allgemeine hacking

2007-11-22T13:45:00.000+01:00

It was decided on Tuesday afternoon, that I should converge to Nuernberg to meet my favourite manager JPR and my dear colleague Florian, in order to complot about possible changes to Writer core in order to increase its interoperability with one competing product.

Since the decision was a little bit short notice, the only way to travel was the train. So, I took a night train from Geneva at 19:14 and passed Basel and Frankfurt Flughafen (sic) to arrive finally at 8:02 to Nuernberg.

What a wonderful journey of almost 13 hours of pure hacking pleasure. Especially that German ICE trains have electric plugs under every seat. Like that I was always able to recharge my laptop battery and use it during the long hours between trains. Especially, during the more then two hours in the empty Frankfurt Airport railways station.

An attentive reader is asking "why all this bla-bla?" And the answer is that thanks to Jaroslav Fojtik of wp2latex fame, thanks to his reverse-engineering skills, and also thanks to my dearest employer and to this long journey, I was able to touch 275 files and make the HEAD CVS libwpd support conversion of password-protected and encrypted documents for almost all supported file-formats (except WP6+ that we were not able to decrypt for the while).

I also forgot to mention, that since a week or so, the duo libwpd-writerperfect is not depending on any desktop specific library. Writerperfect can still use libgsf for the zip output if one insists by giving it a configure option --with-libgsf, but it produces nice ODTs and ODGs even without it. Thanks to Ariya's FemtoZip namely.

Happy Birthday, Darling !

2007-10-25T12:04:00.000+02:00

Happy third birthday, Patrick!

ITO, Sweet ITO

2007-10-18T13:21:00.001+02:00

I love this! I misunderstood a bit what the ITO was and at certain point I realized that I have accumulated quite a load of it. So, I had to spend it somehow. Here is the account of what it was useful for:

Finishing off what was started in Barcelona

Naturally, if you speak to me about fun hacking, I will always have the image of libwpd or libwpg coming to my little brain. As I blogged earlier, the first thing I used some time for was to finish off the work on tabulator conversion for WP 2.x - 3.5e for Mac and WP 5.x for DOS/Windows file-formats. It was started in Barcelona and the principle of release soon, release often dictated that it should not sleep on my disk for long time. I got even a very positive echo of the overwhelming part of our Mac OSX user community after the 0.8.12 release (Yes, Smokey, you are about 87.49% of our active Mac OSX community :-)). So, if you did not do it yet, throw yourselves on the binaries and sources while they are still available.

Buffered stream implementation

An old proverb says that one should try not to put off useful people. This was also one of the reasons why I deprecated in latest releases of libwpd our libgsf-based stream implementation. Understand me well, libgsf is a fine piece of software and is doing a lot of good to the world. Nevertheless, since libwpd is used by the Koffice people along with some other office suites, I did not want to keep it depending on a platform specific libraries. Moreover if the functionality that we use from these libraries is really very very very very tiny. So, we used a hacked version of Ariya's pole (one header and one source file) to provide two additional WPXInputStream implementations that do not depend on anything besides a working STL implementation.

So far, so good, but... As long as the C++ streams (as I call them) became the default for the internal libwpd tools, we came to the point where the above mentioned proverb was going to be put into task again. Suddenly, Sum1's QA run started to take several days instead of a night with the libgsf-based stream implementation. And putting off Sum1 is something that every hacker with some rudiments of sane mind should avoid at all cost.

In the same token, some other performance issues were reported from other contexts and the fact that libwpd reads at most 4 bytes in a single read started to show as a real problem. So, some time ago, inspired by Kendy's intent to implement read-ahead to WPXSvStream, I hacked something similar to the sample WPXFileStream. But, there was a problem. Very probably a hideous bug in libstdc++ is causing in some specific cases a seek backwards by 2 bytes with a consecutive read of 1kB result in a badbit being set. And this mainly on x86 with gcc 4.x (x86_64 was working well). Even x86 with gcc 3.4.x worked well as well as our distinguished competitor's Visual Studio compilers. Since I did not have much time down there to look into it, I left the idea of buffered read hanging.

Now, during my ITO, I gave it a quick second shot and by a sheer luck I found a workaround that is not really penalizing in terms of performance, and makes the buffered stream work well. The solution is simple: before the read, seek to the end of the block that we want to read and then back to the position where we want to start to read. With this incantation, the subsequent read does not set any badbit anymore and everything is nice in the best of the worlds. Nonetheless, if a libstc++ hacker is interested in investigating into this bug, I have a historical version of the stream with a sample document that always triggers the above mentioned behaviour.

Positioned objects in WP6+ and ODF reference implementation woes

The main task I was prepared to accomplish since our dear hackweek already, and waited only for an enough long contiguous free time, was to get the size, the position and the anchoring of the positioned objects (images, text boxes,...) right. The problem was in multiple places. First of all the way the box information is encoded in WP6+ files is really only remotely connected to an idea I have about a fun hacking. The box/frame style is an old good way binary encoded information (if the bit 15 is set, the following information is available... etc.). The code that is inserting the box/frame can override style and whether it does so or not is again given by some bits of a number at the begining of the code. Added to it an acquired mistrust towards the Corel provided documentations, I was not really hot to spend my week-ends on this. Nevertheless, some recent user demands and the vision of a well used ITO made me to bite the bullet. And surprisingly, the main difficulty was not where I expected it to be. This time the documentation was right (at least for the information that I currently use). This time the tool I trusted always the most, wplook.exe from the WordPerfect Office 11 SDK, got some things wrong and segfaulted on accessing the box/frame information of certain documents. But once understood where to put my trust, the information I needed was extracted and ready to be processed.

And it is there where the real problems started. On one hand the WP6+ file-format is much more expressive then ODF concerning the object position. Thinks like shifted one inch left from the center of the page text area is not mapping directly to anything in ODF. And given also the fact that in WordPerfect the page margins can be set different for each paragraph, the position is not as easy to compute as one could expect. Nevertheless, the biggest surprise came from the way the reference implementation of ODF handles page anchored objects that correspond to the second case of text:anchor-type="page" referenced at page 294 of this file as well as at the page 303 of this one. According the specifications, if one omits the text:anchor-page-number property, one would expect that the frame appears at the same page as the character immediately following the draw:frame closing element. And this is exactly what happens inside WordPerfect, so I was happy like a child. Premature happiness though. When our dearest reference implementation finds a page-anchored frame without text:anchor-page-number property, it assumes apparently that this is just a forgetful mind of the filter writer and adds a text:anchor-page-number="0" attribute. This invariably places all the page-anchored frames outside the specified range for page numbers (1..N) and thus outside the document (although they still remain in the file fortunately).

So, there are two possible workarounds that I managed to find. Although none of them is satisfactory for different reasons. The first consists in using the hints WordPerfect formatter leaves in the file for itself to improve the speed of rendering. One can use the soft page breaks and hard page breaks to count on which pages we are in the given moment positioned. It is a possible workaround, but does not do good job unless the documents are reasonably short. With long documents, font substitution and not completely correctly converted spaces between headers, footers and the text area, one will find oneself with frames drawn on completely unrelated pages. So, this solution has little of my sympathy. The second workaround is to use the text:anchor-type="char" with which the anchoring is also relative to the position of the first character that follows the draw:frame closing element. And it works well as a workaround. My principal caveat is that although our reference implementation of ODF standard accepts in such cases the values "page" and "page-content" for style:vertical-rel attribute, according to the tables on the page 660 of this file as well as at the page 672 of this one, they are not conforming to the specifications given the anchoring. So, although the files I generate using this workaround do the right thing^TM for me, they are strictly speaking not a correct ODF (although the validator is not detecting it) and other implementations might refuse them or render them completely incorrectly.

Frankly, I did not have time to dig into the code that implements this frame-related stuff, but someone who knows where to look might maybe find it rather trivial to fix. I said, "maybe", right?

The people wants screenshots!

I know, I know, and here is one.

I want hereby also thank my distinguished employer for making this possible by giving us the privilege of ITO (Innovation Time Off). It is really sweet to be able to accomplish things like this in the company time. And for those who face customer requests for converted WordPerfect documents with images and text boxes, the std::answer you will receive from me will be from now on to use the CVS HEAD of libwpd, libwpg and writerperfect to produce the wpd2odt tool that knows it all.

Password protected WordPerfect documents

2007-10-11T12:57:00.000+02:00

Because we live in very interesting times, some information might be pushed to background. Even though it is very important one. Last week-end an e-mail of Jaroslav Fojtik announced me his success in finding a way to decrypt password protected WP 3.x for Mac files. Just to precise, it is not about the protection cracking, just about the ability to read password protected documents using FOSS once you know the password. This masterpiece of reverse engineering (which is completely legal in some parts of the world) adds this file-format to the WP 4.x and WP 5.x for DOS/Windows that FOSS world was already able to decrypt.

The only file-formats that we are still not able to decrypt are WP 1.x for Mac and WP 6+. For the former, work is currently being done, for the later, it looks like it is a bit harder nut. If anybody of people reading this entry has any useful information about the encryption of the WP 6+ documents, please read here. We would like to stress that we are not interested in the password cracking and other illegal activity. We want simply to make it possible to a user of FOSS to read its own WordPerfect documents that she once protected using a password. So, abstain from communicating information whose publication could be illegal in your jurisdiction. OTOH, a genuine help with the decryption is most welcome.

The API for decrypting WP documents will most probably be part of libwpd 0.9.x series.

Barcelona no s'acaba a Catalunya

2007-10-11T12:02:00.000+02:00

libwpd 0.8.12, codename "Barcelona no s'acaba a Catalunya", left its mother's womb yesterday. Besides build fixes for the upcoming gcc 4.3, this release features initial (although not really so lame) support of tabulators for WP 2.x - 3.5e for Mac and WP 5.x for DOS/Windows file-formats, tabset conversion in WP 5.x, as well as some font ID to font name mappings for WP 1.x for Mac file-format. The cross-compilation framework was drastically updated and a compilation with Sun Studio 12 for Linux was also tested and this release is reputed to work with it.

As the name indicates, the major part of the code for this release was done during the wife-less nights spent at the OOoCon2007 in Barcelona. Although, as the name also indicates, some code was added even once outside the territory of the Paisos Catalans.

It is a free software released under LGPL and it is free as beer, so take it, taste it and enjoy it. The rpms for different SuSE and Fedora flavours can be found in my openSuSE build service home repository. The cross-compiled win32 binaries are to be found on the libwpd download page.

Besides the normal authors and contributors, the thanks for this release go to our very own Smokey Ardisson, to Rob Staudinger, Hub Figuiere and, last but not least, to my dearest employer, Novell, Inc. and its Innovation Time Off program.

Honest Broker?

2007-10-04T16:02:00.000+02:00

John, I completely agree with your point that one can assign copyrights to an entity if one can trust it. Where I don't agree is in your conclusion whether Sun Microsystems is such a honest broker. My own experience from the project is that a lot of lip service is payed to improvements of all kind, but that more the things change more they are the same. And I have a tendency to distrust entities that are full of good intentions at long term, but when the long term becomes shorter one is seeing basically no result.

Not that I am expecting anything good to happen. Between helping to build a strong developer community and trying to accumulate other people's code for proprietary licensing, Sun already decided anyway.

On the way to desktop freedom

2007-09-28T14:00:00.000+02:00

Not so long ago, I blogged about my strive to get rid of the need of running non-free operating systems while developing and maintaining my (our) different libraries. One step towards freedom was to find different emulators that could help to generate WordPerfect documents. That problem solved, one last thing was forcing me to have access to a non-free operating system. By the respect to our users and knowing that a typical Windows user is not really very good at operating compilers, for each release of libwpd, libwpg and other related packages, I was booting the win32 partition to maintain the MSVC project files and to build release binaries for Windows.

This time is over now. First to contribute to my liberation was the CrossToolChain repository in OpenSUSE.org Build Service. The MinGW cross-compiler is already successfully used to build the infamous unowinreg.dll during the OpenOffice.org build process. So, I gave it a shot and, after some modifications of the makefiles, the cross-compiling is a reality. What a pleasure to be able to run make -j9 and have the libraries built faster then their shadow! Inspired by some ingenious scripts by my distinguished colleague Tor Lillqvist, I even have a creation of release packages scripted and don't have to do much manually anymore. Some of the credits go also to an extraordinary expert in cross-compilations of all kinds, Rob Staudinger, who helped to implement some of needed configure options.

The last milestone that was remaining on the way to freedom was a need of Microsoft Visual Studio to maintain the MSVC project files in the different libraries. But, this problem is solved now too. The key of the solution is called CrossOver Linux Professional. Last night, helped by Uncle Google, I managed to install Visual C++ 6.0 SP6 in a managed bottle. The cool thing with the Professional version is that once the bottle properly installed and configured, one can create an RPM package and deploy it on other computers. And the application is basically accessible to all users of the computer.

For those who are impatient to see how MSVC running under linux looks, here is what you are waiting for!

By the way, I forgot to say that if you are running one of SuSE or Fedora Linux distributions and are keen to have the newest of the newest for libraries that I maintain or contribute to, you might find useful packages in my home project in our dear openSUSE build service. It is enough to chose your distribution and point the yum, zypper or other favourite package manager to the corresponding repository. So, pick the packages and taste how sweet they are!

WPG Import

2007-09-25T17:50:00.000+02:00

Recently in Barcelona, I was interested to hear there was some confusion over how the beautiful Word Perfect Graphics import was created. Since it is now shipping in most ooo-build based OO.o 2.3 versions, it'd be great to give full credit for all those who got over the time involved in it's creation and use in various FOSS applications.

Time-line

At certain point in 2006 I realized that because of lack of time I was neglecting one little useful library that I was also developing, libwpg and I made in my blog a call for new developers hoping that someone will bite the bullet.

Mathias Bauer (from Sun Microsystems) contacted me and proposed me work on this library could become part of OpenOffice.org SoC projects for 2006. After some e-mail exchanged, we agreed on the modalities of the participation: that libwpg will be developed as a standalone library (since it already existed as such) under LGPL because it was meant to be reused by other applications. Another agreement was that the goal was that the output of the Summer of Code will be standalone converter for wpg files to odg file format. This converter was supposed to be licensed under LGPL and its author was supposed to share his copyright with Sun under our beloved JCA terms.

Summer of Code candidate

I was contacted by several students in private and my choice was Ariya Hidayat of Koffice fame. The reason was that I knew his code and was quite sure about him being able to do good work on the library and converter. I worked (in my own free time, btw) together with him during the SoC on separable features, like reverse-engineering the WPG1 palette or implementing OLE2 stream abstraction. And it was really great to collaborate with Ariya who is an extraordinary hacker. What was promissed, that was done. Ariya's project got accepted for the Google summer of code, he signed the JCA, developed the library and added into it WPG2 support. In the same time, he developed a command-line tool called wpd2odg in a separate module. When the GSoC came to the end, the conditions for a successful accomplishment of the project were satisfied, the student got USD 4500 reward, Team OpenOffice.org e.V. got USD 500 and some Google swag like t-shirts and (paper) notebooks changed hands too. The "contract" was fullfilled and everybody was happy in the best of the worlds.

During the work on his Summer of Code, Ariya did an extensive blogging action, and one of Inkscape developers used a private copy of not yet released libwpg as a base for their WPG import.

After the Summer of Code

During the nights of the OpenOffice.org conference in Lyon, instead of sleeping, your servant was trying to implement a WPG import plugin as an add-on component for OpenOffice.org, one of those beasts that one builds using the SDK and that one installs using the Extension manager. It became clear that one had to refactor the wpg2odg command-line tool from scratch so that it generates Sax messages instead of a string in ODG file-format. At the end of conference, there was a refactored version of the wpg2odg tool and a non-working wpg import filter plugin for OpenOffice.org. Your servant's little brain managed to debug the plugin and have it detect correctly the WPG files and actually show some images only at the begining of October 2006.

Immediately, an OpenOffice.org integration was intented. It means, your servant tried to incorporate this plugin into the OpenOffice.org build-system as a new module. Nevertheless, since the implementation met some problems of locking when trying to imbue a locale into a C++ std::ostingstream, and your servant was pretty busy looking for a job that would help him feed his family, the work stopped there.

In the meantime, Dom Lachowicz of AbiWord fame modified the libwpg library adding into its API an SVG generating class so that it could be better reused by SVG supporting applications. Later, Marc "uwog" Maurer, implemented, using the libwpg SVG generation class, an AbiWord plugin enabling use of WPG cliparts in AbiWord files.

Hired in March 2007 by a nice company to work on OpenOffice.org, your servant requested permission to do some work on WPD and WPG filters in company time. The permission was granted and the whole odg generation framework used in the previous plugin rewritten to avoid having to use the problematic imbue on the stream (or the stream at all). The result is a patch that landed into ooo-build on 7th of May 2007. Since then, it underwent many changes due to the sharp eyes of Rene Engelhard and also due to an extensive QA by Sum1 whose favourite sport of trying to crash and tear down anything I produce is a great way to assure that my software is having a good quality inspite of the limited brain activity that TrainedMonkey is capable of.

This Summer, Ariya worked again on the libwpg library and added conversion of WPG bitmaps embedded in WPG files. During the Novell Hackweek, your servant wrote a BMP generator to be able to pass these bitmaps to OpenOffice.org and also added handling of embedded binary objects in alien file-formats (wmf, png, eps, ...). Again, thanks to AbiWord's Sum1, we were able to add handling of bitmaps in almost all supported colour-depths (indexed and direct). The result of this work is to be found in any ooo-build based OpenOffice.org 2.3 version.

Who did what

Libwpg library was written by Marc Oude Kotte, Marc Maurer, Fridrich Strba, and Ariya Hidayat. Some of the copyrights are owned by Novell because your servant was working on this library in his company time too. Some build patches were submitted by Robert Staudinger of AbiWord fame. The library is released under the terms of LGPL with some parts being licensed under BSD license.

Wpg2odg (command-line tool and the OpenOffice.org WPG import plugin extension) was written by Ariya Hidayat and Fridrich Strba. Sun Microsystem is joint copyright holder of some parts of Ariya's code and Novell is owning copyrights of some code contributed by your servant. This tool/plugin is released under the terms of LGPL with some parts being licensed under BSD license. One can use it as an OpenOffice.org extension, provided that one has libwpg library installed on the system. As a matter of fact, it runs even on some non-free operating systems with some non-free office suites.

The integrated WPG import filter in ooo-build is licensed under the terms of LGPL and copyright holders are Fridrich Strba with Novell for a huge majority of the code, Ariya Hidayat jointly with Sun Microsystems (for certain parts of the code). The patches from the above-mentioned links are free software and can be used by anyone under the terms of LGPL.

I would like to thank all those who contributed to making wpg import a reality on the free software desktop. And for those who are interested, libwpd 0.9.0 (whenever its release happens), will use libwpg for converting images embedded in WordPerfect documents.

Emulators

2007-09-13T11:49:00.001+02:00

Writing import filters for different WordPerfect^TM file-formats can be fun and rewarding. Nevertheless, apart one rare exception, all WordPerfect^TM versions run on non-free systems (Solaris Sparc, Mac OS, Micro$oft Windows). As a free and opensource software developer, moved by the will to be consistent with my own convictions, I develop using free operating system. But for filter development, and especially in cases where the documentation available is rather sketchy, it is a necessity to do some reverse-engineering. For this, one is really obliged to have working copies of different versions of WP. And also to run some versions of non-free operating systems. Nevertheless, having a server room in one's appartment can be considered as wife-and-kids-unfriendly act, and rebooting every five minutes to another partition is not so efficient workflow-wise. The good news is that a big part of the problem can be solved using free software.

For Windows versions, a solution was already outlined by Ariya some time ago. I modified a little bit the software installed and managed to fit all the necessary Windows and DOS versions of WordPerfect^TM on less then half GB disk image running a 98lite-modified version of Windows 98SE. The screenshot of this situation can be found here. The advantage is that such an image is fitting quite confortably on a USB stick.

Another problem was the Mac OS versions. And here it is really crucial to have a running versions. Because for some of the old file-formats, the only documentation that web is able to give you is the libwpd source code and this blog entry. And inspite of the "Mac Mini donated by unnamed on behalf of unnamed", the situation was less then optimal. Besides the fact that my KVM switch supports only PS/2 keyboard/mouse ports and a normal VGA monitor connector, it is quite difficult to use the Mac Mini when traveling. And frankly, MacOSX is quite an overkill for applications that install and work happily in a MacOS8 running on a Quadra. After looking on internet, I decided to give a ride to the Basilisk II emulator. After some time of struggling and helped extensively by Filip Molcan, Hubert Figuiere and hsfutils, I am able to run a little MacIntosh on each of my GNU/Linux computers, i386 as well as x86_64.

Although I installed all this thing on a disk image of 512MB, I realze that I have still more then 400MB of free space there. But even 512MB is small enough to back it up on a CD and put it somewhere on a USB stick. I also have to admit that I am quite impressed by the stability and speed of BasiliskII emulation. On modern processors, the emulated MacOS8 works faster than I ever have seen it running on a native Apple hardware back then when we were still young and handsome. I will not go into the details of the installation, but if the popular demand is there, I might document things that are not well documented on the web.

So, now only one little problem remains. How to run the WordPerfect^TM 5.1 for Solaris Sparc? Anybody knows about an emulator that could run the version 2.5 of this proprietary system?

Amused

2007-09-12T12:10:00.000+02:00

I was mildly amused when reading Michael's blog. Suarez-Potts had nothing else to invent to bash Novell then to start to criticise the way they spent money to bring developers to one place. In my not so humble opinion, inspite of the food served in the "World Capital of Gastronomy", there was not really anything to say against that experience. It was way better then the some-special-selected-people dinner sponsored by another company in Koper.

I was just wondering whether it is the same Suarez-Potts that did not refuse to profit of Google hospitality (flight and hotel payed) during the Google Summer of Code "Mentor" Summit. Funny enough, I don't remember suddenly what was the Summer of Code project he was mentoring last year.

But the bottom-line of this is that LSP is even criticizing the fact that people want to contribute engineering resources and try to encourage developers to contribute to OpenOffice.org. Just wondering whether Louis has an idea about what a FOSS project is. It is all about producing software and as good software as possible. As a by-product of this, there can be human relationships of other type that come to be, but the focus is still on the software.

Although it is true that OpenOffice.org project "structures" are quite efficient at producing talking shops instead of developers, this situation should be considered more as a dysfunction then as a normality. As well as it should be considered dysfunctional to have someone who is not developer representing OpenOffice.org at hacker's meetings. What is the argument such a person can have to encourage contributions? What level of understanding of everyday concerns of a FOSS hacker?

Frankly, it should be our target to fight these dysfunctions. And only if we succeed in this, there is hope that the "OpenOffice.org Community Manager" will have actually something to manage.

x86_64 tinderbox slave

2007-09-07T14:04:00.000+02:00

Just to tell you that we just added into the OpenOffice.org tinderbox build an x86_64 slave. Moreover, it builds with gcc-4.1.2 and with FSF java. So, if your build breaks in java files, check whether you did not write in an "übersetz mich" or other umlaut containing words. The other typical errors might be extra qualifications in declarations of C++ classes and/or treating of ULONG/LONG as if they were assured to be 32-bits. Keeping this in mind might prevent too frequent red status.

Thanks to Intel that provided this machine last year. And thanks to Christian Lohmaier who is guiding it and forcing it to do the right thing.

libwpg-0.1.0 - Initial release is out of the door

2007-08-08T18:25:00.000+02:00

Not without a lot of joy, I am announcing you that early this week a happy event took place and libwpg-0.1.0 official release came to be. Libwpg is a library for reading and converting WordPerfect^TM Graphics (WPG) files. In can be used by converters or renderers. A convenience tool and API for generating Scalable Vector Graphics (SVG) from a WPG file is included. A private statically linked version of this library is currently used by inkscape in their WPG import filter. The Novell Edition of OpenOffice.org 2.3 will also contain a WPG import filter based on this library, as well as the upcoming AbiWord 2.6 release.

As every initial release, even this baby might have some rough edges. Nevertheless, we are confident that it will bring joy to its parents and to all those who will have any use of it.

The thanks for this release go to many people. First of all to Mark "uwog" Maurer and Will Lachance, of multiple fames, for instigating this project. To Mark Oude Kotte (aka Foddex) for starting it, to Ariya Hidayat for doing almost all the work. AbiWord's Sum1 took his time tearing the library down and filing bugs against it before it even came out of its mother's womb. Thanks to him, we were able to catch and fix many bugs. Last and indeed least, the thanks go also partly to me who will take all the glory and to whom you can send your donations and thank-you contributions. We definitely should not forget Uncle Google who put some $$$ into this endeavor in the frame of Google Summer of Code in 2006.

The source packages and convenience binaries for Win32 platform can be found here. You are encouraged to download the, pet them, delight in them and maybe use them before they finish in the Wastebasket.

Enjoy and send your bug reports and feature requests to libwpg-devel mailing list (you'll have to register to do that though). And if you feel an urge to code, your patches will be most welcome at the same address.

Hackweek: The devil is in the details (or ODF generation woes)

2007-06-29T15:04:00.000+02:00

Yesterday, I was poluting planets with my story of converting images included in WordPerfect files. There were still some missing links and thus it is nice to give you another screenshot. Christian Lippka, a supreme intelligence of OpenOffice.org Draw and Impress fame, helped me to examine the OpenDocument Drawing objects that I was creating. The problem was that they were not scaling to the frame they were supposed to sit in. And we found the problem. (OK, he found it!) The solution is to include in the drawing document this XML snippet:

<office:settings> <config:config-item-set config:name="ooo:view-settings"> <config:config-item config:name="VisibleAreaTop" config:type="int">0</config:config-item> <config:config-item config:name="VisibleAreaLeft" config:type="int">0</config:config-item> <config:config-item config:name="VisibleAreaWidth" config:type="int">7941</config:config-item> <config:config-item config:name="VisibleAreaHeight" config:type="int">19124</config:config-item> </config:config-item-set> </office:settings>

The two magic numbers are width and height of the image in 100ths of milimeter. BTW, I am interested to see what will happen when this code starts to be used for their WordPerfect importer also by KWord. But, here I will leave Ariya the pleasure to handle possible implementation differences.

And so, thanks to the light of Christian, this is how the document looks like now. Ok, I have to confess a little cheating. The dimension of the frame is for the while not read from the file by a parser. I was quite lazy to start to parse this information. It is for the time being hardcoded in the code, but should not be conceptually difficult to parse the information,... just boring to death.

I would like to mention also another person that gave me a useful tool that I was using during this week. Far from being anything close to XSL(T) fan, I found extremely useful Svante Schubert's transforms that allow to load and export files in OpenDocument flat XML format in and from OpenOffice.org. It is a load easier like this to make little experiments with document without having to run zip and unzip zillion of time.

So, what remains to be done? Naturally, to write the parser of the box information for different WP file-formats inside libwpd. This is something I am really finding very boring and ungrateful task. So, if you want to be my personal hero, send me a patch.

For those WordPerfect users that have a load of documents with images, the documentation says that one "Graphics Filename" prefix packet can point to several "Graphics Cached File Data" packets that contain the graphic information. Nevertheless, the documentation does not say how the data is split in this case. In several full-blown WPG streams or in one WPG stream that has chunks stored in different prefix packets? I was unable to create a document with several pointers in the "Graphics Filename" prefix packet, so if you have some of them, here I am to receive them. Or even better, send me a patch for their handling in libwpd.

Hackweek @ Novell

2007-06-28T16:21:00.001+02:00

It was said, it was repeated and repeated. This week, is a hackfest week at Novell. We are hacking (programming for those who do not like the word) on different projects/ideas that are close to our heart. Naturally, TrainedMonkey(tm) could not remain behind. And guess, what projects he chose to hack on... Naturally, libwpd, libwpg and their OpenOffice.org-related filters. It is really nice to be upstream maintainer of several things, like that, when given a week of free hacking, one is likely not to lack ideas what to do with the time.

The import of embedded pictures is one of the most overdue features in our WordPerfect(tm) converter. So, I jumped on the task when this opportunity came. And fortunately, not everything had to be coded from scratch. Ariya, in the frame of a Google Summer of Code project gave a decisive push to libwpg and he has kept working on improving it. We came to the point, where standalone WordPerfect Graphics files can be nicely converted to SVG or ODG including embedded bitmaps. So, what were the steps to do?

First of all, it was necessary to port libwpd/writerperfect to generate OpenDocument instead of the legacy OpenOffice 1.0 file-format. This was needed because of the nice way, one can incorporate binary objects in OpenDocument represented as flat XML. This was accomplished on Monday, and the conversion of all documents from libwpd regression suite produces a nice ODF stream that validates against the OpenDocument 1.1 strict schema.

Next step was actually to parse the embedded image data in WordPerfect documents. A nice discovery is that in old WP5.x documents, the images are stored without the WPG header. So, one had to hack into libwpd a possibility to force the document parsing even if it does not recognize it. This was done on Tuesday.

The same day in the evening, I started to hack on passing the data libwpd gives us to libwpg and processing its output and incorporating the images into the OpenDocument text stream. This work continued on Wednesday and resulted in finally seeing an image. The scaling and anchoring was not good, but at least one could see something else then lines of cheesy C++. But it was really hard to make the generated documents (although valid according the ODF schema) load nice and display the pictures well. I poked some people whose brain should be a bit better then a brain of TrainedMonkey(tm), but did not manage to get more intelligent. And since a soup is never eaten as hot as it is cooked, I left it there for the night.

A night brought some rest for the mind and I started my day by a nice chat with the guy who is expert in pictures embedded in OpenOffice.org Writer documents. I came to the conclusion, that the best would be to include those images directly as <draw:object> inside a <draw:frame>. The advantage is that this would keep the images as editable pictures, the same way the Corel customers can edit them inside the WordPerfect Office(tm) applications. This meant to throw a big chunk of Wednesday's code in favour of a more elegant solution. For those familiar with libwp* world, the OdgExporter class was copied to writerperfect too :-) (Now it is part of Novell's WPG import filter for OpenOffice.org, of wpg2odg tool, of perfectspot graphics viewer, and now of writerperfect). There is a way of improvement, since it is likely, that it should not be very difficult to join wpg2odg and writerperfect in one source package.

I spent some time trying to debug my code, since the images were not showing. Then, by chance, I discovered that the devil was in the right office:mimetype attribute of the embedded object :-(. So, now one can see the images inside a frame. The scaling is still not correct, and I am not sure at this moment, whether I will be able to scale the image as whole or will have to scale every single shape on import, but it results in something that is editable. And that was the goal of the rewrite.

Tomorrow, I will try to code again the parser of the box information, so that we extract from the file information of the box anchoring, size and position. This promisses to be quite a boring stuff, but necessary missing link.

BTW, libwpd and libwpg will not refuse any fine hacker that would like to contribute to enhance our conversion feature-set while still keepeing our stability and document import rate.

An (almost) ideal OpenOffice.org build box :-)

2007-05-21T11:03:00.000+02:00

It was supposed to be cold this weekend, since it started by quite an abundant rain, and our central heating is not considering May as winter month anymore. So, I decided to do something to increase the temperature of our apartment. I have a little Sony Vaio PCG-SR11K laptop lying around. It used to be a nice little tiny and portable machine in its time with the cute 10.5" display with a resolution of 1024x768. But, given the Pentium III processor, running at 600 MHz, and 256 MB of memory, it is working quite well for reading e-mail, browsing web and doing some little other tasks, powered by SuSE Linux Enterprise Desktop naturally, but building OpenOffice.org does not look like a great idea.

Yes, sure, if a quick build is what one wants to obtain... Nevertheless, I gave it a try. Just for the sake of preventing the processor of being too idle during the build I tried two builds in parallel. A stock ooo-build (our dear Novell version) configured --with-distro=SUSE and a vanilla SRC680_m210 with the CWS writerfilter2 integrated. The later build was with as many system libraries as possible and also with the --without-stlport4 option.

I put the laptop in a position to be sure that the CPU fan has a lot of air circulation and fired the two builds at 19:00 on Friday. The ooo-build one (without the SDK and binfilter) was finished on Sunday somewhere between 10:30 and 13:00 (I checked it when back from Church, so I cannot give smaller time fork). The vanilla build finished partially on Sunday at about 18:00. Yes, I forgot about the damned extra qualification in sd module and was quite happy that I issued the build command with --all --ignore options. Once fixed that one and patched out some stlport-only features from the writerfilter module, the successful packaging process was announced at about 22:00.

The bottom-line is that the laptop survived it although it was nearly impossible even to read the e-mail while building. The temperature of the house was quite pleasant, partly also because the Sun (not Microsystems) decided to show up from Saturday morning. Nevertheless, the temperature of the air going out of the fan is indicating that putting that "laptop" on one's laps while building can be quite dangerous for reproductive health :-)

Qemu and kqemu on SLED10 SP1

2007-05-21T10:23:00.000+02:00

Ariya, maybe an even easier way would be to take the kqemu and qemu source packages from SuSE Factory, and run rpmbuild --rebuild kqemu-<version>.src.rpm and install the resulting kernel modules. After that, rpmbuild --rebuild qemu-<version>.src.rpm and install the result as well :-) This is the way it worked nicely on my SLED 10 SP1.

Completely agree that Qemu is a nice tool for all filter developers.

Focus on the logic behind a file-format, not on the container used

2007-03-02T11:44:00.000+01:00

Diving into the Office Open XML file-format and in the same run into all other MS Office file-formats (if one makes abstraction of the container, content is the same), one realizes the different ways that can be used to store the same thing. Unless designed with interoperability as a pricipal aim, a file-format's logic reveals nicely the way the application that produces it operates. And the fact that it be binary file, a compressed structured document composed by several xml streams, or a RTF-like file is not making much difference (apart of the difference in human-readability). And it is exactly the difference in the underlying logic that renders impossible to reach an optimal result in translation by using a descriptive tool like XSL transformations. Let's enjoy together the main difference between MS Office, WordPerfect and ODT/HTML file-formats:

MS Office

The main idea behind the MS Office word processing file-formats (.doc, .rtf, .docx and other xml-based attempts) is that the node (paragraph, section, ...) properties are "stored" in the "character" that marks the end of the node. I.e.: The paragraph properties will be all properties that the paragraph break marks finds set. And these properties will be properties of the paragraph whose end this paragraph break marks. From the point of view of the document flow, with some nuances, the paragraph properties are applied retroactively to the current paragraph. And it is similar with the section properties.

WordPerfect

The WordPerfect file-formats reflect also well the underlying conceptual model of its application: typewriter. As well as in MS file-formats, a WP file can have the paragraph or page properties defined anywhere in the document. And the codes are applied as soon as the formater is able to do it. Nevertheless (with the exception of headers/footers that appear in certain WP file-formats on the same page as the code that defines them), they are never applied retroactivelly. The properties are crossing the boundaries of hierarchically superiour elements. I.e.: If one sets direct code of "bold" somewhere in the document, this code will be valid and have effect untill another code that cancels it. The number of page, column, ... breaks between the two does not matter. If one changes a paragraph alignment in the middle of the paragraph, for instance, newer versions of WordPerfect will add a "temporary hard return" code before the alignment code to hint that a paragraph break is necessary in order to render this change. Nonetheless, older versions of WordPerfect (still using the same file-format) will leave the task to realize this fact to the formater/layout engine itself.

A direct result of the typewriter conceptual model in WordPerfect are those special tabs that disregard completely the tab-table and that contain themselves the information about their position and alignment (hard tabs, center on margins, flush right,...). A reader that did not yet get lost in this entry will appreciate how nice it is to translate these beasts into a file-format in which all tab properties are defined in a tab-table that has to be known at latest at the moment of beginning of the paragraph.

ODT/HTML

These file-formats have the particularity that the information about an element is enclosed in the element's opening tag. The alignment, indentation or spacing of a given paragraph is known at the begining of the paragraph, better said at the moment of the paragraph opening. This is the same with the tab-table for those file-formats where it makes sense (a tab-table has no sense in the html file-format although some extensions are trying to implement it in some way).

Document conversion

Given the fact that a word processing document is read sequentially, the conversion between these different formats can be quite a pain in some posteriour place of one's body. For instance, if you want to convert directly from a MS Office file-format to HTML/ODT, you will have to parse the document two times: first to collect the styles of the nodes (that become manifest at the end of the corresponding node) and second to parse the content of the nodes. For a proper conversion of WordPerfect "special" tabs and headers/footers, one has to proceed in a similar way, besided a deep magic to incorporate the "special" tabs' position and alignment into a tab-table of the given paragraph.

New OpenOffice Writer filter API

As I already mentined, the direct conversion is likely to be a pain in some posteriour place, unless...

...an import filter does not need to do the whole conversion and can feed the information into a structure where one can modify the properties of a given paragraph/section/text-run aka text-span during the content parsing. And this is exactly what the new filter API that is being implemented in the writerfilter2 CWS is designed for. Stay tuned for some more exciting adventures concerning the file-formats!

Just to add that the libwpd API gives the "node" properties in the openFoo callbacks. And it is more or less trivial to plug this logic into the new Writer API logic. Because it is much easier to have some information early and keep it for later on then the opposite case.

Work @ Novell

2007-03-02T11:42:00.000+01:00

I am owing thanks to the Lord God Almighty for being merciful and gracious with me. He extended his hand over me and my family and accompanied us through the desert of the year 2006. He blessed us abundantly and exceedingly above all we could ever hope for. I am writing this because I want to give all the honour and all the glory to The One Who deserves it in first place.

I want to thank also the Novell OpenOffice.org Team for having considered me with favour for a position of Software engineer working on interoperability issues between MS Office and OpenOffice.org. I am excited to see my hobby activity becoming my regular job. I cannot thank you enough, guys, for giving me an opportunity to work full-time on this great FOSS project.

I want to give my thanks also to other people who were ready to help me throughout 2006: Sophie Gauthier, the lead of OOo French native language project, Mathias Bauer, former framework and current sw lead (and friend), Thorsten Behrens, the gsl co-lead (and a great person and friend), Marc "uwog" Maurer and Dom Lachowitz, AbiWord admins and fine hackers (and helpful friends), and to all those people from OpenOffice.org, AbiWord and libwp* projects that were very supportive in action and in kind words: Rob, Sum1 (really SomeOne), Will, Andrew, Ariya, Pavel, Caolan, Heiner, Stefan, Eric and all those whose names I might have forgotten to put down here. Thanks, good pexcitingeople, for your willingness to put hand in fire for a TrainedMonkey and to help and encourage him. I really appreciate!

Last, but in no way least, I want to thank particulary Michael Meeks, Distinguished Engineer and "hacker extraordinaire", for injecting in me the uncurable virus of OpenOffice.org. He helped me to enter the community. Thanks, dudie, for guiding and supporting me and for being a friend inspite of my big mouth!

I am looking forward to advancing the cause of the Free Desktop inside of a company that believes and puts in practice what it preaches.

Miriam (and us) in pictures

2007-02-21T14:07:00.001+01:00

Because of high demand from the public, here you can find some photos of Miriam and associated folks.

Miriam Terezia Gladys

2007-02-14T07:28:00.000+01:00

The Lord is Good and his mercy endureth forever.

We thank the Lord God Allmighty for his goodness and abundant mercy. He blessed us in a mighty way by giving us our daughter and sister, Miriam Terezia Gladys Strba. She came to this world on 14^th of February 2007 at 4:15 AM. Both Miriam and her mother are doing great and, together with the rest of us (in the role of passive spectators), thanking God for His blessings.

Uberdedicated TrainedMonkey?

2006-12-21T16:53:00.000+01:00

For those that do not want to wait for the integration of the CWS wpsimport01, the Uberdedicated TrainedMonkey cloned the standalone writerperfect from the libwpd CVS repository and created a command-line tool alowing you to convert now your stack of MS Works documents into OpenOffice.org 1.0 file-format. The cloning was done mainly for debuging purposes, because the com.sun.star.comp.Writer.XMLImporter seems more picky than the normal document-loading process, but it is usable for converting documents too. The module is called "wps2sxw" and lives in the libwps' subversion repository.

So run there to pick it while it is hot! And do not forget to subscribe the project's mailing list and send some encouraging words to Andrew. Believe me, he does an excellent work!

One word that matters

2006-12-19T12:51:00.000+01:00

One word can change many things. And sometimes the words that one is not used to use so much might have a kind of magic in them.

Here is the problem we were facing yesterday. This piece of code was refusing to compile:

1 #include <vector> 2 template< class key , class hashImpl , class equalImpl > 3 class OMultiTypeInterfaceContainerHelperVar { 4 typedef ::std::vector< std::pair < key , void* > > InterfaceMap; 5 InterfaceMap *m_pMap; 6 inline void * find() { 7 InterfaceMap::iterator iter; 8 return NULL; 9 } 10 };

The complain was about an expected ';' before 'iter' on line 7. One can conclude by looking close that the code is correct and start to file a bug in the gcc bugzilla :-) And yet, inspite of the fact that it looks very reasonable for anybody who has a basic knowledge of C++, this code is not valid C++ and the compiler is right. For the line 7 to compile correctly, InterfaceMap::iterator has to be a type. But the compiler cannot be 100% sure about this at the moment when it is compiling the template class. It will know more at the moment of the instantiation of the template, but not before. There is still a tiny little probability that there will be a specialization for a certain class key where the InterfaceMap::iterator will not be a type. Because the compiler is not 100% sure that the InterfaceMap::iterator is a type, it assumes that it is not.

But now what? How can one make the compiler accept the code? Simply, by telling it that the InterfaceMap::iterator is a type. Here it is where a word that is not so often used (In fact, your servant never had to use it for the while) becomes handy. If you precede the statement at the line 7 by the keyword typename, everything compiles without a quirk. The presence of this keyword takes the compiler out of the uncertainty and, with its mind settled, it accepts the code.

So, the rule of the thumb is to precede the use of the dependent nested types in templates always by the magic word typename.

Having said this, I am quite happy that the long winter evenings spent with Scott Meyers were not amiss.