Free Software, Metal and Uncalled For Angry Rants

A Decent Compressed Filesystem At Last?

2009-08-11T23:14:00.002+01:00

I have a 2.4GHz CPU, a 250GB hard drive and I want to store 29GB of zipped music I just downloaded. The music is in a variety of ProTracker formats, so what's the best way to store it?

I know for a fact that tracker songs are very easy to compress, since the format originates on computer systems like the Amiga where any compression more sophisticated than Run Length Encoding takes away valuable clock cycles which are often needed elsewhere. Such uncompressed, or crudely compressed, data is ripe for extra compression (hence why it is transported in a zip archive).

Actually playing this music is now nontrivial though, since whilst all of my music players support protracker, not all of them support reading them from zip files. The obvious thing to do is decompress them, but since trackers are highly compressible, this means that the 29GB archive will certainly become even more vast when decompressed, and that's just a waste of storage space. The solution is to use a compressed filesystem.

For those who are stuck on shit operating systems, I'll give you a little insight into how your computer works. Let's say we're running Microsoft Windows, and we insert a CD into our CDROM drive. In "My Computer" we may see our CDROM is called "D", and if we double click on it we see the things stored on it. However, we then go back to My Computer and right click on D, then tell it to eject. We take out the CD and close the drive. Now we tell it to eject again, and the drive opens. So, what is "D"? It's our CD, since we saw the files on it, but it's also the drive, since we can tell it to do things when our CD is nowhere in sight. Which is it really? Well, that depends completely upon what you want to do, since Windows is a very poor system indeed.

Let's imagine a similar thing on a UNIX system like Debian. Our CDROM will be something like /dev/hda. We can send it commands, for example "eject /dev/hda" (or do the same thing in whichever graphical environment you happen to use). Can we access the files on a disc through /dev/hda? No. What we can do, however, is take the data given to use by the drive, and reconstruct it somewhere. This is what we mean by mounting the filesystem. The filesystem is the particular way the ones and zeroes represent our files and folders (CDs use the standard ISO9660, which is often extended with certain proprietary formats from Microsoft to allow longer filenames. Ironic, considering that the original format only had short 8.3 filenames to make sure it would work on Microsoft's POS crapware), whilst mounting it means making it available to peruse. Here we can choose where we want to see things, so we can run a command like "mount /dev/hda /our_cd". Now if we go to the folder /our_cd the previous contents will be hidden (until the CD is unmounted) and the contents of the disk is accessible. /our_cd is a different file to /dev/hda. The same is true of hard drives.

In fact, filesystems can be anything. The FUSE (Filesystems in USErspace) driver allows regular programs to be written which can be accessed like filesystems. This is how the Wikipedia filesystem works (mount it somewhere and that folder becomes filled with files for each article), how the GMail filesystem works (data is stored online as a series of emails in a GMail account, which can be retrieved from anywhere and are automatically converted back to their original state), and many others. There are several FUSE filesystems which transparently compress and decompress their contents, ie. everything copied into the filesystem is sent to a compression program and saved somewhere, and everything read from the filesystem is sent through a decompression program before it reaches the destination. Thus any program, as long as it can load files, can load compressed files, and any program, as long as it can save files, can save compressed files.

Seems cool, but until recently the only ones I'd tested were pretty dire, most noticably CompFUSEd. INCREDIBLY slow and memory hungry, it was not worth using, and that was only when a few MB were put into it.

However, FuseCompress has recently been added to Debian, and I'm trying it out for these tracker modules. Whilst populating the filesystem is taking a while (I'm having to decompress all 122 thousand songs (since that's how we want them to be read and therefore written), then move them into the filesystem where they're recompressed (although using LZO this time, which is damned fast). All I can say is thank Guido for Python, since it makes automating such things a breeze :)

I thought I'd finish with a little introduction to compression, and what it actually is. Compression is completely based on Claude Shannon's Information Theory (also the basis of using switches to represent Boolean logic, ie. allowing a physical way of building Alan Turing's theoretical "computers"). Information, measured in bits, is irreducible, you can't throw away any bits without losing some information (compression which does this is called "lossy" compression, for example the Vorbis audio codec. However, the algorithms are crafted in such a way that information is only thrown away when it is imperceptible to us, in the case of Vorbis we can't hear the difference), but the key thing to know is that each bit of information is not necessarily mapped one-to-one with each bit of whatever material is being used to store it (eg. magnetic domains on a hard drive). A bit of information is required to describe a situation where there is a 50/50 chance of the next bit being 0 or 1, but this is only true for random sequences, or those which appear random. A sequence like "AAAAAAAAAAAAAAAAAAAA" is not random (each bit isn't independent of the previous), and thus does not require all of the 160 bits that are being used to describe it. Information theory gives us a lower limit, saying how many bits are REQUIRED to store the given information, so we want our algorithms to approach this limit as much as possible.

A really simple type of compression is Run Length Encoding. This looks for repetition and replaces it with multiplication. For example, the sequence "ABCBAAAAAAAAABBCCCCCCCCDBDDDDB" could be compressed to "ABCB9ABB8CDB4DB". Our algorithm here is simply "if you find more than 2 of the same letter in a group then replace that group with its size followed by the letter", so a stream of "AAAAAAAAA" becomes "9A". To decompress this we use the algorithm "If you find a number, put that many of the next letter".

There is a slightly more general form of this, where instead of grouping similar things, we use a pointer. The pointer is a number which means "I am the same as whatever is this far behind me". In this way "AAAAAAAAA" can become "A12345678", where each number is telling us to get the next value from this far back (they all point back to the first A), however pointers can also point to pointers, so we could just as easily put "A11111111", since each of those "1"s becomes an "A", so it's perfectly valid for the next one along to point to it. This is easily compressible with the runlength encoding seen above (which is just a subset of this form of compression), but is more powerful. For example "ABABABABABABABABABABABA" cannot be compressed with runlength encoding, but using pointers we can compress it to "AB222222222222222222222". Now this is easily compressible. This works for any size of pattern, although since it is a search operation it can get quite slow for large search spaces, so files are usually split into more manageable chunks first.

A final form of compression I'd like to mention involves binary trees. Let's say we made a survey of the occurances of every letter in a file. We could say, for example, that "e" was the most common, followed by "a", followed by "s" and so on. Now we can compress these from their usual 8 bits to a much more compact form. First we define a binary tree, that is a tree where every non-leaf node (branch junction) has a left and a right branch. If we meet a "0" we will go down the left branch and if we meet a "1" we will go down the right branch. Our tree will begin with two branches, and we can stick "e" at the end of the right branch. On the left branch we put another node with two children, on the right we put "a" and on the left we put another node with two children. On the right of this we put "s" and on the left another node with two children, and so on. Now we can replace each letter by the path we must take in our tree to reach it (with our tree we know that "1" marks the end of a letter). Every time we find an "e" we simply put a "1", since that's how we get to "e" from the top of the tree (1 means go right) This saves 7/8 of the space every time. Every time we find an "a" we replace it with "01", which saves us 3/4 of the space, an "s" with "001", and so on. By constructing optimised trees (which is once again a search operation) we can get really good compression ratios.

Anyway, rant over since my script has finished moving all of the "A"s :D

The Ugliest Hack I've Written So Far

2009-08-04T19:28:00.006+01:00

raise :i ::= :a ?(a.__class__ == Raise) => 'raise '+', '.join([t[0] for t in [[e] for n,e in enumerate([a.expr3,a.expr2,a.expr1]) if e is not None or any([a.expr1,a.expr2,a.expr3][-(n+1):])] if (t[0] is not None and t.__setitem__(0,t[0].rec(i))) or (t[0] is None and t.__setitem__(0, 'None')) or True][::-1])

This is a 1-line PyMeta rule which means the following:

Define a rule named "raise" with an amount of indentation "i", which applies to anything, which we'll call "a", as long as "a" is a type of 'Raise'. Upon finding such a thing we should output a string 'raise ' followed by the first item of every list in the set of singleton lists of "a"'s attributes 'expr3', 'expr2' and 'expr1' when reversed which is either not equal to "None" or else comes after a non-None attribute, if the element of these lists is not None and swapping the first element for its contents recursively at the same indentation level, or if it is None then replacing it with the string "None".

If I were writing this normally it would be something much cleaner like:

def raise(i, a):
  if a.__class__ == Raise:
    to_return = 'raise '
    attribs = [a.expr3, a.expr2, a.expr1]
    to_keep = []
    not_end = False
    for att in attribs:
      if att is None or not_end:
        to_keep.append(att)
      elif (not att is None) and (not not_end):
        to_keep.append(att)
        not_end = True
    to_keep.reverse()
    to_return = to_return+', '.join(to_keep)
    return to_return

but the default PyMeta grammar only allows a single line of Python in the output. Whilst the point of OMeta is that I can subclass and rewrite it to work in whatever way I want, I can't get the hang of subclassing grammars yet, and hence this mind-bending, yet at least partially elegant, functional approach.

Some Nice Things

2009-04-27T01:31:00.004+01:00

I've not posted for a while, due to a mixture of an increasing workload, the ability to let off a constant barrage of my thoughts to Identi.ca rather than build them up into a blog post, and my constant disdain for Web-based apps.

So what do I want to blog about? Nothing particularly structured, just some stuff that I find interesting. Keep in mind though, that my definition of interesting includes the fact that 12cm optical discs have increased their storage capacity by 2 orders of magnitude in the 27 years from the CD to the BluRay, whilst in the same time frame the capacity of 3 1/2" hard drives has gone up 12 orders of magnitude. (I'm writing an essay on Optical Data Storage for a Physics module :) )

For those of you who may remember Deluxe Paint on AGA capable Amigas I can heartily recommend that you check out Grafx2, which seems to work on pretty much every OS and has recently been added to Debian, so you can install it by ticking "grafx2" in any package manager, it will be downloaded and installed along with everything it depends on :) Doesn't seem to do animation yet, as far as I can tell, which is a shame.

Also recently added to Debian is Closed World Model, cwm. This is pretty special, since it takes cutting edge computer knowledge representation as used by the Semantic Web, and makes it accessible via a tool similar to UNIX's (and of course GNU's) classic sed tool. For example, you can use a command like "cwm --rdf inputfile1.rdf inputfile2.rdf --n3 inputfile3.n --rdf --think --pipe > output.rdf" to take at all of the knowledge from the RDF files inputfile1.rdf, inputfile2.rdf and inputfile3.n (in RDF-XML and Notation3 formats), comparing the knowledge they contain, and dumping all of the new knowledge it can infer into the RDF-XML file output.rdf. For example, inputfile1.rdf could contain statements that Chris Warburton is a student, Chris Warburton has a website http://www.freewebs.com/chriswarbo and that Chris Warburton has a brother David Warburton. inputfile2.rdf could say that Brothers are related and that Brothers share a Mother. inputfile3.n could say that David Warburton has a blog at http://fun-chips.blogspot.com and David Warburton has a mother Cheryl Warburton. cwd would then combine these and the output file would contain deductions such as David Warburton is related to a student, http://www.freewebs.com/chriswarbo is run by a student and Chris Warburton has a mother Cheryl Warburton.

This is pretty cool, since it commoditises the previously tricky area of RDF access, allowing it to be scripted, for example in the backend of Web sites, in the same way that Imagemagick has done to images (eg. for thumbnailing).

Pretty cool. Anyway, it's getting late so I should get some sleep now.

I'm going to post some of my programming experiments soon, so look out for them :)

BBC News comments are broken

2009-03-24T20:15:00.001Z

Tried to post on this http://news.bbc.co.uk/2/hi/uk_news/7955205.stm but it failed :( Didn't want to lose it though.

The harrowing trend to notice amongst government statements on these issues of technology, privacy and civil liberties is the focus on the meaningless technology arguments rather than the important freedom related ones.

To me the idea of "a profiling tool which examines a child's behaviour and social background to identify potential child offenders" makes a sickening mockery of the notions of innocent until proven guilty, freedom of speech and expression and equality. I don't care if it's encrypted or 'secure', or how much such a thing would cost, it simply shouldn't exist in the first place!

The same sidestepping of the main topic can be seen in most of these stories, even across the world. There was an article posted recently about Australia's Internet blacklist, and whether it is an offence to the human right to Free Speech. The conclusion was that such a blacklist might slow down the Internet, and wouldn't stop everything, which once again I don't much care about.

Technology is advancing ferociously, and will continue to do so. Making important decisions based on technological issues sets an unnerving precedent. In the Australia example, in a few years or decades time I'm sure Internet latencies will be so low that such a blacklist would be unnoticable. From the misdirected conclusions of that article then it should, since the technological issues raised will have been fixed.

In the case of these databases, if technological advances such as quantum entanglement cryptography fix the security concerns, and supercomputer-esque processing power and storage are available for pennies, does this mean that all such databases should be made? Of course it doesn't, yet that is the argument being put forth by the government.

I call to reject any spin-ridden arguments based on petty implementation details and keep the focus on where it matters, the reasons for and against even contemplating the possible existence of such systems.

Learned Helplessness in Computing?

2009-02-06T02:37:00.003Z

I know I should be revising, seeing that my Atomic and Laser Physics exam is mere hours away, but I ended up on another Wikipedia trek, and came across the article on learned helplessness. Reading through it, I found that I could make many connections with the currently depressing state of computing, and attributing it to the complexity and proprietaryness of software.

Learned helplessness is a much-studied psychological phenomenon, where a subject gives up trying to change their situation. An example cited consists of three groups of dogs; group A is a control group and are put into harnesses and left for the duration of the experiment; groups B and C are put into harnesses but are also given unpleasant electric shocks. Each group B dog has a lever in front of it which does nothing when activated, whereas each group C dog has a lever which turns off the shocks to that dog and one of the group B dogs. The dogs in group C learn that the lever turns off their shocks, and they use it whenever they start to get shocked. Group B dogs, however, learn that their lever does nothing, whilst their shocks seem to stop randomly (remember, each B dog is paired to a C dog's lever, so the B dogs don't know why their shocks stop).

After this stage of the experiment all of the dogs are put into part two. where they are unharnessed in a pen divided in two by a small partition. The half of the floor with a dog on is electrified, whilst the half without is normal. Dogs from groups A and C would hop over the partition, away from the electricity and thus away from the pain. They don't know that the other side's not electrified, but they have a go and find that it's not. The dogs from group B, however, just lie down on the electrified floor and whimper, as they are repeatedly electrocuted. They could hop over the partition, but don't bother trying. These dogs become depressed.

The conclusion of the experiment is that the sense of control is very important. Dogs in group B and group C got exactly the same shocks (since they were both controlled by group C's levers), but only group B got depressed. Essentially, they learned that nothing they did would stop the electricity, it just stopped randomly. They then applied this knowledge to the second situation and took the shocks, rather than trying the new possibility of jumping over the divide.

This can be seen in people, where some parents can end up neglecting their babies since they 'learn' that the child doesn't stop crying whether they give it attention or not, and thus ignore it, thinking they are helpless to stop its cries.

The psychological explanation for this is that the depressed subjects, in an attempt to rationalise the seemingly random lack of control, think of it as an inevitability ("Babies cry"), blame themselves ("I'm making it cry") and think of it as pervasive ("I'm a bad parent"). This learned helplessness digs a psychological hole which is notoriously difficult to break out of, and even causes feedback loops, for example a neglected child will cry more and have more problems than that of an attentive parent, thus reinforcing the "I'm a bad parent" and "I'm making it cry" beliefs. In fact, even knowledge of learned helplessness can make things worse, since it can act as a confirmation of the helplessness ("You've told yourself that you're helpless when you're actually not." "See? I TOLD you I was a bad parent!") and others can end up blaming the condition for things rather than the person ("It's not your fault that your baby's ill, you've learned to be helpless at looking after it." "Yes, you should probably take it away since I'm too learned-helpless to look after it.")

So, aside from knowing more being awesome, how does this apply to anything I'm interested in? Well I couldn't stop contrasting the explanations with computing. The dominant computing platform these days is Microsoft Windows which, although all software has bugs, seems to be full of them. A lot of these bugs are user interface related, where the required action to achieve the desired task is non-obvious, or a seemingly obvious action produces an unexpected result (which includes 'crashes', where a program disappears without the user telling it to). Although anyone more involved in software development would view these as bugs in the software which should be reported and fixed, frequently less technical users (which is the vast majority) view such things as inevitable ("Computers crash"), as their fault ("I made it crash") and pervasive ("I'm bad with computers"). Just look at the currently running adverts for the Which? PC Guide: A bunch of regular people saying how their computers keep messing up, and then an offer of a guide to show them how it's all their fault because they're doing it wrong.

Since I write software, I would say that the Which? PC Guide is a complete hack: It's fixing something in the wrong place. A broken piece of software should not be fixed by telling each and every user how to work around the broken bits, the software should be fixed so that nobody ever experiences those issues again. However, since it's proprietary software, nobody is allowed to fix it other than Microsoft (although there are numerous other hacks to work around the broken bits, some of which have created an entire industry, such as firewalls, anti-virus/spyware/adware programs, etc.).

The majority of computer users, however, do not think like me, since I am a group C dog: I know how to fix things. In fact, in human experiments into learned helplessness, it was found that people could concentrate more and solve problems more quickly in the presence of an annoying and distracting noise if they had a button which could turn it off, than those subjected to the noise without such a button, EVEN WHEN THE BUTTON WASN'T PRESSED. So on a Free Software system, where I know that it is possible for me to fix something if I truly wanted to, I don't get depressed, however on a proprietary system I frequently get annoyed, angry, irritated, etc. when the software behaves in undesirable ways.

For example, clicking a link that says "Download this program" in Idiot Exploiter 8 doesn't download the program, it just gives an subtle message under the toolbar that Internet Explorer has "protected" me by preventing the program from downloading and that I should click it to change that, and when clicked presents a menu with the option to download the program (how is this any different to the previous promise of a download?), which when clicked brings up a box asking if I want to save the program or run it, so I click run and when it's downloaded I get a warning saying that programs can do stuff to the computer, do I want to run it? I click run again (how is this any different to the previous promise of running the program?) and Windows pops up a message saying that the program is doing stuff, do I want to allow it to continue? I press continue an FINALLY get the the "first step" of the installer.

On Debian I could give a similar example that I can't get the Gdebi package installer to work, which means that I have to save packages and install them with the command "dpkg -i package_filename.deb", which can result in a broken setup if the newly installed package depends on other stuff, which means I need to install the stuff to fix it with "apt-get -f install" and press "y" to confirm it. This may seem annoying, but I know that if I wanted to fix it badly enough then I could, and would even be encouraged to do so (afterall, Gdebi works perfectly well in Ubuntu).

Whenever my wireless card messes up on Debian, on the other hand, I get incredibly frustrated and annoyed, and often need to walk away from my laptop and have a break, since I feel completely powerless over it. The wireless firmware I use is proprietary, since Broadcom don't tell anyone the language that their wifi chips speak (although clean-room reverse engineering of a Free Software replacement in Italy seems to be showing some promise), so even though I'm running completely Free Software applications on a Free Software kernel of Free Software drivers (in my case Linux), and can look at the code at any time to see what it's doing and possibly fix any problems, when it comes to my wireless card the disconnects are seemingly random, as I have no way of inspecting the firmware since it is proprietary. I therefore feel helpless to stop it disconnecting, and can't remedy the situation in any way other than disabling the Wifi, unloading the driver, reloading the driver, enabling the Wifi and trying to reconnect. If that doesn't work then all I can do is to try it again. In fact, I've even written a little script which does all of that whenever I run "restart-wireless". It's so bad that the developers of NetworkManager, the (currently) best network control system on Linux, do the same thing. If network manager's running and I get disconnected then I see the wireless network icon disappear and the wifi LED turn off. After a few seconds the Wifi LED comes back on, the Wifi icon comes back and it tries to connect. If it doesn't work then it happens again. It's depressing.

So there's one reason I think computing is in the sorry state that it is, people are being conditioned to think that "computers crash" (which conveniently keeps the cost of quality control down), that they aren't "using them properly" (which conveniently keeps the costs of having good designers down) and that they're destined to always be "clueless with everything computer-related" (which conveniently keeps people upgrading stuff they don't need to on the advice of the 'experts' selling the upgrades). This was caused by the proprietary software world, since with Free Software the volunteers, users, developers, companies and organisations which make and sell it actively encourage all users to "hop the partition" and 'scratch their own itches', since it results in less work and more Free software for those doing the encouraging. Whether it was intentional or not is debatable (never assign to malice that which can be explained by (in?)competance).

This unfortunately means that people like my Mum have some kind of internal off switch which is activated by the word "computer", so that when things like the broadband packages they are paying for are discussed with a sentence like "This one has a limit on how much we can do per month, so they'll charge more if we go over it, but this one doesn't" are met with a response such as "Well you know I don't understand these things" (which is the exact same sentence used for every attempt at explaining something, no matter how basic). It makes me *REALLY* frustrated when people don't bother to apply mentals skills which even five year olds possess, simply because they know computers are involved. Discuss the exact same thing with phone contracts, or even the price of meat per kilo, and they'll readily discuss the merits of each option, and even go into the small print, but with computers they've learned to be helpless, and thus think they have no control over anything related, and feel much more comfortable being extorted by monthly bills twice the size of what they could have, where the value calculations are worked out by someone else, than they do with having to confront some computer-related thinking for a few minutes.

Another big cause of computer-helplessness is a genuine problem with computing today, Free Software or not. Empirical evidence does say that just the presence of control, whether or not it is used, is the important bit (like my access to the source code for everything I use), but it's still a chore to actually make use of that control.

As an example, a few years ago the Nautilus file manager changed so that icons got bounding boxes. This meant that before the change if I clicked in a transparent corner of a circular icon then nothing would get selected, but after the change the circular icon would be selected because I'd clicked within the bounding box. This is a good thing usability-wise, but I was rather annoyed with the way it interfered with my specific setup. I had cut out images and assigned them as icons to the various folders in my Home folder, stretched them rather large and arranged them manually so that they filled the Nautilus window without overlapping, so that clicking the visible parts would select the icon, whilst clicking on a transparent part would 'fall through' and select one visible below. I was very proud of this, and it had taken quite a while to do. Then, after an update, all of the icons got bounding boxes and thus clicks in transparent areas no longer fell through, making selecting and double-clicking things unusable. I had to make all of the icons small again, and arrange them in a grid, destroying the previous awesomeness. I took it upon myself a few months ago to bring back the no-bounding-box Nautilus as a well-buried option, so I got the source code to the most recent version of Nautilus and looked through the version control history to find out when the change was made which added the bounding boxes (I think this is where it changed http://svn.gnome.org/viewvc/nautilus?view=revision&revision=9123 ) and replaced that section in the latest code with the old code, and it worked. However, this took a few days, since I've done very little C programming and never used GObject with C before, and I didn't even have to write any code (it was just copypasta). If I want to fix every bug I find it would take an intractable amount of time, even though I can fix any bug I want to individually.

There looks to be some promising stuff going on to rectify this at the Viewpoints Research Institute, an organisation funded by the US government with awesome Computer Scientists like Alan Kay. One of their aims is to "reinvent" computing, which basically involves making (yet another) computer system, but one which is as understandable (and hence small) as possible. They're aiming for a complete, working system in under 20,000 lines of code (for comparison, Windows has around 40,000,000), and have got some nice tools like their "Combined Object Lambda Architecture" programming system, which aims to be written in itself and be able to compile down as far as FPGAs (ie. rewiring the microchips themselves to represent the program), and OMeta which allows very compact and easy to understand programming language implementations (for example they have an almost-complete (missing "try"/"catch" and "with") Javascript interpreter which is only 177 lines of code), which, like their COLA, is written in itself. This allows COLA-based implementations of other languages to make their system, with new languages so easy to define that each part can be made in a tailor-made language, even being redefined in places where it makes things more comprehensible.

Hopefully having more understandable and approachable code will mean it is easier to find and fix bugs, so that nobody has to experience them for long. It might also help to reduce the number of people who teach themselves to be helpless at computing, although as for the ones who are already learned-helpless it will take a lot of effort on their part to break out of it, which won't be helped by proprietary companies trying to dress up their shit code as some kind of magical snake oil which cannot be obtained from anywhere else, or be written by mere mortals (which GNU set out to disprove with UNIX and has done a pretty fine job), and the media displaying binary all over the place whenever the internals of computers is mentioned.

OK, I think I should carry on revising now, as I've gone off on a bit of a rant, but damn it my blog's still not boring or crap! :P

Retarded = backwards

2009-01-30T17:02:00.002Z

It seems that our literally retarded government is mulling over the idea of a "broadband tax" of 20 quid per year for everyone with broadband (which is actually everyone in the country, since they also want universal broadband access) which will be given to the "music industry" and the "film industry".

With this in mind, perhaps now is the time we can finally recover our failing Ice House economy with Ice House taxes on all fridges? Our TV channels and film studios can receive tax on VHS tapes and DVDs? Our dwindling equine propulsion industry could benefit from a tax on all internal combustion engines, electric motors, petrol and diesel. Our whale oil industry could benefit from a tax on light bulbs, flourescent tubes and LEDs. The typewriter industry could have a tax on all computer sales, as can gramophone makers. Whilst we're at it, why not charge for Wikipedia and give all of the proceeds to the desperate Encyclopedia Britannica?

In fact, to encourage these ideas I propose that everyone who uses a fridge, a video or DVD player, any form of engine-powered travel, electric lighting or computer should be labeled as a criminal, since it is completely within my rights to make such accusations when there is no legal basis at all for it. Actually, criminal is too light a word, afterall some crimes are legitimate in certain circumstances when they're the lesser of two evils. The label I propose therefore must be something that's never legitimate, so that juries can have their minds made up for them beforehand rather than having to go through that tedious business of deciding guilt (which, after all that effort, might not even get me the result I want!). They should be called some kind of word which implies rape, murder and that sort of thing... how about serial killer? Yeah, that works. OK, now on with the spreading of my message with twisted logic, brainwashing and of course my all time favourite, the outright lie.

Here's one to get started with:

"You wouldn't strangle a toddler

You wouldn't stab a pregnant woman in the womb

Refridgeration is murder

Murder is a crime

Don't let the serial killers get away with it

Copyright the Respected Icehouse Association of America"

Or how about a more subtle brainwashing?

"He's the, kind of man that makes edits in-place,

He kills post men and shits on their face,

He wanks in the sandwiches you leave in the icebox,

He makes documents using a word processor, what a fucking cock,

He's a, Nerdy Nigel, a Nerdy Nigel

Nerdy Nigel word processes his documents

(Copyright the British Typewritographic Institute)"

Or, perhaps, we should actually EMBRACE technological innovation? ESPECIALLY innovation which looks set to destroy the crumbling monopolies of the 20th century's "music industry" (ie. the few rich sods who decide that you don't want to listen to the vast majority of bands and thus give millions to Britney Spears whilst decent acts end up working in McDonalds) and "film industry" (ie. the few rich sods who decide that potentially good, inventive ideas are too risky to back compared to more sequels of the same old shit). Giving them such a "tax" is not only COMPLETELY disruptive to the economy, but offers NEGATIVE incentive for them to do stuff, since their income could come straight from the tax without wasting any money doing any of that 'making stuff' kerfuffle.

Labour need a firm kick to the teeth for all of the bullshit they're shovelling over us. They're as conservative as the Conservatives, leaving the Liberal Democrats as the only viable way out (and that's still rather tenuous). The problem with the Lib Dems is that they don't seem to have any morals either, not in a 'Fuck you, I'm in charge' Labour way, but in an 'OK as long as you vote for us' way. I think forming policies specifically to cater to a minority so that they'll vote for you, regardless of how it affects the majority, isn't a particularly good thing (here I'm not using minority in an ethnic sense, but for instance their stance for legalising cannabis. Whilst this is obviously backed wholeheartedly by that minority who abuse cannabis, it's impact on the majority of the population is far from clear cut).

On a side note, I really really really wish that Java dies a quick death. It's a fucking terrible language, restrictive, verbose, full of boilerplate bollocks, full of glaring errors which, for some ungodly reason, are standard, making every Java environment broken (either in the sense that they're nonstandard but work properly, or they follow the standard but are full of fucking retarded shit like 2147483647 + 1 = -2147483648 (which *IS* an error, as anyone over the age of about 5 can tell you)). Object oriented my arse. What the hell are these "base types" then? What about methods? What about classes? Fuck off Java.

The End

PubSubClient back in development

2009-01-25T01:57:00.002Z

I've been messing around with my pubsubclient library again, and the issue with publishing now appears to be fixed (either thanks to me or thanks to the ejabberd team if it was a bug in the server).

Anyway, it's working again, and it's now an opportunity to get more reply handlers and documentation written. Since I'm currently in the middle of my exam period this obviously should not be a full-steam-ahead effort, but I'm going to keep chipping away at the TODO list (which is now formalised into a not-yet-up-to-date SPEC-COMPLIANCE file). It's now just a case of putting in the work, since it's not hard thought involved, it's just extracting data from XML and putting it into sensible, pure Python representations.

The website is still around at http://pubsubclient.sourceforge.net/ and the code is still on GitHub at http://github.com/Warbo/pubsubclient/tree/master

Enjoy :)

The Importance of Transparency

2008-11-12T19:57:00.004Z

I know some people who read this can't be arsed with the technical posts, but I do use this blog to tell the world, including my friends, what I'm up to, so please bear with me :)

People > Data > Code > Hardware

That sequence represents two things. Firstly, if those are treated as arrows, it shows how computer programs are generally used. A person inputs some data and the code does something with the data by running on the hardware. Another way of looking at it is as an inequality. People are more than data, data are more than code and code is more than hardware.

Hardware is a lump of plastic, silicon, germanium, steel, etc. It only exists to run code, therefore code > hardware.

Code only exists to manipulate data, whether those data are numbers in a calculation, images to be displayed, music to be played, messages to be sent, etc. Therefore data > code.

Data is only kept around because it is of use to people. Despite our best efforts, hardware and software cannot appreciate the humour of a LOLCAT image. Therefore people > data.

This relationship can be seen in many areas. If I have the most awesome server ever, nobody gives a crap if my Web site is crap. Google's search engine started life on incredibly underpowered, unreliable hardware, but nobody noticed because the code was redundant and reliable. Mugshot.org may be coded better than Myspace.com, but nobody uses Mugshot and there are far more data in Myspace.

Hardware doesn't matter, so as much as possible should be as cross-platform as possible. I have Linux running on my desktops, my laptops and my 'phone. Windows will only run on x86 and x86-64 machines, which means no phones, no PDAs and very few embedded devices like set-top boxes and games consoles. If code is cross-platform then users don't need to give a shit about hardware, which makes life a hell of a lot easier.

Code doesn't matter as much as data, so as much as possible should be in standardised, implementable, documented formats. The spreadsheets I write in OpenOffice.org also work fine in Gnumeric and have live copies saved on Google Spreadsheets. Spreadsheets made in Microsoft Office 2007 can only be opened in Microsoft Office 2007, since everyone else's attempts at compatibility are flawed. If data is openly standardised then users don't need to give a shit about software, which makes life a hell of a lot easier.

This just leaves people and data, which are the only things that are important (code and hardware are just tools used by people to manipulate data).

Using the examples above, I can save a spreadsheet on my desktop and access it from anywhere in the world via Google Spreadsheets in the browser on my phone. The proprietary alternative is to only be able to use Microsoft Office 2007, which requires Microsoft Windows, which requires x86/64 hardware. A very cosy position to be in for Microsoft, but for the vast majority of the world who are not Microsoft employees, why give up so much? This isn't just a feature argument either, since Microsoft could make a browser-based spreadsheet system. The argument is WHY DO I HAVE TO WAIT FOR MICROSOFT? If you hand someone the keys to your data, you should expect to be taken for a very long ride, at the end of which you might not even have that data any more.

The same goes for Facebook and other proprietary applications. (Free Software doesn't always use standard formats, but the formats are at least documented to some small degree in the code. Proprietary apps give you no code.)

END COMMUNICATION

Medibuntu :P

2008-10-28T15:03:00.004Z

Just read that the one of the GNU project's Free Software desktops, Gnome, is being used in a new machine which uses flesh elasticity to find breast cancer. That's an awesome reason to hack on Free Software, since its impact on the world literally IS saving lives :)

The company, SuperSonicImagine, aren't just (ab)using Free Software because it's royalty free, like many companies do. They use the Cairo vector imaging library, amongst others, in their own software and contribute patches back upstream so everyone benefits from their improvements :)

Also, if you're in a recent version of KDE 4, try opening the Akregator feed reader, opening some Web pages in its embedded Konqueror part, open Epiphany, then drag a tab from Akregator over the panel entry for Epiphany until it switches, then drop the tab into epiphany and watch it open. Pretty nice integration going on :D

Amarok and roll

2008-10-18T18:40:00.003+01:00

Just installed Amarok 2 via the Neon nightly build service (for Kubuntu, but still works on Debian Unstable :D ). I have to say, it is absolutely teh awesome! It really shows what KDE 4's "pillars" can do when used in combination.

Don't get me wrong, I've used Amarok 2 before. However, the first time I tried it it kept crashing and didn't have much functionality. My attempts since have been greeted with a feature-packed player, which couldn't actually add anything to the playlist other than Internet radio streams :( With the nightlies it seems to be rocking full steam ahead, and doesn't take up too much space either :)

The only problem currently is that the Qt it uses doesn't seem to have antialiasing enabled for text, but that's a very minor issue.

Now to get Nepomuk to run without making my machine crawl. Perhaps the Java backend would make it work faster?

I CAN HAZ OPENMOKO?

2008-10-09T13:52:00.002+01:00

Firstly, thank you to everyone who gave me mobile phone advice. I've probably disregarded all of you with my eventual purchase, an OpenMoko Freerunner :D

The Freerunner is pretty nice, except of course that every piece of software on it is an utter pile of unusable fail. However, I paid for the phone and not the software, so I'm going to try and make some decent apps for it.

At the moment I'm getting responsibilities piling up though :( I've got homework assignments, Free Software Society organisation, Computer Science Society collaboration to manage, Access Space, ShefLUG and Manchester Free Software collaboration to manage, Sun Microsystems relations, out-of-hours programming for my course to catch up on, relationships, general society stuff like RockSoc and things, Free Software programming (some of my projects are in dire need of some time, which I simply don't have), I have a pile of books which I want to read, the constant barrage of email, XMPP and RSS, going to the gym (which I still haven't done), sorting water stuff, sorting rent and contracts, sorting legal bollocks, sorting TV license, sorting the electoral roll and to make matters worse I'm rather ill, resulting in painful sneezing and coughing fits (I've even had to take some days off, which means missing lectures, which means catching up :( ).

I think the way to tackle this is delegation. The Free Software Society takes up a lot of my time, but is also full of awesome people who gladly help out. By getting these people to help out, like Arthur organising the socials, it will result in a lot more free time in which to do the other stuff.

Still, I've only got the one life, so why waste it? It does annoy me though, when I spend two days attempting some Physics homework and make absolutely no progress :'( That's at least 12 hours I could have used to get something else out of the way, but unfortunately I fail at Physics too much. The solution is, of course, to work even longer on it. Bugger.

Ah well, I don't have time to blog away so I'll wrap this up now.

By the way, PubSubClient is now hosted on GitHub rather than Gitorious due to a request in my blog comments.

Until next time!

On People

2008-10-05T01:56:00.002+01:00

People are intelligent, sentient beings.

If you try to deny the intelligence or sentience of any human being then I suggest you give me an incredibly wide berth.

Sentience, in my mind, is the only thing of any value in the entire Universe. I don't give a crap how well you can recite Pi, or tell funny jokes, or how awesome you are at anything. If sentience is at stake, I'd bet you and I together in an effort to save it.

For those who belittle the sentience of others, I severely wish you're being ironic, or you may meet an untimely end at my hands. Yes, that is a death threat, but laws don't mean anything compared to the continuation of sentience, so I would gladly accept whatever temporal punishment may be wrought against me in a fascism such as the UK if it means an ontherwise unobtainable continuation, or possible continuation, of sentience.

Some people are crying out for bricks to reshape their skulls. However, such sentient beings are part of the solution, rather than the problem (for the time being), so the best course of action is to disassociate oneself with acts one finds disagreeable.

I hope that offers some useful philosophy for someone, as it is useful to me.

Brains are Object Oriented

2008-09-21T04:54:00.002+01:00

Following on somewhat from my browser UI rant, an important thing to realise when designing interfaces is that the brain is used to realistic, real-world things. It gets confused by magic, tries to reject Quantum Physics, and so on. In other words, the brain is Object Oriented.

To be Object Oriented simply means that there are a bunch of "things" which can each do stuff and each have stuff. For example, a ball is a thing. It can bounce and roll, and it has size and colour. Being Object Oriented means breaking apart some things which seem to logically go together, like verbs (or in the case of a computer program, functions). However, with really huge projects it is clear that separating like from like and instead combining things based on context (into an object) is less confusing, since this is more understandable to our brains.

With user interfaces there is magic and quantum "spooky action at a distance" which the brain does not like. This is where changing one thing affects another. For instance, the up and down arrows on a scroll bar. These arrows are not connected to the slider, so how do they manage to move it up and down? By magic. How often do you drag the slider on a scroll bar rather than press then buttons? This is because the brain thinks of moving down as a method of the slider, rather than the arrow being a function which moves the slider. For this reason I propose that arrows be taken off scrollbars (some themes to it already).

Another problem is toolbars. They contain precisely that, tools. In the real world when we want to join two pieces of wood together we think about putting a nail through them. In order to do this we get a hammer, but that's just an implementation detail. If we could push the nail in then we would do. Thus toolbars are bad because they are a kind of magic. They contain functions, ie. they perform actions, like save(document). However, the interface *should* provide methods, ie. the documents should be able to perform the action, like document.save().

A key thing to realise is that your application *is* the GUI. Therefore your GUI should be Object Oriented as far as possible, and should itself act like an object. Developers often think of their code as being the heart of an application, with the graphics as a glossy veneer on top. Users, however, think of the graphics as being the heart of an application, and the code just sticks it together. There are far more users going to be using your software than developers (you should hope :P ) so keeping this in mind is important.

Video or it didn't happen

2008-09-21T03:57:00.002+01:00

I've made a screencast to show some functionality that PubSubClient provides. Doesn't play in SWFDec at the moment, so you might have to use something like this. I accept Google Video's terms of use, but not YouTube's.

The video shows the Browser test application being used to find the nodes on a server and their sub-nodes, add and delete nodes, and add and remove node owners. I've been busy tonight, so that the display issues which cause me to close and reopen windows to show updates are fixed, I've got the Owners functionality working for Publishers and Outcasts too, and I'm mulling over a Subscriptions tab in the node properties (awkward since it may involve message stanzas, not just iq stanzas), generating the contents of the Metadata tab dynamically based on what's defined and changing the access model and other configuration options (awkward since it involves requesting, submitting and cancelling XMPP forms).

For now though, I'm getting tired so it'll have to wait.

PubSubClient has a home

2008-09-20T17:03:00.003+01:00

I set up a Gitorious repository for PubSubClient a while ago, but couldn't actually put any code in there because my SSH key wasn't working with the pass phrase I remembered (ie. I forgot it :P ). Now I've revoked that key and added a new one, so I can upload again :D You can find the complete source there. I've also created a Sourceforge project for it, so that I can give it a free website (Google Code hosting doesn't allow GNU Affero GPL licensed software :( ). The project is still pending approval, but I feel confident that it'll get approved since it is pretty novel.

The repository contains the library in its current form, as well as the ever-improving "browser" test, the currently-stalled "reader" test and the not-really-started-yet "writer" test. To use the library itself all you need is lxml and xmpppy, then just stick the "pubsubclient.py" file in the same folder as your application and you're ready to go. The tests make use of some other libraries, since I wanted to demonstrate that it is easy to integrate into any existing toolkit. To use the browser you'll need pygtk and kiwi, the reader needs pygtk, kiwi and webkit, whilst the writer isn't worth bothering with ATM.

Have fun :D

PS: In other news Jabber Inc. have been bought by Cisco, which is pretty cool. There's some confusion though so remember, "Jabber" is just a nickname for the messaging protocol. Its actual name is XMPP (eXtensible Messaging and Presence Protocol), since it may have originally been created by Jabber Inc. a decade ago, but for the past few years it's been looked after by the XMPP Standards Foundation. Jabber Inc. (who own jabber.com but NOT jabber.org) can of course add to and change the standard, but they must follow the same proccess as Google, Microsoft, Facebook or anyone else who wants to do so. In other words: This is a Good Thing(TM) since XMPP is being adopted heavily by a huge firm like Cisco (who previously used the SIMPLE messaging standard, which is anything but), and the XMPP standards are still being kept safe by XSF as they were already. Rock on!

Spot The Difference

2008-09-20T00:54:00.006+01:00

Here's Google Chrome on Fedora:

Here's Firefox with a "Google Chrome theme" on Fedora:
What's the WTF-were-you-thinking mistake? Can't figure it out? How about if I tell you it also applies to the default Firefox theme on OSX:

But not in the default theme for Windows:

Or Ubuntu:

Still can't see? How about an analogy:

Still don't understand? Let me spell it out:
Chrome gets this (as does Opera, but I don't give a shit because their browser is still proprietary), but this supposedly-Chrome theme (as well as the OSX theme) have made the bad UI of Firefox EVEN WORSE. The default Windows and Gnome themes have tabbed pages, with some unknown, magical force changing the address bar when switching tabs. This is BAD since it requires either reverse-engineering, or study of the code, to find out the behaviour. However, with those 'upside-down' tab themes the tabs are now controlling the address bar, with an unknown, magical force changing the page content accordingly. This is EVEN WORSE since it is the page contents that people actually care about! Hype Firefox all you want, but I prefer looking at Web SITES to looking at Web BROWSERS.

In short: EPIC FAIL

Wanna Be A Programmer?

2008-09-19T11:39:00.004+01:00

Computers are all around us, and our lives are to a large extent controlled by them. Your bank balance, your phone calls, your email, messaging and Web browsing, games, wages and bills, music, video, literature and image production, trains, planes and cars, it's all controlled by computers.

Software is the stuff which controls computers, and therefore our lives. So who controls the software? Well, if you deal with proprietary software then it's usually controlled by some foreign corporation trying to wring money out of you. If you deal with Free Software then the control is still in the hands of the developers, however YOU can choose the developers. If you don't like what the original developers are doing then you have every right to fork the software and get some other developers to work on it and take it in the direction you want. Often this involves paying the developers a wage, but there is also another option: YOU can develop it yourself! The only skill you need is computer programming, which is a skill that can even be picked up by young children, as long as they have some exposure to it.

So how can YOU learn to program? Well if you're a complete beginner I would follow these steps:

1) Forget EVERYTHING you think you know about programming.

It's most likely bollocks, perpetuated by the media's obsession with stereotypes and putting flashing, unintelligible crap all over any computer monitor that appears on screen, and the proprietary software companys' desire to look like the crap they serve is somehow magical and unequalled. It's not. Programming is easy. It has to be, because computers are so unfathomably stupid. Just because Turner is able to create beautiful seascapes doesn't mean that a two year old can't scribble with a crayon, in the same way that simple programs are easy despite huge ones being hard.

2) Learn to think like a programmer.

2.1)

A nice way to get the hang of this is to play games like this one (NOTE: You may need to install SWFDec to play that). Light Bot features a very simple programming language to control a robot, it is easy to use since each command is an icon and the program (called "main method") is a grid which you drop the icons into then press Go (the program runs like English text, ie. starting at the top left and going across, then going down a row starting at the left and so on). The nice part about it is that the program can only be a maximum of 12 commands long, which is not enough to complete some levels, but there are two functions (reusable blocks of code) available to make repetitive things take less space.

2.2)

After you've completed that game a few times (there are 12 levels) you can go on to Guido van Robot (this may be in your package manager already), which is very similar to the Light Bot game but uses typed commands rather than movable icons, allowing you to make much longer programs. Even without Light Bot's restricted space it is still important to use functions like you've learned, as they are the best way to make large programs more manageble and less confusing. Transitioning from Light Bot to Guido van Robot is an important skill, since it makes sure you know the principles of programming (commands, functions, repetition, etc.) and how to use them, rather than just learning one specific programming language.

2.3)

After you've mastered Guido van Robot and can comfortably tackle any problem without difficulty or confusion, it's time to move on to something a bit less rigid in structure, like the old favourite Logo (there are a number of different Logo programs out there, you can try XLogo and UCBLogo). Logo is a programming language in its own right, but is often used with the "turtle" graphics system, a very similar concept to controlling the Light Bot and Guido van Robot (it can actually be used to control a real robot, but I'm assuming you don't have one :P ). The turtle can be moved to any pixel on the screen by giving it commands like FORWARD, RIGHT 90, and so on, plus it can draw a trail as it goes, allowing you to draw images. Taking advantage of the incredible speed of modern hardware you can make animations, by drawing lines, erasing them, then drawing them again in a slightly different way. Give yourself some problems to tackle, such as drawing a circle, making a function which can draw a circle to a given size (this uses the concept of "arguments"), drawing a spiral (this uses the concept of "recursion"), a stick man, an animation of a bouncing ball, making the ball squish slightly when it bounces, a windmill with turning sails and a growing vine. All of those are possible with a little thought, but *remember to use functions* and leave yourself *a lot* of comments throughout your code to remind yourself wha each bit does. When you can make those given examples without much thought you've mastered a lot of the art of programming. Well done :D

2.4)

There are many books out there which can help you learn to program. For instance How to Think Like A Computer Scientist. Googling for guides and tutorials on your language of choice is usually helpful too.

3) Learn one powerful programming language

Logo is a nice little language, but your programs can't really talk to anything else. This requires software called "libraries", and for libraries which are not written in your language of choice you will need a "binding" to access them. I don't know of ANY libraries which are written in Logo, or any bindings to let Logo access libraries written in other languages. This means you'll have to move on to something more powerful :( Since you now know Logo, which is a simplified form of the LISP language, it would be sensible to learn another simple, LISP-like language. I would recommend Scheme, which is a very powerful type of LISP, without a lot of the unneeded crap which exists in LISP. Guile is a very good Free Software Scheme system, which has bindings for many libraries, therefore I recommend your next step is to dive in and learn Scheme using Guile. It is very similar to Logo, except that every command is enclosed in brackets. If you want to make real graphical applications, there are bindings for the Gnome desktop, and even the new Clutter 3D graphics library. Now that you know Scheme you can do anything. Although some things are harder than others...

4) Learn the Object Oriented paradigm

In the same way that functions are an INCREDIBLY useful tool to make programs short, readable, understandable and maintainable, there are a few different progamming 'paradigms'; ways to structure and think about your programs and how they work, which will make your life easier and avoid getting headaches.

The first paradigm you used was "procedural programming", where the code describes the procedure to follow, "do this, then do that", but there are many more, some of which you've touched on. "Object Oriented" programming (OOP) takes functions one step further and groups them together into so-called 'objects'. To run a function you must ask the object which contains it. Objects have a type (a "class"), which can be a subclass of another class. For example "dog" might be a subclass of "animal", so that every function which "animal" can do ("breathe", "eat", etc.) a "dog" can do implicitly. Every object, for instance "Lassie", is an 'instance' of its class, for instance "Dog". Objects can also implement different roles (called "interfaces"), like "Lassie" can implement "Entertainer" and "Pet" (since not all "dogs" are "entertainers" or "pets"). By dividing a problem up into the different classes and objects it involves, the program becomes as simple as describing how they interact (eg. "lassie.fetch(big_stick)"). Learning the Object Oriented paradigm will make your life a LOT easier when you start to make large programs, but it is not very easy to do on your own.

5) EXPERIMENT!

If you have an idea or a question, try it out and see what happens! You can ONLY learn through experimenting and playing, there just isn't another way to do it.

6) Learn more languages

Scheme isn't the only kid on the block. There are many other languages you may like to learn. Good choices would be Smalltalk, Ruby, Python, Javascript, Vala and, if you're feeling masochistic, C, C++ and Java. The more languages you learn, the more approaches you learn to tackle problems with and the more you learn about why certain languages do things in certain ways (and therefore how to use them more efficiently). What may be hard in one language may be easy in another, so having a substantial set of knowledge is always a good thing.

7) Contribute to Free Software!

It might seem a daunting prospect to make your own Web browser, file manager or music player, but there are already many such projects out there with the code fully available for you to download and play with. Every Free Software project welcomes new developers, even if you're just fixing a bug which really annoys you. If you contribute to a widely-used piece of software like Firefox or Gedit then your changes, if approved by the maintainers, will be used by millions of people worldwide. If your changes aren't approved, but you would still like to make them available, you can keep them in a distributed version control system like Git or Bzr, letting you cherry-pick the upstream changes you want, and keep any changes you make yourself, and also let other people get your modified version.

Most importantly, don't be scared by all of the complicated stuff you see computers doing every day. Computers don't get harder and harder to program over time, they get easier and easier, since all of that fancy stuff has *already been written*! You might not understand how an image gets from the hard drive to the screen, but you don't need to know or care (I don't!), other people have already written libraries to do that. You just have to say something like "display(my_photo.jpeg)" and voila!

With computers becoming more and more integral to our way of life, I encourage everyone to at least attempt following some of those steps, if only to realise that computers aren't unknowable, magic things; they're understandable, controllable and most importantly they are changable. Posessing an attitude (like my Mum) of "I don't understand these things" is retarded. I didn't know how to piss standing up until I tried it.

...And Laws Were Most Numerous When The Commonwealth Was Most Corrupt

2008-09-18T12:51:00.002+01:00

Personally, I would say that power corrupts more than TV, miniskirts or saggy jeans.

I do, however, completely agree with Nsaba Buturo's statement "What's wrong with a miniskirt? You can cause an accident because some of our people are weak mentally,". He could've given an example if there was a mirror nearby.

The Effects of Promoting Propaganda

2008-09-17T18:10:00.002+01:00

Richard Stallman and the Free Software Foundation are often the butt of jokes due to their campaign of explaining "GNU/Linux". The problem I have with their tactics is that they bring the issue up where it isn't particularly appropriate, for example in the recent Steven Fry video. There was no need to blabber on about such unnecessary bike shed issues in a video that is solely written by the FSF (ie. there's no need to correct anything, since they control its contents). I do, however, agree with the basic reasoning they have, that the phrases and words we use can shape our minds, and especially the minds of those we are talking to (if they are new to a subject).

In their case it is the use of the name "Linux" to describe an entire computer system, where Linux is only the kernel, and the use of the phrase Open Source instead of the phrase Free Software since the former promotes an efficient way of programming whereas the latter promotes the freedom of computer users. An example more common to people who aren't me would be the gross misuse of the word "pirate", used instead of "copyright infringer" in a context which implies that infringing copyright is akin to looting, murdering and raping on the high seas, and thus biases the minds of those who hear the phrase (murder, theft and rape are obviously wrong and criminal, and very serious matters indeed, whereas not only is copyright infringement an intangible and often debatable issue, it is not clear whether infringement is even wrong, despite its status as criminal).

One term a lot of people who know me will know my chargrin at is "secured" and "unsecured"/"insecure" when describing an 802.11 wireless computer network. The phrase is used to describe WEP and WPA encryption, but completely biases the minds of people who use and hear it. Deciding whether to allow or disallow devices to connect to a network is a matter that can be debated by rational people, whereas framing such a debate into choosing a "secure" network or an "insecure" network can easily end all discussion in favour of the option labelled "secure", ie. restricted access.

Such framing is plain wrong: First, a Virtual Private Network (VPN) can be used over any network, even an "unsecured" one, to establish secure communication between machines. For instance, a wireless access point can be used without encryption, thus allowing anyone who wants to to connect, whilst certain computers (for instance the tennants in a house) can use a VPN on top of this network to allow each other to access shared files and networked printers and nobody else on the access point can access things. A network could even be set up to allow the VPN as much bandwidth as it wants, giving what's left over to anyone else (so the tennants wouldn't even realise other people are online, for example, whilst people in the area can get free access to the knowledge and commication abilities that the Internet provides, although at a reduced priority).

Second, WEP encryption was broken years ago, thus any network which is "secured" by WEP actually has no security at all. Any device can use a Free Software tool like Aircrack to monitor the network automatically until it gets enough data to present the user with the encryption key being used.

The reason I'm going on about this is a very serious one: It might soon become illegal in India to allow people to connect to an 802.11 wireless computer network. Now, of course those behind this aren't using those words, they are saying "unsecured" networks will be illegal.

Oh yeah, it's also because those who use "unsecured" wireless networks are apparently terrorists. Since the T-word is in there I wouldn't be suprised if this does become law and sets an unnerving precedence.

BTW, I just noticed this (both by Uli Kusch though):

You may need to install SWFDec to watch those videos.

I am teh awesome

2008-09-12T13:09:00.003+01:00

Seems I was overly cautious in my last post, since node deletion now works :D

Next I think I'll work on affiliations, which might take some thinking to do sensibly...

The Power To Change

2008-09-12T10:16:00.006+01:00

As I've blogged about before, I'm currently working on a general purpose Python library which implements an entity in the XMPP/Jabber Publish/Subscribe specification XEP-0060. The library itself can currently send equivalent stanzas to all of the examples given in the specification. It does not yet handle the replies from everything, doesn't handle incoming messages (ie. updates) and its error handling consists of printing the error stanza to the terminal. However, at around 1,500 lines it's already a bit too much to grasp with only a specification to go on. For this reason, once I had implemented every needed method I decided to take a pragmatic approach to the reply and message handling.

This originally started as a PubSub reader application, similar in form and function to current news reader applications like Akregator and Liferea but using PubSub nodes instead of RSS or ATOM files. This proceeded quite nicely until I hit a bit of a wall: There wasn't anything to read!

To fix this I started to write a PubSub writer application, similar in form and function to current offline blogging applications like KBlogger and Gnome Blog but once again using PubSub nodes instead of RSS or ATOM files, or Web service APIs. This was a bit premature of me, however, since there was nowhere to write things TO, and then I came across x60br and thought it would be the perfect kind of tool to make next, thus I started a simple PubSub browser.

The concept of the PubSub browser I am making is simple, and I have blogged about it before. You're presented with a window containing minimal controls. Into a text box at the top the address of a PubSub-capable server is put, then the "Get" button next to it is pressed. The browser then adds this server to a tree view and uses the PubSub library to send a message to the given server, asking for a list of the nodes it contains. When a reply is received any nodes are added to the tree in a level below the server. Each of these nodes is then queried for any nodes that they may contain, and when the replies come in they are added to a level below their respective parents. This isn't quite a perfect way of doing things since it doesn't take into account the loops which the PubSub system allows, but is good enough for this tool's purposes.

With a tree of nodes available the next step to take was implementing some ability to alter the structure, ie. add, remove, change parents, etc. I've now achieved this, as can be seen in the following screenshot of my current ejabberd server.

Until this point the choice of server hadn't mattered to me (it's all standardised), and since ejabberd is a) packaged in Debian and b) all the rage these days, I figured it was worth using. There are currently VERY few servers which can actually understand PubSub, the most popular being ejabberd and OpenFire (the former being used by jabber.org and the latter being used by the University of Sheffield), but since it's such a new technology there are issues arising with its server-side implementation (which my library is not designed to address). Essentially ejabberd is currently a bit broken, the most important deviations from the standard being that it allows items to be published to a collection (analogous to writing text to a folder, rather than to a file inside the folder), it allows leaf nodes to contain children (analogous to a file containing other files and folders (OK this is possible with archives, but that's not the point :P )) and it doesn't allow subscribing to a collection. Aside from those, a rather obvious restriction (although still standards-compliant, as far as I can tell) is that a rigid filesystem-like structure must be adhered to, with node names following the structure /home/servername/username/... This means that pressing the Add button when the server, /home, /pubsub or /pubsub/nodes is selected will result in a failure, and with /home/localhost selected only one name is allowed, /home/localhost/test1 (since the browser is logged in as the user test1). However, it does work as can be seen (I've created the node /home/localhost/test1), and in the process I've managed to delete quite a lot of code from the library (note for non-programmers: The more code you can throw away, the easier your life becomes :P )

Next on the agenda is deleting nodes. If I can't get that done today, however, then I probably won't get it done until Wednesday. I'm going to see Jooooooooooooooooooooooooooooooooooooooooooooooo...ingtons.

PS: If you want to get the code there's a little difficulty at the moment, since I can't seem to sign anything :( I've set up a git repository here, but at the time of writing it's empty due to the aformentioned difficulty.

Model-View-Controller

2008-09-11T10:25:00.003+01:00

First there was Last.fm's site redesign, now there's Facebook's. Both have been given a hard time, however I don't personally give a flying fuck about their website design. The *ONLY* useful thing from last.fm is the artist information. Fuck the personal statistics. Fuck the Shoutboxes. Fuck the Friends. The *ONLY* reason accounts exist is to prevent spamming which would skew the artist information database. So, do you want to see how last.fm REALLY looks? Well there are a few styles: text and XML. There is also an API (Application Programmer's Interface) available, but it requires an account to use so it can go and die in a bin.

If you are particularly clueless about how computers work then you might be thinking to yourself "those look like shit", and you would be right. However, structure is FAR more important than presentation, and those documents are well structured. The reason thinking along those lines is clueless is because such people have never seen a computer program, they've never pressed the View Source button in their Web browser and they've never taken apart any gadgets in their life. Those of you who have done at least one of those things will know that beauty is only skin deep. A website might look pretty, but the HTML the pages are made of look as ugly as sin. The HTML will have a decent structure though, which means a Web browser can be stuck in between the user and the HTML to make it look pretty.

What I am saying is that last.fm allow access to their database in a structured way, which allows applications (which are completely stupid and need to be told exactly what to do, hence the need for structure) to display the data in whatever the hell way the user wants. You don't need to use their website, since code for putting data into and getting data from last.fm exists inside every decent music player (Amarok, Banshee, Listen, etc.). Your choice of application is completely up to you, and you can keep using that application for as long as last.fm's web services maintain their current structure. If the structure or protocol or something changes, then that's not too bad for anyone using a well written piece of Free Software. If you're using a proprietary program to display it then you're knackered and I hope you've learned a valuable lesson.

Now let's look at the recent Facebook change. Can I access Facebook's database via a well defined and structured interface? Can I bollocks. That means I'm stuck with whatever the almighty Facebook deities bestow upon me, and I'd better pray that I like it because it's all they allow me. Thank fuck I don't use it.

For those of you who haven't realised it by now I am talking about the Semantic Web and the Model View Controller architecture. In the World Wide Web the stuff that gets passed around is HTML. That HTML can contain any data, is layed out in a certain way and the structure is very freeform, as long as it meets a few rules defined in the HTML and XHTML standards. The Semantic Web is different. In the Semantic Web the data is structured in a very specific way, in RDF triples to be precise. There is NO layout in the Semantic Web, since it is not about anything visual like pages, it is about *concepts* and *meaning*. Applications you use which can access the Semantic Web (there are as many as people can create, rather than the Web's singleton the Web Browser) can do whatever the hell they like with the structured information they receive. There's no arguments about the layout and look of a semantic Facebook because there's NO SUCH THING as the "layout" or "look" of anything on the Semantic Web. The layout and look are ENTIRELY up to the user and which program they decide to view it with. Some users may even view it with applications they access via the World Wide Web.

PS: I would just like to point out the first sentence of Wikipedia's World Wide Web article:

"The World Wide Web (commonly shortened to the Web) is a system of interlinked hypertext documents accessed via the Internet." Thank you.

Services, integration, communication and standards

2008-09-07T19:23:00.004+01:00

These days it is possible to do pretty much anything over the Internet. There's eBay, online banking, PayPal, Flickr, OpenStreetMap, OpenDesktop, Email, chat, forums, Wikis, Facebook, scrobbling, blogs, etc. The big problem, in my mind, is that of technological walled gardens.

A walled garden is an area which cannot be accessed without permission, usually by registering some form of account. Facebook is a classic example of a walled garden. Walls are useful, since having everyone's bank accounts available without restriction would be a problem to say the least. A technological walled garden would be an enclosed, restricted area which can only be accessed via certain technological means. Technological walled gardens are often simpler to implement than open systems, but often the reason the garden operator does this is because they see this as a way to run a dot-com or Web-2.0 business.

Let's take an example, Yahoo! Mail, Windows Live Mail and Gmail, which are all walled gardens in the classical sense, an account is needed and the login details must be provided in order to access anything. The first two, however, are also technological walled gardens: whilst mechanisms to send, retrieve, check and manage email have been around for decades, from "get my mail" (POP3) and "send my mail" (SMTP) to more sophisticated "synchronise this lot of email with that lot" (IMAP) and are well defined, standardised, understood and implemented, in order to access Yahoo! Mail or Windows Live Mail you still need to log in via their website because they don't use any of these standards. Gmail supports them, which is how I can use Evolution and Kmail to manage my Gmail account. Yahoo and Microsoft specifically disable them (I know Yahoo used to allow POP3 and SMTP access, when they stopped I moved away from them) with the reasoning that Evolution and Kmail don't display adverts, whereas their websites do. Here interoperability and standardisation desired by customers (if it wasn't used then there's no point disabling it, since nobody would be unexposed to adverts and the POP/SMTP/IMAP server load would be zero) is sacrificed in order to force adverts onto users who don't want them. This of course doesn't even touch upon the flexibility of using an email client (screen readers and other accessibility for the disabled, offline access, complete choice of interface (including Web), etc.).

That is the major reason why I refuse to join Facebook, MySpace, etc. I cannot buy or download Facebook's software and run it on my own machine,and even if I managed to write my own there would be no way to make it talk to Facebook's own servers. Since the entire point of Facebook is the stuff in their database, this would make my software useless. Hence Facebook have created a technological walled garden: If I joined Facebook then I would be sending any data I entered into a blackhole as far as accessing it on my terms is concerned.

Last.fm is better, since although their server/database software is a trade secret (as far as I can tell), the Audio Scrobbler system they use to gather data is completely documented and has many implementations, many of which are Free Software (including the official player). The contents of their database is also available, and doesn't even require an account (I have tried to think of ways to submit similar artists without an account, such as submitting two tracks at a time and building the data from that, but I suppose spam would be too much of a problem). Only the artist recommendations/similarity, tags and thingsare available, but that's the entire reason I use it, fuck the site with all of its Flash, poor social networking, confusing messaging system and stuff, that's all fluff around the periphery of the useful information. Essentially last.fm is like Gmail: I can't get the code which runs it, but I can get my data in and out in standardised ways which can be implemented with Free Software. I could make my own server which synchronises with with their database via the available protocols, and thus get out everything that I put in.

Now, the issue of synchronisation is interesting. How do you keep two things synchronised? There are a few different approaches and each has its place:

Known, unchanging, unmoving data

Here HTML can be used, ie. a web page. This is fine for people, and for applications needing that data it can simply be copied into the application once. An example would be an "about" page.

Unknown, unchanging, unmoving data

Here HTML can still be used, but since the data is not know beforehand it can be hard for an application to get anything useful from it. RDFa can be used inside the HTML to label each piece of information, thus an application only needs to be told what to find and it will look through the labels until it does, regardless of page structure, layout, etc. An example would be a scientific paper.

Changing data which is accessed once at a time

Here RSS or ATOM can be used. This allows changes to be specified in the file. ATOM is a standard, but RSS is a dialect of RDF which means labelled data is possible. An example would be a changelog.

Changing data which is accessed on every change

Here XMPP PubSub can be used. This means that there is no checking for updates since the source will push any changes out to subscribers when they are made. This doesn't use a file, it uses a stream. This is what my library is designed to accomplish. An example would be a blog.

Two-way communication and instruction execution

Here systems such as JOLIE can be used, overlaying protocols like SOAP. This can be used for dynamically generated data like database queries and searches, as well as for sending instructions such as "Approve Payment". An example would be a shop.

Notice that the first technology, HTML, is the only one which needs a Web browser to access. RDFa, ATOM and RSS are all structured in a way that allows applications to handle them directly, no human is needed and thus many more possibilities are available. XMPP can also be structured, since RDF, ATOM and RSS can all be sent over XMPP, allowing machines to handle the data, but doing so in an as-needed basis which makes things more scalable. JOLIE covers a range of protocols which are all inerently machine-focused, they are executing instructions. This might be a "buy this laptop" instruction when a button is pressed, or a "search for this" instruction when a query is entered.

These technologies allow data and communication to break free of implementation and visualisation. The next Facebook doesn't have to be a centralised Web site, it can be a distributed system with many servers run by different people interacting with each other to allow a scalable, unowned network, like the Web but made of pure information without the overhead of layout, styles, widgets or interfaces. This is called the Semantic Web. All of the visualisation, interface and style can be swappable and implemented where it is wanted, for instance as desktop applications on user's machines, or as Web sites running on various servers. There is no reason why, in an interconnected world, I should have to visit Facebook.com in order to interact with Facebook.

Except, of course, that Facebook wants to force unwanted adverts onto their customers.

NOTE TO SELF

2008-09-07T01:34:00.002+01:00

When trying to communicate from a machine to itself, for example trying to run the ejabberd server, it is a good idea to have a loopback network interface. Spent over a week trying to hunt that issue down, now I can finally get back to working on my library :D

Physics + Programming = Pretty

2008-09-05T23:16:00.005+01:00

Whilst looking for a better-than-mine Python implementation of attractive-and-repulsive-based-on-the-inverse-square-law particle simulation I came across this, which is basically a really simple model of gravity. You tell it how many balls you want it to start with, which are then placed at random points on the screen with zero momentum and a mass of 4. Then the simulation starts, running one time step of the simulation and redrawing the screen, then repeating this indefinitely. Particles which collide coalesce into a single particle with the combined mass and momentum of the original. This results in one of three things happening, either all of the masses coalesce into a big, stationary ball (since the net momentum is zero, due to the law of conservation of momentum), two or more masses form a somewhat stable orbit around each other, or one or masses hit the edge of the window during the simulation and 'reflect' (the vector of their velocity perpendicular to the edge is reversed) which breaks the physical simulation by violating conservation of momentum. After this third scenario all hell can break loose, usually resulting in everything coalescing into a huge ball, moving with momentum twice the magnitude of that which was flipped by hitting the wall, and opposite in direction.

Anyway, I looked through the short source file and had a bit of a play. Eventually I gave particles a colour which changes subtley every time they are drawn. I also disabled the old-frame-removing screen fill, allowing each frame of the animation to build upon the previous and create a trail. A little simple colour-cycle logic and I the following patterns appeared to my delight :D

HERE'S THE SOURCE (not properly licensed due to original author. Let's say you can treat is as Public Domain to save complications of licenses)