The Schedule 7 Experience

Intelligent Design

2004-10-18T20:28:00.000+02:00

My recent article about the limitations in the current theory of evolution caused something of a stir. I've been reading up on the matter a bit, and it turns out that this "anti-Darwin"ism has a name, so-called Intelligent Design. This is the new, hipper, term for creationism, that carefully avoids using the bible, but still argues that natural selection is wrong, and that an "intelligent being" (God or Aliens) must be responsible.

It does seem that many of my naive criticisms of Darwinian theories of natural selection fall under the Intelligent Design camp, which is probably not a good thing. (Intelligent Design is essentially a pseudoscience). What has troubled me, however, is the assertion by many scientists, who should be in the know, that natural selection can explain all my concerns, yet they seem unable to back up their claims. So I did some reading. As I've been discovering more and more lately, wiki is a great place to start. That article goes some way to exploring possible explanations for "irreducible complexity". This article is a well written case study of how a particular example of irreducible complexity (hemoglobin) may have arisen through natural selection. It's not totally convincing, and lacking in hard data, but it's definitely plausible.

A recent article in Wired magazine highlights the ongoing battles the Intelligent Design camp are having to get alternative theories to evolution discussed in American schools. It's an exceptionally one-sided article, but has good breadth and discusses many of the role players in the current debate and as such is well worth a read.

Update: This article at talkorigins discusses Irreducible Complexity (IC) in some detail, and essentially dismisses it, establishing that either systems are not IC, or demonstrates ways in which known IC systems could have evolved by means of natural selection.

It identifies possible ways in which IC systems could have evolved:

Previously using more parts than necessary for the function
The parts themselves evolve
Deployment of parts (gene regulation) evolves
New parts are created (gene duplication) and may then evolve.

For instance, the swimming mechanism of flagella may well have involved an arm waving about in earlier forms. This arm may have been insufficient to propel the flagellum, but it would have disturbed the water around it, which in itself is a useful function, in terms of bringing food within range.

Why Your Anti-spam Solution Won't Work

2004-09-08T16:58:00.000+02:00

The Last Word

Skype Reviewed

2004-09-08T07:57:00.000+02:00

People keep asking me about Skype, and whether they should try it out. This
review of Skype is well balanced, pointing out some of the obvious and not-so-obvious flaws.

From my own experience, my broadband connection, which makes use of Wifi technology, suffers from very high latency, and hence renders Skype unusable.

If it's usable on dialup, I'd be interested in hearing about that.

Stored Procedures, what are they good for?

2004-09-01T13:38:00.000+02:00

We're all familiar with the traditional benefits of stored procedures. For instance, this site, which talks about implementing stored procs in mySQL, lists them nice and succinctly:

Better performance Stored procedures are faster because they are pre-compiled SQL code. This reduces the “Compile and Execute” step to just “Execute’ in most cases. Also, only the call to the stored procedure needs to be sent to the server instead of chunks of information – this reduces the information that needs to be sent to the server and acts like a call to a remote procedure. This is an advantage when it comes to code that is called repeatedly. However, the load on the server is another point to consider since most of the processing will now be done on the server.
Easier to maintain Since all the SQL can now be stored on the server, it is easier to make changes to the stored procedure than to a bunch of SQL statements distributed all over the application.
Security Although the use of stored procedures is not by itself a guarantee of security, it can be used to create an environment where applications and users can only access database tables through the stored procedures, instead of giving them direct access to the tables. The benefit is a layer of abstraction.
Optimization When a SQL statement is parsed by the server, it is optimized internally by the server. If a bunch of SQL statements are sent to the server, repeatedly, they have to be optimized each time. The SQL statements in the stored procedure that is in memory have to only be optimized once and an execution plan is created for the SQL statements in the stored procedure.

But, is it as simple as that? Not if you believe Frans. He argues that dynamic SQL is the way to go. He makes some good arguements, for instance, pointing out that for MS SQL Server in particular, stored procs have no special priority over other SQL code, and are in fact compiled at run time (thus countering, to some extent, the performance arguement, although he does ignore the benefit of reduced network traffic).

He also objects to the API that you effectively create with stored procs, which can be a problem when the underlying table changes, as this forces the CRUD procs (at least) to change accordingly, breaking any existing client code.

My take? Having worked on many systems where the DB layer was accessed by many systems and languages, I'd say a fixed API was a good thing. Also, it limits the rewriting of identical (byt error prone) code over and over again.

Aside: many of the arguements for and against ORM tools apply equally well in this case.

Everything you wanted to know about blogging but were afraid to ask

2004-09-01T13:16:00.000+02:00

Fascinating list of blogging tips. I especially like the frequent reminders that this a time vacuum that suck up more time the more popular it gets

Simon World :: Everything you wanted to know about blogging but were afraid to ask: "Everything you wanted to know about blogging but were afraid to ask"

Making Money From Open Source

2004-08-30T10:37:00.000+02:00

This is the first reasonably coherent effort I've found to explain how one makes money by releasing open source software. I'm not sure it really applies to the little guy, but it seeks to explain how people like IBM and Sun can make money by giving away products like Eclipse and Project Looking Glass

Making Money From Open Source

The End of PC's As We Know Them

2004-08-26T10:49:00.000+02:00

Smaller and smaller computers is the current trend (palmtops, cell phones and so forth). The only thing holding them back from being utterly dominant is the awkward user interface - the keyboard/selection devices become hopelessly unusable, and the screens become unreadable.

However, check out this technology

There's no reason they couldn't eventually do this for screens as well.

Java Faster than C/C++

2004-08-25T17:56:00.000+02:00

This fascinating article refers to numerous benchmarks and provides sensible academic reasons why Java is as fast or faster than C/C++ in most cases. The most compelling reason is that the JIT compiler knows precisesly what machine it is running on, and can thus tune the code much more effectively than a normal commercial c/c++ compiler. It's almost unfair, really

Java pulling ahead? Java versus C benchmarks

Copyright vs P2P

2004-08-22T23:08:00.000+02:00

In a remarkable decision for Peer to Peer developers and users, the USA Ninth Circuit recently ruled in favour of Grokster in MGM vs Grokster

from EFF:

...the Court observed that, in the long run, a competive, unfettered market for innovation ends up helping copyright owners (even if it doesn't help today's entertainment industry oligopolists). In fact, today's ruling will likely be remembered as yet another example of the courts rescuing the entertainment industry from its own short-sightedness. In the words of the Court, "Further, as we have observed, we live in a quicksilver technological environment with courts ill-suited to fix the flow of internet innovation. The introduction of new technology is always disruptive to old markets, and particularly to those copyright owners whose works are sold through wellestablished distribution mechanisms. Yet, history has shown that time and market forces often provide equilibrium in balancing interests, whether the new technology be a player piano, a copier, a tape recorder, a video recorder, a personal computer, a karaoke machine, or an MP3 player."

There is an absolutely fascinating one hour mp3 of the oral arguements from both MGM and Grokster/EFF available here. The arguements are very persuavive from both sides, and very clearly spelt out. It really does make all those endless courtroom dramas you see on TV quite dull by comparison, I strongly recommend downloading and listening to it.

Chess Legend Bobby Fischer Arrested

2004-08-21T21:52:00.000+02:00

OK, this isn't exactly new news, it happened about a month ago. But its taken a few twists and turns along the way.

First, lets recap. Back in 1992, Bobby and Boris Spassky had a rematch of their famous 1972 battle. This was held in Yugoslavia. At the time, the USA had imposed sanctions against Yugoslavia, and as a result, issued a warrant for Fischer's arrest. He vanished for a long time, only to resurface in the Philippines after 9/11, to express his delight at the attack at the WTC attack. Not exactly endearing himself to Uncle Sam, then.

Anyway, he was recently arrested in Japan, attempting to travel on his apparently invalid USA passport. Appeals against his deportation are currently underway, although he is now attempting to renounce his US citizenship.

In a recent bizarre development, Boris Spassky has written to President Bush, asking that he too be arrested, and placed in Fischer's cell (along with a chess set).

The End of Secure Transactions Over the Internet?

2004-08-21T20:58:00.000+02:00

Some of you may have heard that someone "cracked the system used by secure sites". In fact, what has happened is that someone has managed to produce a "collision" on SHA-0. This means they have managed to generate two different messages that, when they have the SHA-0 hashing algorithm applied, generate the same hash result. The implications are not fully known yet, but note that they are not able to produce a collision message against a particular message. In other words, they can produce a message M and M' where H(M) = H(M'), but can't currently produce a message M1 against your message, where H(M1)=H(your message)

This site explains it as clearly as I've seen so far. It also suggests some of the implications of the results.

Great Hacking vs Great Writing

2004-07-30T11:26:00.000+02:00

The java blog world is up in arms about Paul Graham's anti-java tone in his recent controversial post.

See here and here for instance.

If you are a passionate programmer, Paul's writing can be very inspirational. His ongoing anti-java bias is, however, very offputting. Regardless of your preferred language, at the end of the day, we're programming finite state automata. Different languages give us more or less abstraction levels, possibly allowing certain concepts to be expressed more succinctly, but nothing can change the fact that all we are doing is manipulating bytes, comparing bytes and diverting to other memory locations based on those comparisons.

Update:

Here is a well written response, the last sentence in particular is very telling.
Here, on the other hand, is a cutting response, and the comments are even better - check out Starsky McFlirt's comments: " Oh sorry, I forgot, he also applied a basic statistical concept to email filtering as well that worked for like at least one year. Somebody give the man a f**kin Nobel prize tout suite. " still has me chuckling.

Open Source Databases Compared

2004-07-27T15:16:00.000+02:00

MySQL, PostgreSQL and Firebird: well-known databases to be sure. But which one to choose? This site provides an excellent comparison and checklist of which features each database supports.

MySQL Gotchas

2004-07-22T08:34:00.000+02:00

This site contains a very detailed list of gotcha's with MySQL (i.e. unusual behaviour not consistent with the majority of other databases out there). They are not necessarily bugs. It's quite hard core and very detailed, but could be useful if you're ever scratching your head wondering why something odd is going on with your MySQL queries.

Topics include

Strange null behaviour
Varchar fields are non-case sensitive by default
Peculiar comment behaviour
Division by zero gives null rather than an error

and dozens more

Genetic Algorithms vs Natural Selection

2004-07-20T12:55:00.000+02:00

A recent post by quinton got me thinking about genetic algorithms, and more specifically, about the fact that the algorithm is essentially based on Darwin's theory of natural selection. It's a great technique for solving certain classes of problems, but it troubles me somewhat that the process of evolution (upon which the algorithm is based) is not properly established.

In general, I think the evidence for evolution is very strong. However, the mechanics of how it works still needs proper explanation - at the present time natural selection seems unable to explain things properly. Let me outline a few objections to the theory of natural selection:

1) How the first replicator could arise is not at all explained. (The first replicator is the first entity able to reproduce itself). There are any many suggested theories, none satisfying. Currently, the first replicator mechanism is not explained, not proved and not reproducable.

2) The fossil subset is very poor. Only a tiny fraction of species show intermediate forms, and no species show smooth transitions. Darwin himself thought that the lack of fossil records was the biggest threat to his theory, but was confident that in the fullness of time, sufficient fossil data would come to light. Good hominid fossil records have been found in recent times (presumably because human evolution is more interesting), but that aside, it's arguable that over 100 years after his death, the fossil records are even poorer (as some promising fossil records from his time were proved to be incorrect).

3) Irreducible complexity. This is a popular weapon of creationists, typically they refer to complex components such as the eye. More compelling are some of the metabolic pathways present in organisms - the fact that complex molecules are synthezed in 12 or more steps, with no useful by-products. Hence the whole process would have to have been discovered "at once".

4) Staggeringly variable rates of evolution. Certain species seem to have "forgotten" to evolve despite being subject to the same evolutionary stresses.

5) Problems with DNA itself. Despite the genome mapping project, it is becoming increasingly difficult to see how DNA could contain enough information to define a complex phenotype.

6) The timescales present major difficulties, perhaps even the most significant objection. The timescales to evovle from hominoid to hominid seem too short by many orders of magnitude.

Now, the bulk of these objections can be overcome by either (a) substantially increasing the time available or (b) coming up with a better mechanism than natural selection.

Let me stress again that I have no time for creationists, who for me fall into the same category as astrologists, homeopaths and psychics (i.e. people who believe in things despite the absence of scientific evidence). However, based on the points above, I think the theory of natural selection is currently inadequate to explain evolution properly. And as such, its use as a basis for a computer algorithm is suspect.

Thanks to Alex for his input in structuring this article.

Struts Tutorial

2004-07-19T08:00:00.000+02:00

I've found what seems to be a well put together HTML-based tutorial for Struts. Part 1 is here and Part 2 is here

How Can Challenge Response Deal with Spoofing?

2004-07-15T15:47:00.000+02:00

The original premise of C-R is that when spam is sent, you will effectively double the resulting internet traffic, because every spam will result in a challenge (which presumably will be ignored, or at least that's the premise upon which the C-R solution is based). However, if the spammer spoofs a legitimate email address, which I believe is very common, then the situation is much worse, as I illustrate below.

Let's say Spammer Sam sends a spam email to Alice (who uses a C-R system), but spoofs Bob's email address. There are two cases to consider:

1. Bob does not use a C-R system

- Sam sends email to Alice
- Alice's C-R system sends challenge to Bob
- Bob gets puzzling challenge mail and sends angry mail to Alice
- Alice receives this angry mail as well as the orginal spam, as Bob's mail has now passed the challenge responded to the challenge

Two extra emails have been generated (effectively tripling the email load of the original spam), and confusion for Alice and Bob is the result. Alice has also received the spam.

2. Bob also uses a C-R system
- Sam sends email to Alice
- Alice's C-R system sends challenge to Bob
- Bob's C-R system sends challenge to Alice
- this may carry on indefinitely, but one would hope the C-R systems would be smart enough to recognize duplicates and ignore subsequent emails

In this case, neither Alice nor Bob receive any confusing mail, Alice does not receive her spam, but again the original spam has been tripled.

Challenge-Response Dashed Upon the Rocks

2004-07-15T13:30:00.000+02:00

Well, wasn't that a short-lived moment?

This article makes the important point that a critical part of the C-R system is that the challenge e-mail (that gets sent from the recipient to the sender) MUST reach the sender. If the sender's anti-spam system blocks it, then the whole process falls over. Therefore, all spammers need to do is to construct their drivel in the same way as a challenge e-mail. Game over.

My good friend rant came up with an even more damning problem: spammers use real emails which belong to others.

So the spammer sends you an email, you send a challenge back...that challenge goes to a spoofed email address...that spoofed user gets annoyed and writes you an angry email, or more likely (and much worse), gets flooded by thousands of challenge emails.

Dealing with Spam - Greylists

2004-07-15T10:12:00.000+02:00

Yet another anti-spam methodology is gathering momentum, so-called Grey Lists. This is a server-based method for dealing with spam that sounds intriguing. The basic idea is to block unrecognized mail for a short period with a temporary failure method. The premise being that typical spam tools don't deal with timeouts and temporary failures gracefully, they just blitz the internet with their evil content and terminate (fire-and-forget). Legitimate (but unrecognized) mail with still get through, as mail servers are designed to deal gracefully with temporary failures.

One obvious concern is that you'd be increasing the traffic on the internet by some degree, as you're asking all unrecognized mail to be sent twice. On the other hand, the majority of your mail (I hope) is from people you know already (and would thus be on your white list - these would not be considered for grey listing).

Further articles can be found here and here.

Further to my recent post regarding simplified Challenge-Response systems in the ongoing battle against spam, one concern that has cropped up is when you register with internet sites and mailing lists - you typically get a confirmation e-mail to which you are expected to reply. I'm not sure how these would get through the Challenge Response system proposed.

.htaccess Explained

2004-07-15T09:19:00.000+02:00

Well, I'm not going to explain it, I'll leave it to this site. They provide a brief, clear but comprehensive explanation of what .htaccess can do for you and provide good examples in each case.

Topics include:

- Error Documents
- Password protection
- Enabling SSI via htaccess
- Deny users by IP
- Change your default directory page
- Redirects
- Prevent viewing of htaccess
- Adding MIME types
- Preventing hot linking of your images
- Preventing directory listing

Update: Here's another tutorial (please ignore the spelling errors)

Byte Code Enhancement

2004-07-15T08:52:00.000+02:00

I recently posted some concerns about byte code enhancements (BCE). I've tried to do a little research on the matter, but there isn't much information about it on the internet. However, having thought about the issue a little further, I've realized that, conceptually at least, there is no difference between trusting byte code enhanced classes and trusting a 3rd party library. If a 3rd party has written poor code, you're going to get bugs either way, and both cases are out of your control. Just something to be aware of, though: if there are errors in your enhanced classes, your class is still going to appear in the exception call stack - so be ready with your explanations when irate users or developers moan at you!

Writing Your Own Firefox Extensions

2004-07-15T08:35:00.000+02:00

Ever fancied writing your own extension? Seems it's easier than you may have thought. Mostly it's just some javascript that needs to be packaged up in the right format. Here is a clear and well-written introduction.

JDO Concerns

2004-07-14T13:19:00.000+02:00

JDO is largely a very impressive specification. Some issues remain troubling, however.

Firstly, and most importantly, I'm concerned about adopting JDO wholeheartedly. I know from experience that any successful database is going to attract users who may not use java. They may find it very difficult to plug their excel-ODBC or php web pages on a JDO-created database.

Secondly, the issue of byte-code enhancement. In theory, there's nothing wrong with byte-code enhancement. But the grim reality in the real world is that sometimes you have to deploy your code on less-then-perfect JVM's (HPUX anyone?) When the code falls over, you want to be sure it's your code, not the enhanced code.

Answers anyone?

Here's a good article on using JDO with legacy database.

A "new" Plan for Spam?

2004-07-13T11:01:00.000+02:00

This article on a new plan for spam makes for an interesting read. The author asserts that Bayesian filters are becoming less and less effective, and that a radically simplified challenge-response system is the key to really killing off spam.

Personally I'm finding that Popfile (a Bayesian filter) combined with an anti-virus checker works very well. However, I accept that some of the anti-Bayesian devices he describes may become more common in the future. His simple challenge-response approach essentially revolves around asking the unknown sender (for it is only unknown senders that are an issue) to send a second mail. I think it has a lot of merit, with one important caveat that he doesn't mention - you will still need a virus checker, as many e-mail viruses rely on sending mails from trusted/known individuals.

Web Services and CORBA

2004-07-12T17:42:00.000+02:00

A sobering article about how web services are evolving to reach the same complexity as CORBA. As someone who has used CORBA for many years (and still does in fact), I can tell you it solves the multi-language RPC problem very very well, but it's a large and complex beast. Web Services should be very wary of going down this route. Note that CORBA spent many years evolving under the guidance of the OMG - are Web Services as well controlled?