PhiRatE

To wit, offense

2009-03-28T17:37:00.003+13:00

Sometimes, the things people take offense at really seem like an offense to offenses. There are things that are genuinely offensive. Some things though, seem to generate rage entirely out of proportion with just how offensive the actual offense is. It's as if we for a moment forgot just how dark and evil the world can be, and turned all our energy to the defense of something only slightly out of the banal. People can be wrong, without being the devil incarnate - even if they refuse to apologise.

Is New Zealand so bereft of injustice that this kind of thing warrants national fury and debate?

Moment of interest

2009-02-15T23:55:00.003+13:00

Did a talk the other night at the Wellington Python User Group about using Python for log parsing. Couple of other talks, including one on using python on the nokia platform. The real tragedy being that without a marketplace to build software for and help distribute it, the far-superior development experience is sadly handicapped, which is why the iPhone is getting all the cool apps despite ObjectiveC hell.

An interesting discussion of an outlining editor which left me vaguely bemused. The individual giving the talk spoke eloquently about how much time and effort the logical outliner saved him (seemed interesting), and yet looked at me strangely when I asked him about tab-completion. C'mon people, I'm aware that most of us have been in the industry a while now and we're pretty fast typists, but seriously, tab completion is a trivial thing to enable and use, it improves the readability of your code (because you're not constantly short-cutting your variable names), drastically reduces the incidence of character-swap typos, and saves your poor little fingers. This isn't rocket science, our productivity is in debugged code per hour, tab-completion is a significant win.

Similarly, having an editor that can mark syntax errors in real-time, track TODOs etc is all pretty basic. If you don't have this stuff, get it now. You're not impressing anyone if you're using bog-standard vim, you just look like an idiot to all the pros. Anyone can code on a stone tablet if they have to, people doing this stuff professionally should be using every advantage they can get their hands on.

Welly PUG

2009-01-08T22:26:00.003+13:00

The wellington inaugural python users group meetup was really good, which was totally contrary to my pesimistic expectations.

Probably the most fascinating part of the whole exercise was that the presenters were basically flying seat-of-pants (huge, huge kudos to them for the courage), and as a result they turned more into audience-participation coding than anything else.

A few meta-aspects were really quite interesting. It was weird how the same features that were being touted as python strengths (generators, list comprehensions, closures) were relatively poorly understood by the people touting them - not a criticism of the people concerned at all, simply an observation that the features are either A) awesome even when only half understood or B) barely used but at least a point of differentiation.

I'm going to go with a mixture of the two, but I thoroughly recommend that coders using python dramatically increase their familiarity with all three, as well as other concepts such as decorators. They can drastically improve your coding efficiency.

For those who were not python coders who were interested, I suspect there was some confusion as to precisely why generators are cool. The examples used, while valid, didn't demonstrate the intrinsic advantages over a standard list.

A classic example is an infinite set:


import math

def is_prime(n):
        for i in range(2, int(math.sqrt(n))+1):
                if (n % i) == 0:
                        return False
        return True

def set_of_all_primes():
        n = 2
        while True:
                if is_prime(n):
                        yield n
                n+=1

calling set_of_all_primes() returns a generator with an iterator that can be used to obtain prime numbers. A second function that acts on iterators can then be meshed with it:


def increment_all_numbers(iter):
        return (n+1 for n in iter)

So now:


for n in increment_all_numbers(set_of_all_primes()):
        print n

Will print out the set of all primes, each incremented by one.

This is useful in a wide variety of scenarios where your data is infinite in extent, expensive to generate or simply demand-driven.

Similarly while closures were discussed, there was no discussion of decorators, which is their primary use in python. Everyone looked rather blank when I mentioned them, which is weird considering how useful they are.

The general confusion about twisted was cute :) "It's asynchronous...and they hate threads...". Yes, this is true, but the important bit about twisted (aside from its huge protocol suite) is the use of Deferrals, a programming construct that attempts to abstract the worst bits of asynch programming into something more intuitive. A deferred lets you return something that, rather than being the result of a function call, is a promise for that result to be delivered at a later time. These promises can then be passed around and, at the time when they are finally actually required, resolved into the answer. This is extremely efficient when you're dealing with problems that have concurrent atomic access issues, since it avoids the need to context switch or lock shared data, while still maximising available processing power in the face of extended IO wait times.

There was a bit of incorrect stuff floating around but most of it got resolved in-flight. One of the more interesting ones was this scenario:


i = 0

def foo():
        i+=1

It was posited that, because += is an operator on i, it would in fact increment the module-scoped i variable, rather than the equivalent:


i = 0

def foo():
        i = i + 1

Which was proposed to create a new, locally scoped i with the increment of the module-scoped i.

In fact, both constructs fail with an UnboundLocalError.

The only real wince moment was when someone demonstrated the way python casts objects to booleans for truth testing. While the feature is indeed neat:


if not obj:
    # obj can be an empty sequence (list, string etc), 0, False, or None
    raise WhatTheFuckException()

Using it for "defensive programming" is precisely what you should NOT do:


def read_file(f):
    if not f:
        return None
    return f.read()

With relatively few exceptions, this is Bad Stuff. We have exceptions for a reason. Return values are for success. In the event of failure, an exception should be raised.


def read_file(f):
    if type(f) != file:
        raise TypeError('f must be a file')
    return f.read()

We've come a long way from C, we don't need to have special return codes for failure anymore. That said, I prefer to avoid type assertions anyway, since I'd rather have anything that is a file *or file interface compatible* as a valid input. Usually, I let the exception system take care of things automatically (ie, if it's none, I'll get an AttributeError automatically anyway).

I came out of the PUG feeling much more interested in python the language than I have in some time. I guess using it day in day out, it became part of the landscape, and I forgot what a fun, intelligent, interesting thing it is. I shall definitely be attending future meets, and I look forward to playing PimpMyScript.

iPhone web enhancement

2008-12-08T11:14:00.002+13:00

Notes from my talk at SHDH on improving iphone browser accessibility on regular websites:

Layout

You can use a separate stylesheet to overwrite style settings for your site:

<link rel="stylesheet" href="mobile.css" media="handheld, only screen and (max-device-width: 480px)" type="text/css" />

You almost certainly want to reset the Viewport size. By default mobile safari likes to pretend it has a 940-ish wide screen and just shrinks stuff. You can control this with:

<meta name="viewport" content="width=320" />

Allowing you to be more precise with sizing.

For layout, you're basically looking at a single column. The screen simply isn't wide enough to effectively support two columns or sidebar nav, even in the horiztonal mode. I recommend leaving the column width unspecified, that way it'll resize nicely on orientation switch.

Your mobile stylesheet may need to have some font-size tweaking, you really need to test to check that out.

Specify logo and header images etc in stylesheet, not within the HTML. This lets you replace them with smaller-size versions in the mobile stylesheet.

Typically, a small number of major functions should be buttons across or near the top. Put useless functionality at the very bottom (if at all)

Forms and buttons

Go with native controls where-ever practical, specialised selectors generally mean that your attempt at styling form controls will be partially successful at best.

A persons thumb is roughly 40x40x or worse - nothing like a mouse pointer. Control size matters

Control-misses are much much more common - don't put irritating combinations of controls together - "Save" and "Navigate away from this page and forget everything you did"

Activity-response. If you're scrolled down on the page it's entirely possible to get no feedback at all on a link click. At the very least make sure you get the link changing color with :active.

Images

Preferably, generate mobile-sized thumbnails. 3g is really slow.

Caching

iPhone has a really small cache, both on numbers of items and size. Maximum per item size is 25kb, roughly 18-19 can be used maximum. LRU replacement.

iPhone will *only* cache stuff with Expires or Cache-Control-max-age headers. If you don't set these headers, stuff will get reloaded again and again.

Cached stuff is stored *uncompressed*. sending compressed content won't help with cache size (will help with traffic though)

Minify etc are hugely valuable here if you're javascript-heavy

Javascript etc:

Mobile Safari is pretty much a full webkit implementation. There's not a lot it can't handle. Unless you're doing something really really crazy it'll probably be fine.

Flash does not work at all. Don't even bother.

Video
mpeg4, preferably encoded for iphone size - 480x320, unless it's screen linked. 640x480 is max.
Link directly to the mpeg to get the in-screen view.

Youtube is a bit weird, you may need to test things.

Separate sites

If you're doing a mobile version of your site, rather than just integrating the stylesheet, the convention is m.. Some sites automatically redirect you from the main site if you turn up with a mobile useragent, and give you a cookie-triggered option to go back to the main site (in case you want functionality not implemented on the mobile version or whatever).

Examples

m.flickr.com

Internal messaging systems

2008-12-05T06:00:00.000+13:00

In social apps it's not uncommon to have some level of internal messaging system, the general idea is that it allows two users to get in touch without exposing emails or other contact details - keeping initial contact at arms length and within context.

The traditional fast solution to this is to have a table within the database that takes a set of messages,each containing a sender and a receiver, plus subject etc.

This works fine until it comes to deletion (In case readers haven't noticed, deletion is the cause of much trouble within web application databases). The problem is that when the receiver deletes their email, the sender should still see it in their "Sent" box. Similarly, deletion by the sender from the Sent box should not remove the message from the reciever's inbox.

Initially the issue appears simple. We add two flags, sender_deleted and receiver_deleted, and that's that. This works until someone requests group messaging, Cc: or multiple targets.

At this point, the architecture actually tends to work better if we think of each users mailbox as independent. That is, when a user creates a message it gets saved in their "Sent" folder, and a copy is delivered into the receivers "Inbox", from which they can move it (if you support folders) or delete it, or whatever. Obviously this can be done in a single table, it's just that the mailbox owner and the recipient/sender are split up.

There's an overhead here, obviously we're keeping two copies of every message. We can perform some degree of optimisation for this but quite frankly it's almost never worth it. The reason for this is because independent mailboxes offer us a particularly compelling benefit: it scales like crazy.

Users only ever read their own mailboxes, thus it's a perfect candidate for partitioning. The cross-partition delivery mechanism used when you have independent mailboxes means that there is no synchronisation necessary, it's a smooth (and easy) process. Even better, we all have a good feeling for how it should work, because it's how SMTP works in essence.

In the end, I recommend that, unless you're feeling particularly lazy, you go for the independent mailbox architecture right from the start. It's only marginally more effort.

Cookies and domains

2008-12-04T06:00:00.000+13:00

A quick trick. If you're setting cookies from your site, prefix the domain with a ".". Ie: Domain: .phirate.com

The result of this is that any subdomain of phirate.com also receives the cookie, rather than it being limited to phirate.com. This greatly inproves your flexibility under a number of circumstances:

1. www.phirate.com and phirate.com both pick up auth properly
2. You can create subdomains like search.phirate.com without having to do cookie bounces
3. You can utilize CNAMEs to distribute tasks to other web services like Amazon S3 and still have access to your cookies

While a pure session cookie isn't often particularly helpful in these instances, and thus with sessions it's not such a big deal (you need access to the session db to make any use of a session cookie), it's possible to include various signed annotated cookies for other tasks.

A classic example is the delivery of static content that needs a dynamic menu bar containing the users name. In this case it's not a security risk if the name is wrong, it's simply a convenience and aesthetic consistency. Storing the name in a separate cookie, possibly along with other data, allows a trivial bit of static javascript to pick the name up and put it in the right place, allowing you to maximise your use of static, cached content while still giving the appearance of dynamic content.

Sites like Entrecard make heavy use of these techniques to drastically reduce load in many of the public pages. The pages themselves are static with javascript calls to perform layout of the top bar menu. In addition, dynamic content is pulled in via AJAX if the user is authenticated, otherwise static cached information is used. This improves the site resiliency in the face of sudden traffic spikes, since the majority of new users will arrive unauthenticated, while still allowing customisation of the page view for logged-in users.

There is one important caveat: Do not use the . technique for cookies when you're selling sub-domain hosting or similar scenarios where users can upload their own javascript or other cookie-accessing code. In this instance, it may allow the cookie values to "leak", and while that's not a big deal for just a username, you definitely don't want that happening for session ids etc.

Making your API easy to use

2008-12-03T06:00:00.000+13:00

The web2.0 craze has placed a large emphasis on opening up your data, and some compelling benefits have come as a result of that (not to mention some cool third party apps), but there are a number of decisions you can make that, while simple, dramatically increase the utility of your API.

There are some situations in which SOAP makes sense, but in general if you're talking about a public web application, you want to delivery your API in a simple HTTP/XML or HTTP/JSON fashion. This drastically reduces the workload for people trying to create simple apps. Avoid using odd HTTP methods. Even things like PUT, if you can. Use GET and POST exclusively.

Version your API. The URL to the API should absolutely contain a version number. The main reason for this is that otherwise people who'd like to create apps with a slow delivery pipeline (*cough*apple iphone*cough*) don't dare use your API - if it takes 3 months to get an update through their customers could be down for 3 months if you change without warning.

Design for load. If your data fits the pub/sub pattern, check out services like Gnip which can take all the load off you. If it doesn't, make sure you pay careful attention to caching issues.

Let your API consumers help you. In general, programmers are happy to go a little extra effort in order to reduce the work you have to perform. If you give them the option to retrieve a reduced dataset, or a cached old one, or provide a more complete set of search operators so that the data returned can be limited, all of these will see use by people expecting to hit your service hard.

Make API keys easy to get. Many third party developers get started on a whim, the last thing you want to do is have them put off scratching that itch because getting a key requires your personal approval (This is appropriate sometimes, obviously, but in general not).

Add/Edit functionality is relatively rare in APIs at the moment. This is a shame, it prevents third parties from embedding your apps functionality into desktop clients, iphone apps, firefox toolbars etc. Seriously consider having more than just data retrieval. It might hurt your ad revenues, but it gives you way more staying power.

Simple authentication where necessary. Basic AUTH is fairly standard, and is perfectly secure if you insist on HTTPS.

APIs are, like many aspects of web application design, rather dependant on the problem. That said, it is well worth pushing for as much API as you can build out - every extra method increases the chances that someone will go out and give you a ton of free advertising by building something cool on top of your service.

Finally, designing an API in from the beginning often helps rationalise and make sense of your own software design in a fashion which a straight web application doesn't. The need to abstract controller and model operations helps ensure you factor your code correctly and makes future changes simpler (indeed, in some cases it's possible to base parts of your site off your own API, allowing trivial, secure delegation to subcontractors or junior coders).

Un-fucking-believable

2008-12-02T15:37:00.002+13:00

The potential evacuation of Kiwis trapped by the protests was thrown into disarray after the air force's two Boeing 757s were declared out of action.

I have no words.

"I am sure the Government is trying to do its best, but I was rather surprised to hear that there are no contingency plans," Mr Goff said.

Shut up labour boy. I voted for you, but there is no way you slime your way out of this by blaming the newbies. Contingency plans should have been there a long time ago. Some time in, say, the last 9 years.

Somewhere, there's a flight engineer banging his head against a fuselage going "I FUCKING TOLD THEM AGAIN AND AGAIN AND AGAIN..". He will go home and rant to his poor, long suffering wife, and not a single one of the morons who let this happen will ever be held to account.

Time and testing

2008-12-02T06:00:00.000+13:00

A common problem encountered when creating test suites against stateful libraries (particularly database-backed ones) is the presence of time and time dependant states and actions.

In general, this gets even nastier if you're using NOW() or similar in your SQL statements. So, a few pointers:

Where practical, supply the time to the query specifically. I know it's annoying, but there are a few good reasons for this:

1. Supplying it means that a sequence of actions in a transaction will all get the exact same time, which helps if you're trying to reconstruct things later on.
2. PostgreSQL specific: NOW() is an expensive operation, and *it is not cached*. That is, if you put NOW() in a subselect, it will execute once for every single execution of the subselect. This sucks performance wise and can crush your query if you've got a big dataset.

If you need to use a database call for the current time (or just want to, sometimes it's more elegant, in defaults etc), I recommend using a stored procedure as a wrapper for NOW(). This way, when you're building up your database for testing, you can replace the wrapper stored function with one that returns a fixed time, allowing you to shift time around as you see fit without mucking with the real system time (which is always painful, especially if it's your desktop).

Isolating time like this allows you to test the full gamut of scenarios within your library and ensure that everything will happen as expected in the future - a situation all too many test suites fail to take account of.

I hate TV

2008-12-01T19:47:00.002+13:00

Seriously, the absolute last thing they should be doing is putting that kid on TV. Quite frankly, all it's going to do is encourage all the other script kiddies. Despite the constant assertion of clueless TV reporters to the contrary, creating and managing botnets is neither difficult nor a sign of some kind of amazing insight into information technology. The software to do it is readily available and barely more difficult than installing itunes, just about any teenager who spends all their time in front of a computer could do it trivially, let alone people with actual training in the relevant areas.

The only difference between them and this kid is that they don't, because it's *wrong*.

There are people on the internet, many of them suffering from one form of social dysfunction or another, who are unable to empathise with others, and thus are happy to take advantage of them. This is not news, nor is it confined to the internet. Possibly the only news here in fact is that law enforcement managed to catch him. The people who are actually good at this aren't on TV, because catching them behind myriad layers of fakes, crypto and one-way lines of control is extremely difficult to coordinate, with control relays and cutouts in countries across the globe, often in uncooperative jurisdictions and in organisations with no IT staff. Fortunately for all of us he got greedy before he got better at it, and no doubt a money trail provided both better information and more motivation for investigators.

And will you all stop asking when he'll be offered a job? he didn't do anything that would make him more valuable than the risk posed by his clear ethical deficit. It's not like we (the IT profession) don't know how people like him achieve what he does, it'd be like asking someone who performed a standard smash-and-grab when the police were going to hire him for his insight into how it's done. It's a smash-and-grab, the police *know* how it's done, the difficulty is simply that it's not practical to secure everything against it - we rely on the fact that we catch most of them, eventually, and that the remainder of the population has some sense that it is cruel and unfair to do this kind of thing to others.

Personally, I wouldn't hire him over most half-decent coders his age - at least with the others there's a reasonable chance they won't think they're clever trying to install backdoors in your systems when you're not looking. Experience suggests it takes these fools another ten years minimum before they grow up and start to understand the kind of impact their idiocy has.

SHDH coming up

2008-12-01T10:41:00.002+13:00

New SHDH coming up on the 7th. Yours truly will be doing a workshop on improving your website for mobile browsers. The idea, basically, is that the majority of websites (that have a reasonably CSS-heavy design element at least) tend to display fairly poorly on browsers like mobile safari - they're there, you can read them, but they're not simple. The addition of a mobile stylesheet is often all that's necessary to dramatically improve the usability, especially where touch-screen devices are concerned.

The workshop will involve a short rundown on strategies to achieve improvements, and then a collaborative attempt to improve some sites. There will be no mocking of existing designs so you should bring a laptop with a checkout of your site on it so you can play.

The motivation for this is mostly selfish - I have a mobile browser and I'd love it if more websites sorted their CSS out to make my life easier.

Other secrets

2008-12-01T06:00:00.000+13:00

Occasionally within web applications we have to generate secrets that aren't passwords. While I've covered passwords in general before, these secrets often have a different context.

A classic example is a coupon code. A coupon code is distinctly different from a user password:

1. It is normally one-shot, and often linked in an email, so remembering it isn't such a big deal
2. It rarely has a "key", in the sense that if your keyspace for your coupon code is 100,000, and you have 10,000 coupon codes active, an attacker only needs to guess 10 times on average to hit the jackpot - they don't need the "user name" to go with it.

In this case, you always want a bigger keyspace than a regular password. In addition, you want something that works well when printed and, of course, doesn't contain any naughty words.

One of the simplest ways to make this happen is the following:

1. 12 characters, all upper case
2. Remove confusing characters, I, L, J, 1, 0, O, U, V, 5, S from the list.

This gives you nice readable characters with a space 95,428,956,661,682,176 or so in size. Then, to get rid of all the naughty words, a trivial trick:

3. Remove all the remaining vowels

You can't make dodgy words without vowels. Not ones people can reasonably take offense to anyway. It's simple, and avoids having big long useless blacklists.

And out of it, you get:

HF8DDHNRRPKQ

If you're sending this in an email and it's likely to be a phone-in, remember to give a phonetic representation as well (Hotel-Foxtrot-8-Delta-Delta etc etc). This saves your users coming up with embarrassing phonetics of their own.

Passwords

2008-11-29T23:05:00.003+13:00

Ok, so, there you are designing your web application and you've done the sensible thing, which is to simply pick up an off-the-shelf registration/user management system and plug it in right?

No, of course you didn't. Nobody I know does that. We all like reinventing that wheel too much.

Ok, that's a bit harsh. The truth is that user management isn't something that's just..fixed. A well designed application customises the security to the purpose and data.

A bank application really deserves decent passwords and 2-factor authentication (don't you dare get excited about the fact that I used that phrase). On the other hand, your michelin.com account tracking the size of your tires..well...if someone breaks in, all they're going to discover is that I don't own a car.

So the first thing you're going to do when thinking about your new app is deciding what level of security is really required. This influences a number of things:

1. Do you let users set their own passwords?
1a. What level of complexity do you require from user-provided passwords
2. How do you handle password resets?
3. Do you need password-authorised actions?

In this case I'm just going to talk about 1. People seem to swing back and forth on this one, from aggressively user-provided (User gives password on signup, reset lets them set new password) to mid (User might give password on signup, reset provides new auto-generated password), to aggressively not-user-provided (Signup generates password, reset provides new generated password, user cannot set password, only reset).

There are arguments for each approach, but in general unless you have specific security requirements you should think twice before taking control out of the users hands. The problem is that if the user can't use their usual junk password on your site, they're going to forget what it was. If your site isn't essential to them, they probably won't bother going through the reset process, so you'll be securely protecting a complete lack of interesting data.

What kind of situations would get you to reject user control? the simplest scenario is one where users spend a lot of time in a "compromised domain". That is, for example, an intranet web server where everyone uses the same username and password. If the users don't have any other sites they use, there is an excellent chance all of them will use the same shared password, leading to a miserable failure in isolating identities on the new service you're building for them.

If you find yourself having to do this, for gods sake remember that people need to be able to *remember* these passwords. 8is62jks0-_ is fantastically hard to brute-force or guess, but it's also stupidly hard to remember. Peoples minds do not take that kind of thing into storage well in general.

It's remarkably simple to generate passwords that are easy to remember. The most trivial system is this: follow every random consonant with a random vowel. Try it, pretty much every time you get something you can say to yourself in your head. The consontant-vowel formulation is easy to promounce (even when it's nonsense) and the rhythm makes it simple to remember.

There are downsides, the password becomes rather more predictable, but it has a couple of weird benefits as well.

In English, the consonant-vowel formula is associated strongly with baby talk. That is, despite it being so easy to remember, it is quite rare that you get a valid word out of it at any lengths beyond about 4 characters. Think of any swear word - almost none of them will ever be generated by this scheme.

A common source of confusion in passwords are the letters i and j, or 1 and l. Again this scheme completely dodges the bullet by virtue of context - if it's an even-numbered character it's going to be a vowel. The user knows this at a level below consciousness, they will automatically select the correct character most of the time (not that it's bad to remove these for safeties sake if you can handle the decrease in keyspace)

Finally, the pronounceable part means that it's easy to tell someone over the phone. While I highly recommend including phonetic representations of any passwords that are likely to be phoned out as a matter of courtesy, it still helps when someone is trying to tell someone on their cellphone from the supermarket.

It turns out that the numerical weakness isn't as scary as it would seem at first. It's not anything like as good as a truly random password of decent length, but remember that it is still random - and this makes it a significant step up on normal user passwords that are almost always derived from something.

If we decide to take "A-Za-z0-9_-+ ,." as a reasonably secure character space, with a minimum length of 6, we get a minimum keyspace of 98,867,482,624. That's a hell of a lot, especially since your secure application will start rejecting login attempts at 6 failures a day and you've salted all your stored passwords (right?). If you take a length of 8 for the scheme above, you get 121,550,625. That's 813 times more guessable than the best case, but it's pretty equivalent to the common space of 6 chars a-z (191mil).

What this means is that it's still perfectly secure against anything except the hash of the password being stolen, and quite frankly you might be screwed if that happens regardless.

Bruce Schneier talks about password selection here. He indicates that with a standard guesser, 24% of user-provided passwords can be retrieved within 100,000 guesses. Even if the guesser was designed specifically for the algorithm above, there is no pattern that will enable that kind of thing. The first 100k guesses will only get a tiny fraction (0.08%) of the passwords in the db (if any). Conversely, the first 121,550,625 guesses will get every single password - you get a better baseline level of resistance at the cost of losing everything if someone is determined enough.

Assuming you're using a decently difficult hash algorithm and that you've salted it decently it's going to take almost anyone a pretty decent amount of time to get through that lot but in the end they probably can if they have your database and they want it enough. Assuming the value of the data is reasonably low however, your real concern is the vulnerability at the first N attempts, where N goes from 6 (casual attempt to break in) to 2200 (spends all year trying to get in via the website). In this case, the random babytalk method is considerably more secure than letting users select their own passwords (go go 'password1').

I guess I'm not particularly recommending this method, so much as saying that if you need to generate passwords, pay attention to making it memorable (so people don't sticky it to their monitor), and this is a simple way that isn't likely to be the weak link in the security chain.

Dealing with deleted users

2008-11-28T15:10:00.003+13:00

Someone happened to ask about this in relation to a project of mine so I thought I'd share a few notes.

When writing any user based app, one of the problems that is often missed is "what do we do about deleted users?". The reason is mostly because we're all too busy focusing on trying to create users, and in general most of us have dozens of accounts around the 'net that are still there and we haven't deleted, so we're not that concerned about it.

When it does become an issue is when you actually *need* to delete a user for some reason - they're abusive, or it's a screwed up account or they've demanded to be removed, or even that reports are being generated and they're contributing to useless noise.

The typical thought pattern is:

1. We'll delete the user record
2. Oh shit, there's like a hundred other tables with FK dependencies on the user table, erm, how about a deleted flag?
3. Ack! deleted flag works but now new users can't be created with the username because of unique constraint!, lets add "(deleted)" to the end of the username!
4. ACK! can't delete username a second time because then we get conflict with unique constraint again..erm..lets do (deleted 1), (deleted 2) etc.
5. SON OF A B***H I have to undelete a user oh woe oh woe hack hack hack.

Right. So, if you've already made it to stage 5 you're screwed, you've done the work, you'll have to live with it. But how to avoid this nonsense in the future?

The essence of the problem is that we tend to create a single table for a user. We are overloading this single table with a number of roles.

1. Unique authentication credentials
2. Ownership
3. Identity

Authentication must be unique, always. Moreover, a deleted user should not have any authentication at all, it's gone, no-loggy-inny.

Ownership must remain, as long as any entity within the system has a claim to it. If you have any kind of historical data or a forum or some other conversation that would make no sense if one of the participants mysteriously disappeared, you cannot remove the ownership relation.

Identity depends a bit on the application. In some rare circumstances it might need to go away when deleted (Owned items may be removed, or simply resolve as "Anonymous"). In most cases, it needs to hang around, at least in a limited form, with a flag marking it as inactive.

Now that we understand the roles, we can design a table layout that properly resolves these issues.

In most cases, Ownership and Identity can be combined together. The target of ownership is almost always going to be the users identity. Authentication on the other hand should be separated out to avoid all the uniqueness nightmares.

CREATE TABLE authentication (username VARCHAR PRIMARY KEY, password CHAR(32), identity_id INT REFERENCES identity (id));

CREATE TABLE identity (id INT PRIMARY KEY, email VARCHAR, display_name VARCHAR ...);

You can either place a flag explicitly in identity for is_deleted, or alternatively simply create a view which checks for a null on join with authentication to see whether the identity has authentication ability.

In either case, deleting a user simply involves removing the entry from the authentication table. The identity entry almost certainly remains intact, providing the historical reference for forum posts, logs etc. In the event that you need to restore the user, you can do so by restoring their authentication entry with a new username and password, without having to touch the identity details. An event log of the delete or storage fields within identity could be used to determine the previous username but in the event the username isn't available it's easy to manipulate it in some fashion to one that is.

In the final analysis, this is a perfect example of why it pays to think about the true purpose of tables and their roles, rather than simply following what everyone else is always doing - the one-table-per-user thing is endemic, so much so that it rarely occurs to any of us that it's simply a bad idea.

Screw you WCC

2008-11-26T14:37:00.002+13:00

Screw you WCC.

What would Jesus do about the Economy?

2008-10-08T22:53:00.002+13:00

Note: the following is intended purely for light-hearted entertainment purposes only. If your attachment to a Christian religion is so strong that you cannot find humor in it, I respectfully suggest you browse to another blog now.

In the face of the massive failure of the financial industry and the subsequent market fluctuations, hard questions have been asked of the various Government responses around the world. In evaluating these responses it behooves us to look for persons of impeccable moral repute and, ideally, perfect foresight in determining an appropriate course of action.

As in many situations, we ask ourselves: What would Jesus do?

We identify 3 indicative events:

1. The Loaves and the Fishes
2. Water to Wine
3. The Sellers in the Temple

Analysis:

In historical times before mass transit and refrigeration the ability to move perishable goods was limited, creating a small, local market for goods such as fish. These "micro-economies" meant that the sudden appearance of a large number of free goods would have a significant devaluing effect on existing stock. It seems likely that the sudden creation of extra bread and fish would have resulted in a temporary - but possibly significant if the margins were slim enough - market crash for merchants dealing in basic food goods.

The production of additional bread and fish, as well as the transformation of water into wine, both in situations where there was no obvious local shortage in the market, suggests that Jesus would have been strongly in favor of dramatic intervention even were the situation not as desperate as it is.

Finally, the forcible removal of the sellers from the Temple suggests an inclination towards a controlled, rather than free market. Rather than letting the invisible hand of the market encourage sellers to move to premises that would cause less offense for a growing percentage of their customer base, Jesus took direct action against them. We conclude therefore that direct intervention by Governments in the marketplace would meet with divine approval, rather than allowing market forces to define the course of events.

This analysis points strongly to the conclusion that Jesus would be in favor of the $700billion bailout.

Web app dreaming

2008-09-21T22:52:00.003+12:00

My scratch-list for features I want in 2009's web application framework

The view moves to the browser

We've been watching this happen bit by bit for a while, but for the most part people still think of web applications as a collection of "pages" - the 'url-driven state system'. This simply isn't efficient anymore, with things like Chrome and Firefox pushing Javascript to the next level on highly-capable browsers, our continual insistence on delivering one stupid page at a time is counter-productive. Frameworks like Cappuccino are, in their own idiosyncratic way, pushing people to understand the new reality - a web application is just that, an application, we're not using VT100's anymore.

Demands:
* The framework must provide an application skeleton that works primarily in javascript for the view, the 'application' downloads initially, possibly in segments on demand, and views are generated and manipulated from this rather than page by page.
* The framework must make use of native widgets and looks where-ever possible
* The framework must interact properly with CSS etc, rather than trying to replace it.
* The framework should not demand pixel-based placement but instead work with relative, flow based placement if desired.
* The framework must provide easy solutions for managing mobile displays, this need not be automatic.
* The framework need not support any browser slower/less capable than IE8.
* The framework must provide mechanisms to allow standard browser functions such as bookmarks and the back button to continue to function effectively.

The remote/local split

With the view moved to the browser to such a degree, the "things don't change until I hit refresh" approach has gone the way of the horse cart. The framework of 2009 must provide effective, efficient, rapid methods of communicating between the server and the client. People like the dojo guys have already starting working heavily towards much of this, their 'datastore' interface provides an abstraction over a collection helping to keep things in sync between the server and client.

This becomes ever more critical as we move into complex inter-relations between data on a server in a single view - adding a user changes not only the list of users on the page, but the user-count, and possibly changes billing details etc. All this needs to respond instantly, easily, and without pointless server load.

Associated with this is the dramatic change to what is essentially an event-driven model for views. Web application programmers are going to have to get used to the idea that pages are not static, but instead constantly changing in response to client and server generated events and messages.

Demands:
* The framework must provide event and message propagation via polling and comet as appropriate
* The framework should provide data with timeouts that trigger events
* The framework must provide easy-to-implement r/o and r/w data sources that transparently talk to the server to maintain sync via the event/messaging channel

Scaling and hosting

The framework of 2009 isn't the kind of thing you run on your $95/mn hosted server. A bare machine with an operating system is no longer the appropriate place for the hosting of a web application. Cloud services being made available now solve numerous problems that are extremely difficult and error-prone to handle on regular systems.

The need for reliability, scalability and uptime demand that the framework deploy on - and provide management tools for - cloud platforms that can scale up and down with demand and provide data backends that are inherently safe without the need for fiddly replication and backups.

Demands:
* The framework must provide trivial deployment to cloud platform(s)
* The framework must provide mechanisms for managing testing, pre-production, branches, production, testing and profiling on the cloud platform
* The framework must provide tools and patterns for managing off-cloud backups of critical data
* The framework must provide no-downtime methods for posting updates and rollbacks

Look, feel and use

As web based applications become faster, more responsive and more..application-like, people will start treating them more like applications. Chrome and others have acknowledged this tendency by creating one-click "Make this an app" buttons. The expectation will follow that the app look native where possible. This cannot be met in a cross-platform fashion with frameworks (like dojo and cappuccino) busily replacing widgets with their own - often ugly - versions.

Nor can we continue to ignore the need for more advanced, capable widgets. Tight integration with remote data streams, sorting, filtering, charting and graphical manipulation are the cornerstone of a powerful application.

Lastly, many aspects of user interfaces today demand visual clues that graphic designers and programmers often do a poor job of. Thing like icons indicating success or failure in an immediate visual form, icons for "new", "delete" etc.

Demands:
* The framework must provide widgets with data source integration for instant, no-code responsiveness to events.
* The framework must provide charts which respond to data sources
* The framework should provide widgets that respond appropriately to the write-status of a user
* The framework should provide an icon library for standard operations and visual events

Efficiency

The average web application today is hideously inefficient. Enormous chunks of layout, markup and data are redelivered with every request. As applications become more and more integrated into peoples lives - and as demand for up-to-the-minute information becomes greater, it is necessary to remove as much of this redundancy as possible. Efficiency in delivery can result in decreased server load, massively reduced bandwidth costs and much higher responsiveness for clients.

Demands:
* The framework must provide JSON based event and messaging transport
* The framework should provide tools for managing script and asset compilation and compression
* The framework should provide tools for easily managing CSS sprites to reduce connection overheads
* The framework must provide tools for managing cache settings for delivered content

Advanced data patterns

The amount of reinvention in various data areas is getting a bit stupid, and people are doing it poorly. Areas like full-text searching, data and database versioning, exports etc have been done enough now that we can select good quality patterns and provide support infrastructure for them.

Demands:
* The framework must provide trivial integration for full-text searching
* The framework should provide patterns for managing data versions ( for say, a row in a database )
* The framework should provide tools and patterns to help deal with database structure changes

Security

As time goes by, various classes of security attack are discovered, defenses are devised, and these are then slowly propagated out as developers stumble upon them (or get hacked). The framework of 2009 needs to provide defenses against all currently known attacks in a fashion that makes it completely transparent, or at least very easy, for the developer.

Things like XSS attacks on forms, lack of validation, failures to encode HTML entities etc should not happen without effort on the part of the developer.

Demands:
* The framework must provide anti-XSS form defenses
* The framework must demand validation on all user-supplied data
* The framework must default to displaying quoted entities whenever user-supplied data is rendered unless the developer explicitly overrides it
* The framework must provide functions for managing encoding, decoding and quoting entities as standard
* The framework should provide built-in authentication support, including OpenID in and out
* The framework should provide demonstration or default code which utilizes proper patterns for user data care, such as the three-level login state (none, from-cookie, fully authorized) and re-requests of password for changing critical user data.
* The framework must provide functions and widgets for password entry and rating
* The framework should provide code or patterns for classic attack vectors such as password resets
* The framework should provide patterns for dealing with classic race conditions like prepare-commit
* The framework must provide exception tracking, viewing, notification, application logs and methods for archiving/searching/filtering those logs

Testing

Testing is ever more critical. The massive uptake of agile approaches to web development has resulted in a lot of stuff going out partially functional, and a great deal of post-release breakage. While everyone understands the need to create test suites, full end-to-end testing is currently a difficult proposition - production platforms are often significantly different from test platforms, and a variety of data-related issues make it hard to test for conditions such as "this goes away 30 days later", resulting in releases that look fine, then go weird after some time - often after the developer has delivered the product.

The framework of 2009 acknowledges this issue and makes easy end-to-end testing a priority.

Demands:
* The framework must provide for the easy creation of end-to-end tests
* The framework must support pre-prod on alternate domains within the cloud platform to ensure the test platform is as near as possible to the production one
* The framework must provide not only functionality tests but performance tests, and profiling mechanisms in pre-prod to help identify hotspots
* The framework must provide libraries and support for clock-shifting, so that tests can 'change time' in order to properly test time-related operations
* The framework should provide mechanisms for result-recording and reporting

Supporting infrastructure

A large number of related areas are done over and over again in web applications, often poorly and in some cases this is extremely concerning - poor management of credit-card transaction state for example. The framework of 2009 needs to acknowledge both code and non-code areas where the requirements are common enough that proper, debugged, tested versions should be supplied by the framework.

Demands:
* The framework must provide a standardized interface for credit card payment gateway libraries, and other forms of payment like paypal. This interface should demand proper recording and state management.
* The framework should provide standard templates or tools for things like privacy policies, terms and conditions, site icons etc.

Easy to code

On top of all this, the framework of 2009 should be easy to code in. There is no place in 2009 for libraries and functions full of inconsistent usage, complex heavily-prefixed naming schemes, weird framework-specific replacements of standard language constructs like arrays, or invention of new wacky names for things that already have perfectly good standard terminology.

A wealth of examples is vital, every single API call should be used in a minimum of two examples so that any time the documentation isn't readily understandable actual in-use code can be checked and toyed with.

Demands:
* The framework must provide complete documentation on every single API call, including at least one example of use for each module
* The framework must provide a set of examples. Every API call must be used in a minimum of two examples
* The framework must be language-native, and avoid replacing standard language constructs
* The framework should support easy debugging via standard language support where possible, and a minimum of a debug console where not.
* The framework must provide APIs and libraries in a fashion which can be auto-completed by popular editors where at all possible.
* The framework should avoid magic which makes stack traces or errors difficult to read

Things I haven't mentioned

There's a ton of stuff I haven't mentioned, from accessibility to batch job support, but the above is pretty much the list of criticals I'm currently keeping in mind when watching new technology.

There is no question that the various things above are addressed in one form or another by someone, unfortunately there is no framework I'm aware of that comes close to matching all of the demands above.

Dojo probably comes closest, but is much more like a library a framework might use to achieve the above than a real solution. In addition they are progressively failing to acknowledge the importance of the native-look requirement, and things like their datastore APIs are difficult to implement. I won't even mention the highly variable and often contradictory nature of the documentation (but hey, at least they're trying).

In the end I'm a bit pessimistic at seeing a significant percentage of the above in a single place any time soon, but I can hope right?

How to steal part of a country

2008-08-11T07:35:00.004+12:00

Step 1: "Provide support" for members of said part of country who are unhappy with existing Government
Step 2: When Government finally gets angry enough to send in the troops, Rush in to "defend" with overwhelming force. For bonus points, shoot up parts of the country that have nothing to do with the bunch you're defending. High-value targets like airports are good, even if they're on the other side of the country.
Step 3: Give passports to all the members of the part of the country
Step 4: Ignore repeated, desperate requests for ceasefire from stunned, over-powered Government. Shoot a boat or two.
Step 5: Deploy humanitarian aid for all the victims, including large, rather permanent, military hospitals
Step 6: Finally agree that maybe we could have a cease fire. Get really prickly when anyone suggests moving troops out. If you need to justify anything, maintain they're defending previously installed humanitarian assets.
Step 7: Refuse to leave. Rely on the fact that the only people with the weaponry and motivation to do anything about it have already got themselves into a nice quagmire half way across the planet. If necessary, create a "protectorate" status for a few years to maintain some fiction of self-determination.

There you have it, the 7 step plan to territory acquisition with moral pretext bonus.

*Please note, I have no opinion whatsoever on recent world events with a remarkable resemblance to the above, I know absolutely nothing about the area or its politics

Engaged :)

2008-07-09T16:26:00.002+12:00

So, yeah, sorry for not posting for a while, i've been busy and to be honest, my life is usually pretty ho-hum.

Not today tho, as of last night, myself and my girl serra are engaged to be married :) :) :) I'm cycling between giddy happiness and panic, but I'm told that's entirely normal, and it's mostly giddy happiness :)

Love you babe :)

Birthday code for Sam

2008-05-31T19:35:00.000+12:00

You may need this file

++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>---.+++++++++++++++..+++++++++.>++.<<------.>----------------.+++++++++.++.------------.----.---.++++++++++++++++++++++++.>+.>.

The Coolest Things from Ironman

2008-05-12T15:21:00.002+12:00

Ok, so, everyone knows the Iron Man movie is great. If you haven't seen it:

1. Don't read any further (spoilers)
2. WHAT THE HELL ARE YOU THINKING? GO RIGHT NOW.

Right, now the legalities are out of the way..

i loved the movie for many reasons, but post-watching, a few things stuck in my mind:

1. Robot arms with fire extinguishers!
2. Being able to fly (totally cool)
3. The Anti-Tank Missile complete with Comedic Timing Device
4. Mini-reactor that glows!
5. I love that house. I love it. I want it. And the windows too.
6. Virtual ex-britas butler with sarcasm module
7. Delicious on-screen graphics

But one of the most relevant things was:

8. A competent PA

I don't have a PA. I'd like one tho. My ability to keep a consistent calendar, or todo list, or anything, is pretty weak. I try, but software has not yet evolved to the point where it can keep me organised. Sadly, I can't afford one, nor can I afford a hilarious robot arm or a house built into a cliff in Malibu. Still, I can dream right?

HORRAY!

2008-05-09T20:13:00.000+12:00

I have finally got internet at my new place again. No more slow cellphone connection for meee...*joy*

Caching is stupid

2008-05-04T01:27:00.003+12:00

Before you freak out, let me run with this one for a bit, it's based on some very practical experience.

Caching is one of those things we instinctively turn to when we hit some kind of performance bottleneck within a web application - after all, the web is pretty much designed to be cached and has all these lovely headers and things, so it seems to make intuitive sense that if your stuff might get cached out there, you may as well cache it in here.

The problem is that caching actually implies a serious problem, one that we often simply accept without any real thought: the authoritative source of our data is too slow to deliver the information accurately as fast as it is required.

Now, sometimes this is reasonable - if you're delivering an aggregated RSS feed then fine, you need to keep a cached copy of the source data for a while because it would be rude to hammer other peoples servers for every request.

Other times, however, it's actually just bad, and it's mostly the fault of the RDBMS.

We in the web community have become so used to the authoritative-database model of life that we refuse point-blank to delegate authority to any other system. After all, the RDBMS has all kinds of good things like ACID which will ensure your data is always consistent etc etc.

The problem is that data consistency has become this strangely religious holy grail. Never mind that foreign key constraints slow the living crap out of your inserts and updates, never mind the fact that in many cases using an ORM means you'd never have an invalid reference anyway, never mind that many people understand transactions in a database so poorly that they simply ascribe them magical powers of correctness they have no hope of guaranteeing, it's still "database or nothing!"

I remember the first time I got thrown out of that comfort zone. In 2000 I was in Sydney working for a great company called Iguana as their systems/linux guy. We were struggling pretty seriously with the huge load of information we were pulling in from the NZX and ASX and our database server was a monster. Scott Cooper - to this day my example of an brilliant programmer - was the programmer there, and simply did the obvious thing - he wrote a C daemon that stored the data in-memory, with basic checkpointing and methods for doing data catchup etc. As with everything Scott wrote, it was a work of art and brutally fast, leaving the database in the dust, relegated to dealing with historical data at its slow, databasey pace.

Since then I have rarely come across a situation where a database didn't do the job in an acceptable timeframe, but there have been a few. In every case, my initial instinctive solution was to install memcached and "do something with it". In many cases this was fine, but in a few situations it was a serious, serious error. Case in point, the most recent Entrecard release involved getting rid of half the caching we were doing - which was failing to provide accurate results, and generally screwing up - and replacing the data source entirely with an in-memory daemon that acted as the authority.

This daemon was written in python using the Twisted framework - a solution that allowed me to access data without contention issues. Careful coding meant that there were basically no loops at all within the code, preventing any possibility of lockup and providing very consistent timing for individual operations. The database had grown so large (millions of rows) that providing an accurate balance for an individual user took upwards of *3 seconds* per request, and the parallel nature of the balance modifications meant that it wasn't possible to "cache" the result effectively or accurately.

The daemon, however, consistently delivered a perfectly accurate balance in-flight in under 0.001s, dramatically improving our ability to scale The daemon also took over a number of other high-contention operations, including managing the card drops. This allowed us to return to the "glory days" of Entrecard where drops were counted immediately and the balance was always up to date, rather than the sad state of affairs we had recently where drops were processed "some time later" by a batch processing system.

Next time you're having performance problems and you start thinking about caching, queues, batches etc take a second look at your architecture - is your current authoritative data source really the best solution to the problem? perhaps rather than using a cache to wallpaper over your problem, you'd be better off creating a data source that can deliver the data at the rate you really need.

SuperHappyDevHouse

2008-05-04T00:05:00.003+12:00

A quick note, assuming I wake up in the morning, I intend to be at the Wellington,NZ SuperHappyDevHouse at Southern Cross (the garden bar) on the 4th of May (today), I'll be doing a bunch of statistical analysis on Entrecard, drop by and say hi if you're around, many people there know me so just ask around if you don't.

Oops

2008-04-18T20:26:00.001+12:00

To everyone who offered kind words in the comments of the last post, thankyou! I hit "Reject" in the moderation screen by accident, I didn't mean to delete them! sorry :/ on a positive note, things are much better today :)