smuglispweeny

The King is Dead!? Long live... Scala? Clojure?!

2009-10-31T13:11:00.004-04:00

Whoa, why wasn't I told Java is closing its doors? I guess I have been out of touch, word seems to be everywhere. I had to go here (blog of the guy who created Groovy) to find out. James is whooping it up over Scala as Java's successor. Yes, the guy who invented Groovy prefers Scala. Quite a bit:

I can honestly say if someone had shown me the Programming in Scala book by by Martin Odersky, Lex Spoon & Bill Venners back in 2003 I'd probably have never created Groovy. -- James Strachan

Damn. So he pretty much invented Groovy by mistake? Did not know about the two-year old Scala? No one mentioned it to him? Groovy got admitted to the standard in the meantime?

Well, Johnathan Edwards reinvented Python Trellis (ergo Cells) without knowing it, and I did not know about Garnet's KR or constraints -- but Groovy got adopted as official Java! You think Scala might have come up over coffee. Anyway...

Steele said Java brought the world half-way to Lisp. I do not think Lisp means what he thinks it means. Proof might be how hard it is for folks to climb out of the pit of javathink. If Java had been a stepping stone to Lisp it would have made the next step easier, not harder. But Java still cannot do closures. Please. And a quick look at closures in Scala has me thinking, omigod, they call that closures?

Clojure starts to look like a Good Move. I see it mentioned in writings on the death throes of Java and that is a big marketing win. The superwhacky thing here is that both Scala and Clojure are syntactically discontinuous from Java. Folks always thought successors had to have syntax similar to the succeeded though Dylan should have served as cautionary counter-evidence.

No, it is not the syntax. The necessary bridging element seems to be....wait for it...the Java runtime! How did that tail end up wagging the language adoption dog? But Clojure gets the nod along with Scala just for sitting atop the JRE! You people scare me.

Well, if Steele were right Clojure would prevail over Scala. Right now googlefight has Scala winning five to one. Maybe Rich Hickey can move 17% of the world 90% of the way to Lisp?

To write, or not to write?

2009-08-10T06:01:00.000-04:00

>> By the way, to change the subject a little, who was it who said,

>> "there are no dead languages, only dead minds"?

> Dunno, but to change the subject even more, Socrates objected to writing

> since it deprives an idea of a mind in which it can "live". So yeah.

Interesting. "Free writing" is a form that lives within a mind but also create a permanent record and slow the mind down enough to achieve more coherence so the mind can work out hard problems. Comedy writing is necessary to trigger a laugh response because every word matters, but then the words must be delivered as if they were coming live from a mind. Exceptions are improv and semi-improv such as Eddie Izzard, of which Mr. Socrates would approve because they arise within a living mind.

I get a lot of complaints about the writing in this blog because I deliberately write as chaotically as I think. Other times I found I gave a much better talk if I read from something written beforehand precisely because otherwise the living mind is too chaotic to get the talk done in anywhere near the time available.

I was just getting ready to videotape an improvised bit to get the good bits to then pull into a fixed, written bit because I am finding good stuff comes out only if the mind is not slowed down as by free writing.

The question is whether Eddie Izzard is lazy, or if Socrates is right on this. Does Izzard do better by capturing his improv and distilling it down to a precise fixed bit, or does he do worse? Or does he just lack the ability to deliver the prepared as if it were unprepared.

We're getting pretty close to talking about programming in Lisp vs NotLisp now. Lisp programming unconstrained by static typing and blessed with a rich library once that library is mastered such that it is all at the programmer's fingertips allows the code to flow freely yet mostly correctly from a live mind even as that mind is forming the solution the code embodies. Diagram that.

I Feel A Naggum (RIP) Coming On: Quads

2009-06-26T00:47:00.007-04:00

I sometimes begin c.l.l rants with "I feel a naggum coming on...". What is a naggum? Normally:

naggum (n): A rant along one of Erik Naggum(1965-2009)'s themes.

That might be self-referentially hopeless which is fine because that is not what I am talking about, I just thought that would be a clever title. In this case a "naggum" is a nugget of Erikian technology. First, his specification of what he called quads (see below), and my poor implementation (even further below and good luck even figuring out how to test it) .

From: Erik Naggum (erik@naggum.no)

Subject: Re: XML->sexpr ideas

Newsgroups: comp.lang.lisp

Date: 2004-01-19 04:24:43 PST

* Kenny Tilton

| Of course it is easy enough for me to come up with a sexpr format off

| the top of my head, but I seem to recall someone (Erik? Tim? Other?)

| saying they had done some work on a formal approach to an alternative

| to XML/HTML/whatever.

| True that? If so, I am all ears.

Really? You are? Maybe I didn't survive 2003 and this is some Hell

where people have to do eternal penance, and now I get to do SGML all

over again.

Much processing of SGML-like data appears to be stream-like and will

therefore appear to be equivalent to an in-order traversal of a tree,

which can therefore be represented with cons cells while the traverser

maintains its own backward links elsewhere, but this is misleading.

The amount of work and memory required to maintain the proper backward

links and to make the right decisions is found in real applications to

balloon and to cause random hacks; the query languages reflect this

complexity. Ease of access to the parent element is crucial to the

decision-making process, so if one wants to use a simple list to keep

track of this, the most natural thing is to create a list of the

element type, the parent, and the contents, such that each element has

the form (type parent . contents), but this has the annoying property

that moving from a particular element to the next can only be done by

remembering the position of the current element in a list, just as one

cannot move to the next element in a list unless you keep the cons

cell around. However, the whole point of this exercise is to be able

to keep only one pointer around. So the contents of an element must

have the form (type parent contents . tail) if it has element contents

or simply a list of objects, or just the object if simple enough.

Example: 123 would thus be represented by (foo nil "123"),

123456 by (foo nil "123" bar nil "456"), and

123456 by #1=(zot nil (foo #1# "123"

bar #1# "456")).

Navigation inside this kind of structure is easy: When the contents in

CADDR is exhausted, the CDDDR is the next element, or if NIL, we have

exhausted the contents of the parent and move up to the CADR and look

for its next element, etc. All the important edges of the containers

that make up the *ML document are easily detectible and the operations

that are usually found at the edges are normally tied to the element

type (or as modified by its parents), are easily computable. However,

using a list for this is cumbersome, so I cooked up the «quad». The

«quad» is devoid of any intrinsic meaning because it is intended to be

a general data structure, so I looked for the best meaningless names

for the slots/accessors, and decided on QAR, QBR, QCR, and QDR. The

quad points to the element type (like the operator in a sexpr) in the

QAR, the parent (or back) quad in the QBR, the contents of the element

in the QCR, and the usual pointer to the next quad in the QDR.

Since the intent with this model is to «load» SGML/XML/SALT documents

into memory, one important issue is how to represent long stretches of

character content or binary content. The quad can easily be used to

represent a (sequence of) entity fragments, with the source in QAR,

the start position in QBR, and the end position in QCR, thereby using

a minimum of memory for the contents. Since very large documents are

intended to be loaded into memory, this property is central to the

ability to search only selected elements for their contents -- most

searching processors today parse the entire entity structure and do

very little to maintain the parsed element structure.

Speaking of memory, one simple and efficient way to implement the quad

on systems that lack the ability to add native types without overhead,

is to use a two-dimensional array with a second dimension of 4 and let

quad pointers be integers, which is friendly to garbage collection and

is unambiguous when the quad is used in the way explained above.

Maybe I'll talk about SALT some other day.

Erik Naggum | Oslo, Norway

Act from reason, and failure makes you rethink and study harder.

Act from faith, and failure makes you blame someone and push harder.

(in-package :ukt)

;;;(defstruct (juad jar jbr jcr jdr)

(defun qar (q) (car q))

(defun (setf qar) (v q) (setf (car q) v))

(defun qbr (q) (cadr q))

(defun (setf qbr) (v q) (setf (cadr q) v))

(defun qcr (q) (caddr q))

(defun (setf qcr) (v q) (setf (caddr q) v))

(defun qdr (q) (cdddr q))

(defun (setf qdr) (v q) (setf (cdddr q) v))

(defun sub-quads (q)

(loop for childq on (qcr q) by #'qdr

collecting childq))

(defun sub-quads-do (q fn)

(loop for childq on (qcr q) by #'qdr

do (funcall fn childq)))

(defun quad-traverse (q fn &optional (depth 0))

(funcall fn q depth)

(sub-quads-do q

(lambda (subq)

(quad-traverse subq fn (1+ depth)))))

(defun quad (operator parent contents next)

(list operator parent contents next))

(defun quad* (operator parent contents next)

(list operator parent contents next))

(defun qups (q)

(loop for up = (qbr q) then (qbr up)

unless up do (loop-finish)

collecting up))

(defun quad-tree (q)

(list* (qar q)

(loop for childq on (qcr q) by #'qdr

while childq

collecting (quad-tree childq))))

(defun tree-quad (tree &optional parent)

(let* ((q (quad (car tree) parent nil nil))

(kids (loop for k in (cdr tree)

collecting (tree-quad k q))))

(loop for (k n) on kids

do (setf (qdr k) n))

(setf (qcr q) (car kids))

q))

#+test

(test-qt)

(defun test-qt ()

(print (quad-tree #1='(zot nil (foo #1# ("123" "abc")

. #2=(bar #1# (ding #2# "456"

dong #2# "789")))))))

(print #1='(zot nil (foo #1# ("123" "abc")

. #2=(bar #1# (ding #2# "456"

dong #2# "789")))))

#+xxxx

(test-tq)

(defun test-tq ()

(let ((*print-circle* t)

(tree '(zot (foo ("123")) (bar (ding) (dong)))))

(assert (equal tree (quad-tree (tree-quad tree))))))

(defun testq ()

(let ((*print-circle* t))

(let ((q #1='(zot nil (foo #1# ("123" "abc")

. #2=(bar #1# (ding #2# "456"

dong #2# "789"))))))

(print '(traverse showing each type and data preceded by its depth))

(quad-traverse q (lambda (q depth)

(print (list depth (qar q)(qcr q)))))

(print `(listify same ,(quad-tree q))))

(let ((q #2='(zot nil (ding #2# "456"

dong #2# "789"))))

(print '(traverse showing each "car" and itd parentage preceded by its depth))

(print '(of data (zot (ding (dong)))))

(quad-traverse q (lambda (q depth)

(print (list depth (qar q)

(mapcar 'qar (qups q)))))))))

;;;(defun tree-quad (tree)

(defun testq2 ()

(let ((*print-circle* t))

(let ((q #2='(zot nil (ding #2# "456"

dong #2# "789"))))

(print '(traverse showing each "car" and itd parentage preceded by its depth))

(print '(of data (zot (ding (dong)))))

(quad-traverse q (lambda (q depth)

(print (list depth (qar q)

(mapcar 'qar (qups q)))))))))

American & Iran: Separated at Birth?

2009-06-22T01:16:00.002-04:00

Am I the only one grooving specifically on the fact that Iranians are telling their authority figures to go f*ck themselves? Here is the country we thought we hated but it turns out they are as kick-ass as us when it comes to political freedom, and we utterly respect them for their strength. Omigod, Americans and Iranians are going to get along great!

How to teach math

2009-05-11T00:34:00.006-04:00

> On Sat, 09 May 2009 15:30:52 -0400, Kenneth Tilton wrote:

Well, i was not really trolling, I was forking the thread to make fun of the New Math that tried to get the numeral/number distinction across to five year olds.

Someone responded:

You find it better to start with medieval concepts working gradually on to the mathematics of XIX century, while explaining each next year what was wrong with the things they learnt a year ago?

I said all that? Ma's gonna be right proud. But...

Funny you should ask. Yes, I suspect the path society took to get to what it knows now about math is the path an individual neuronal mass should follow. ie, kids should encounter zero and roman numerals and place value and algebraic variables in the same order society developed those ideas. The history of math is your math curriculum guide.

The New Math erred by selecting the logical organization of mathematical concepts as its curricular pole star. Next came Constructivism, which wanted kids to reinvent math. From scratch. Cool idea, but too slow.

Instead, let the history of mathematics dictate the order in which things are directly taught. Maybe go further and teach math as history with less emphasis on computation. Math often advanced when needed to solve real problems. Maybe we can shut up the little devils asking why they need to learn this stuff.

As for explaining all along the way what was wrong with the ideas taught the day before, hey, ever read a book on programming? They typically develop a chunk of code iteratively, presenting ever more improved variations on a primitive original. Come to think of it, ever develop some software? Same thing.

Here's the deal: most folks do not even know zero had to be invented. One understands zero better if one has done without it and then the teacher invents it for you. Something like that.

Cells: The Secret Transcript

2009-02-04T11:10:00.002-05:00

The Boss asked me to give the group fifteen minutes on Cells because I have been talking about it for a while as a future better mousetrap for us and then suddenly last week threatened actually to apply it to qooxdoo and the front end.

I forwarded to everyone a link to a reasonably complete yet relatively brief write-up which tells you everything you need to know about Cells. I know that if I were in your shoes I would not have read it so I presume no one has. But I would like to determine how many folks I will be boring to tears if I review said document, so I will first cut to the chase and ask if anyone has any questions based on what they read.

........silence..............

OK. Cells is at once the simplest and hardest thing in the world for programmers to understand. Simple because the idea is just to have slot values of objects work like cells in a spreadsheet, and everyone knows how spreadsheets work. What is hard is understanding that one can program computers this way.

I only have fifteen minutes so: Yes, you can. Program inputs are assigned by good old imperative code to input cells the same way a user types values into a spreadsheet when they are doing what-if analysis or recording, say, actual monthly expenditure into a budget spreadsheet. Intermediate, derived, and aggregate cells compute new values based on those new inputs from predefined rules just as user changes to a spreadsheet propagate to other spreadsheet cells. Observers on cells let the emergent working model manifest its decisions with more good old fashioned imperative code, usually by simply updating the screen or playing a sound or controlling some external device over a serial port or updating a database.

Boom, we're done. Why is it so great? What part of the superiority of functional and declarative paradigms should I explain first? As I outlined in the material you did not read, a lot of things have to happen when a program receives an input. The programmer coding the event handler has to look at the event and decide all the things that have to happen in light of that event, and any things that follow from those first things. Not only must they reliably see to all those things, but they must do them in the right order. The analogy to a real spreadsheet is quite strong, if you imagine hand-implementing a spreadsheet with old-fashioned pencil and paper.

So the first things Cells does is eliminate a lot of work and thus a lot of bugs. Because the work eliminated is tedious, Cells also makes programming a lot more fun. But there is more.

The declarative paradigm means I always know why a slot has a certain value, because all the logic appears in one place, in the rule assigned to that slot. Without Cells any number of lines of code may have assigned a value to a particular slot and a unified deriving rule certainly cannot be divined even if one were to track them all down.

There is more. Most people hate OO because it never quite panned out. Objects turned out not to be reusable. One of the nicest features of Cells is that two different instances can have different rules for the same slot. That makes objects reusable. Yayyyyyyy.

I did not use Cells in the Kleaner because that was more of a straight calculation running from start to finish. I like to decribe the role for Cells being in any situation where one has an unpredictable stream of data and one is keeping a model with a sufficiently large amount of internal state consistent with that stream of inputs. Two examples being a GUI and a RoboCup client. The Kleaner worked by compiling statistics from a fixed store and then translating exactly once dirty data into corrected data.

Where Cells would have been useful would have been in implementing Phil's ideas about an ongoing stream of data leading to rediscernment of things previously discerned. Cell rules would take new raw inputs and propagate them over to tables of probabilities which would then reach out to existing cleaned data and possibly redecide from the original raw state a new cleaned state.

So no, I do not use Cells for everything, but in this case it was only because the full functionality had not been addressed.

A fun note is that I have in the past applied Cells to a database, specifically the old AllegroStore persistent CLOS database. This works two ways. One is that a user can be looking at a screen and as the underlying data changes the screen changes. That may sound like old news but with Cells one doe not have to write any code to make it happen. One just says "this view shows this users overdue books" and when the date changes the overdue status on every book gets updated and a new book appears in the list on the screen if someone happens to be looking, simply by someone having written code to list overdue books on the screen as if it were an unchanging value.

The other thing that happens is hinted at above. Things like overdue books and amount of fines owed and paid can be calculated from scratch by reading a users entire history of checkouts and returns, but sometimes it is useful to record such derived values in the database and update them incrementally as books are checked out and returned. We can have code in programs do it and hope they run at the right time, or we can have the code in the database (as datapoints mediated by Cells) and be sure they run and run immediately. We get timeliness of data, efficiency, and we still get consistency even as we introduce redundancy.

I left something out. That is bad because the thing I left out is the thing I am planning to do with Cells on my own anyway and then apply to the FE. Cells makes it dead easy to drive a separate framework from Lisp. In my note I mentioned tcl/Tk and Gtk. These are two killer C GUI frameworks with their own homebrewed little object models. We want to program in Lisp, and we want our models driven by Cells for all the reasons above. No problem. We build a model out of instances of CLOS classes mapping isomorphically onto Tk or GTk classes and use Cell observers to pipe information (thru an FFI or even literally a pipe) to the C library or runtime to drive there the creation and animation of C instances.

Works great, and one amazing programmer Peter Hildebrandt pulled off a trifecta in which he had Cells driving and driven by both GTk and a C physics engine, name forgotten.

For a while I kinda marvelled at how Cells could be so useful for such disparate activities, and do so in the same application, the two activities being building an application model and having some other programming framework dance to that model's tune.

I figured it out in time for ECLM 2008, not they were able to understand me. I opened by telling them that Cells was the single most powerful library they could use, because Cells is about change and nothing is more fundamental than change.

Programming is hard because like someone doing a spreadsheet on paper we programmers end up with the burden of propagating change thoughout our models. It is tedious work, it must be done reliably, and there is a lot of it as internal program state multiplies, exponentially a lot. This exponential growth in interdependence of program state is what led Brooks to declare that a silver bullet was not only unlikely to be found but that it would be impossible to find; he felt the complexity was ineluctable because as states multiply there is nothing that can be done to avoid the explosion of interdependence.

I wrote to Dr Brooks recently and asked him if he had ever looked at dataflow. He said he was familiar with the concept, but no. Oops.

Tinkering

2009-01-25T15:12:00.001-05:00

Spring cannot come soon enough for Bobi, who is losing it badly these days on comp.lang.lisp:

Slobodan Blazeski wrote:

Dear board members

I'm baseball player for a several time periods (days,
moths ,years,decades) I've noticed that interest in baseball is
dwindling, and baseball is becoming less and less relevant and will
soon become extinct with only baby boomers supporting it, and even
those are either going to die or switch to golf. In order to save our
favorite sport I propose we make drastic changes and adapt more modern
things like:
a. Playing on the beach sand wearing swimwear like in beach
volleyball, very modern sport. Check Thiobe for growth rate
b. Replacing bats with hockey sticks. Note that hockey is popular in
many world countries and we should think international
c. Including 24-Second Shot Clock like in NBA that will make our
sport more lively and fast paced
d. Square playing fields should be replaced with the more common
rectangular one like found in many popular sports : soccer, football,
tennis etc

Including this will make baseball prosper.

very truly yours

Concerned Semi-Ex Baseball Player
Avenue of delusional weirdos Number 23

Bobi may not be as crazy as he thinks he is. Baseball suffered extreme popularity anxiety in the late Sixties and did indeed tinker with the game. Thinking more offense would attract more fans the pitching mound was lowered so pitchers did not get extra energy into the ball from falling into a pitch. The American League adopted the designated hitter to eliminate the 11% nil pitcher from batting lineups (eliminating as well an awful lot of interesting strategy). They avoided the salary caps of the NBA and instituted free agency (well, no, they lost a lawsuit) which allowed bigger markets like NYC, Boston, and LA to buy better teams, and bigger markets are always good for ratings. Minnesota fans will follow the Dodgers, Los Angeles fans will not follow the Twins.

The changes went beyond the playing field. Ballparks added mascots and a disgusting cacophony of party music between innings so loud you can barely talk, and limited alcohol sales late in games to make the experience more family-friendly cuz you know how the losing fans get in their third hour of drinking.

Now baseball is hugely popular again so tinkering with grand institutions can work. Right?

Wrong. In the end, baseball is just a great game: multi-dimensional and deep. Quality tells, and which quality one emphasizes matters. Hockey and basketball have non-stop action and are fading in popularity, while baseball and football like great music have a variety, a rhythm, a balancing of quiet against intense. Baseball has the pitch, football has the snap. All scales from small to large from inning or drive to the game or season always and invariably end up condensed into one point of explosive tension when the pitcher releases or the center snaps the ball.

Intense without quiet merely exhausts. A boxing match with two brawlers spurning defense landing bombs back and forth brings the crowd to its feet but those who love the sport do so for its nickname, The Sweet Science. They still talk about one genius of defense who won a round without throwing a punch. Between evenly matched fighters one solid punch (forget the knockout, the cartoon haymakers of Rocky n) brings the crowd screaming to its feet, the culmination of rounds of careful, tentative, mutual exploration. A single knockdown becomes a cause for pandemonium and one punch knockouts almost do not happen between the best and when they do they are talked about for a long time. I digress.

Tinkering. Basketball has all the action in the world and now faces its own popularity crisis. Racism is one factor, another is probably the salary cap that has San Antonio in the championship series instead of New York. Another problem: poor defense, and a twenty-point lead does not mean anything.

But worst of all is the lack of dimensionality. There just is not that much to these games to argue about over the water cooler. Baseball? Boston still talks about the time Grady Little [thx, Xach. ed.] left Pedro Martinez in one inning too long against the Yankees in game seven of the ALCS. Come on, he had thrown a hundred pitches! Everyone knows Pedro is useless after a hundred pitches! You just never hear anything like that about hockey or basketball, which both boil down to great athletes pretty much just playing run and gun.

Baseball never needed tinkering, though tinker they did. The fundamental quality of the game first ensured its survival throught the hard times when fans strayed for the quick fix of non-stop hockey and basketball action. Now the richness, subtlety, and sophistication of the game has some stadiums selling out most games of the year of a very long season.

Moral for Lisp left as an exercise.

Tilton's Law: Solve the Failure First

2009-01-20T13:38:00.005-05:00

The team was at my throat.

"Just use the new search!," they bellowed.

The mission critical, project saving, do or die demo to upper management was eight hours away and we had not even begun the always dicey process of moving the software from the development system to one within reach of the Demomeister, and I was trying to find out why the old search was so slow.

"Soon," I replied.

We had a new search I was told was a screamer but I continued poking around putting in metrics trying to figure out why the old search was so slow. Had we not been a virtual remote telecommuting team I would not have lived to tell this tale but we were so they had no choice and I reassured then that "soon" meant ten minutes and they shut up.

Why was I still trying to understand the perplexing sloth of the old when a whole new replacement module was available and working fine and pretty much the demo on which all our jobs and a cool project depended was coming on like freight train?

Tilton's Law: Solve the failure first.

Early on we learned the other side of that coin: Solve the first problem. The commonality is...no, let's do the war story first, war stories are more fun than preaching.

Back we go a quarter of a century to my first contract with a client who would become my sole recurring client for the next decade. I was being hired to take over maintenance of an application whose author had been one of the first to die of AIDS. I was reminded of the whole business by a conversation with another developer recently about the nature of working on OPC. Other People's Code.

In my IT career I have worked always at the poles of software development, either writing new code or performing massive overhauls of OPC, never that relaxed zone between in which one simply maintains and extends in small ways a long-lived system. The second pole (OPC overhauls) always seemed to me an intimate one-way encounter with some anonymous predecessor, an encounter usually involving me roundly and steadily cursing them out. You can imagine then how eerie it was working on this system from this predecessor who was not so anonymous this time, especially when I learned that the poor guy was in bad shape during one stint but needed the money and so worked on the code I was now working on even as his fate rose up to meet him (this well before the days of the cocktails of today that make ones fate less certain). This guy I do not remember cursing out so much.

But I digress. Our lesson today is how to piss off your coworkers by insisting on solving a failure first, by which I mean even if you do decide to punt on X make sure you understand how X failed. I am not alone in this. In 2001 the movie when the crew determines that the unit Hal said was no good was fine he says fine let's put it back in and let it fail. Sure, he was really looking for a way to kill the crew but we learned in 2010 that Hal was just a computer system and I think the bit about putting the supposedly OK/not OK system back in to see if it failed was one of Hal's systems working nominally in accordance with Tilton's Law: we need to understand broken things.

[Aside: a good example of the Hell of Not Solving Failure.]

And now at long last, my unsolved failure. My predecessor's, actually. The application was a securities database with a nightly feed of data applied to the cumulative DB by a batch program. This is late 80s, primitive stuff. A security could have three IDs because three groups were tracking securities and each had their own ID system. We had tens of thousands of records in our VAX/VMS RMS file, and a separate RMS key for each of the three possible IDs. So far so yawn. Here comes the fun part.

Two of the IDs were populated all the time. The other one was populated five percent of the time. Big deal, right? Right, very big deal, the poster boy for Solve the Failure First. What happened was this quesswork reconstruction:

My predecessor Paul (I picked "Paul" because it easier to type than predecessor) had a problem. His program ran an initial test load of a hundred securities in a few seconds. Fine. Everything looked good. So then he ran it against a full daily feed, which would include news of every security traded that day so it would be -- OK, I confess I completely forget even the order of magnitude, let's say tens of thousands and declare up front that that is idiotic and I am sorry, but here is what happened: the damn thing ran forever. There probably was no immediate specific great mystery because Paul probably had the program printing something (a count, the last ID recorded, something) right to his VT-100 console as it went and he could see that the program had started out zooming along but then gradually got slower and slower until just adding one security to the database (and this is just good old ISAM, mind you) took... wait for it... twenty seconds. Oh. My. God. What on earth is happening?

Paul got a clue. Every once in a while two records were written out bang-bang, as fast as at the start. Dig dig dig puzzle puzzle...ah, there it is. Any record for which we have all three IDs is written out in nothing flat. Any record (you know, the ninety-five percent) with just two will (by the end of the run) be written out three per minute, 180/hour, or 1000/fuggedaboutit.

Paul realized what was going on. The ISAM file system had no problem storing data with duplicate keys, which was a good thing because Paul was storing a whole lot of data with one key 95% the same: spaces. Poor ISAM it seemed was chugging thru all the duplicates looking for the last one after which it would record the latest duplicate. And apparently it took twenty seconds back then to walk (effectively) the entire index of a hundred-thousand record file.

Now the good news is that we would never need to look something up using spaces as the key value sought, so....what can we do? Paul was no slouch. He popped open the RMS reference manual and to his delight discovered he was not the first to pass this way and gleefully added the option "NULL_VALUE=SPACES" (translated: "if the value is spaces, Just Don't Index this record on this key") to the key definitions in the file definition script he was using to initialize the file and recreated the file and re-ran the program from scratch.

The change did not help. At all. I think we all know that feeling, as visceral as a dropping elevator.

There it was, the option explicitly intended to solve the problem he had explicitly encountered, and it did not change a thing. Impossible. But this happens to us programmers all the time. We know what to do. Compile the damn code, because we made the edit change but forgot to compile. Or link. Or, in Lisp, to zap the faultily specialized method. Or something.

So Paul edited the definition again and checked that NULL_VALUE = SPACES was the right syntax and right spelling and on the right key -- to hell with that, he put it on all three damn keys -- and he saved it and checked the date and created the file again and ran his program again and you know it did not run any faster or I would not be telling this story.

OK, time to get serious. Or if he was good he did this without all the huffing and puffing of the preceding paragraph. He Just Typed In "rms/analyze sdb.dat". And RMS looked at the file itself (not the script used to create it) and confirmed that "NULL_VALUE = SPACES" was operative for all indexes.

Momma don't let your kids grow up to be programmers.

What comes next is hard to convey. I can tell you but if you have not worked on this code or (we will learn) run this batch application it is hard to convey how much blood, sweat, tears, CPU time, and delayed nightly batch closes for how many years resulted from Paul's not first solving the failure of NULL_VALUES=YES.

Well, maybe this is a fair glimpse of the enormity that followed: the problem got sorted out only because the head of operations and I got to talking one day and he pretty much ended up down on one knee begging me to find some way to eliminate the two-hour merge step that held up the nightly close every night. I had never heard this before, though I had been overhauling the system for months.

"It just sits there for two hours," he groaned. "It kills us every night. Please, if you can, please, do something to make this go away."

Whoa. I had inherited this system and been asked to enhance it but no one had said a word about this. The code was far and away the best OPC I had ever dealt with so everything got the benefit of the doubt, including the (soon-to-be explained) two hour merge. As in, if it is there, it must be there for a good reason. What was not there was The Story of the Unsolved Failure of NULL_VALUE=SPACES, but even if it had been I would have taken that at face value, too, because the NULL_VALUE option was unknown to me. But enough of this flash forward, let's get back to poor Paul.

NULL_VALUE was not working as it should. Software is like that. Good programmers do not let bad software stop them. Plan B. A rule is born: Thou shalt not write new securities to the securities database where the massive duplicates will make each write take twenty seconds. Paul decides to write them to a second file initialized empty on each run. Since we only got dozens of new securities in one batch, that file would never have the massive count of duplicates and writes would be lightning fast. Then we just do a sort/merge at the end of the batch to combine the new securities in with the old. Oops. "Just."

The funny "you can run but you cannot hide" moral within the moral being that I did the calculations one day and worked out that twenty seconds times the usual number of new securities in a day was exactly as long as the sort/merge that was just killing the folks down in operations. And I bet Paul realized that but only after writing all the crazy code he had to write to work with two files at once as if there were only one file and at that point he just gave up and moved the thing into production. Speaking of crazy code...

You should have seen it. Looking back I cannot recall why it should have been so hard, but I did overhaul that code and I was forever tripping over it. The idea is simple. To look up a security to see if we already have it, first look in the real DB and if it is not there look in the daily "new stuff" DB and if it is not there, ah, it is new. If it is, update it. Just remember to update the right file, because we can get data from two sources about the same new security.

Piece of cake, right? A bottleneck function for all reads and updates... anyway, it seemed like the issue was always getting underfoot as I worked, and just looking at the code one saw again and again this check here/there code, and both Paul and I were the kind of engineers always on the lookout for ways to make code non-redundant. I would think my memory was faulty but I also remember eliminating Paul's Plan B after solving his failure first and that was no picnic. It just permeated the application.

So what was the first failure and how did it get solved? First, I had noticed the issue myself while using Datatrieve to add a record to the securities DB for test purposes. I hit enter and thought I had crashed the system because it went away and never came back and like every egomaniacal programmer out there I always assumed that whenever a system stopped responding the last thing I had done must have broken it so there I sat in dread for twenty seconds until the system finally responds. Wow. Twenty seconds? And then I guess I added a record specifying all three keys and it responded instantly.

But this idea of null values not being recorded in an index was new to me, and we did not have the Internet back then where I could just ask the ether what was going on so it was only a coincidence that just after the guy in operations had begged me for a fix that I was visiting with the lads from a prior contract and I moaned that RMS sucked because it could not handle files with hundreds of thousands of records and they laughed at me and said they were handling millions with RMS.

I can actually remember the look on my face, a neat trick when you think on it.

I haul ass back to work and pull out the RMS reference manual and I can tell you that dead trees aside there is one good thing about paper documentation: right above the entry for NULL_VALUES close enough to catch my eyes was the entry for NULL_KEYS.

Yep. You need to specify both. Paul had specifed NULL_VALUE=SPACES. He had not specified NULL_KEYS=YES. The default for NULL_KEYS? Guess.

I kinda wretch inside even now thinking about the astonishing amount of money, work, debugging, and delayed batches that followed from one simple failure to understand one broken thing.

The metalesson shared with "Solve the First Problem"? In programming, never deal with the unknown. This game is hard enough.

Epilogue

The punchline is that I never solved the first failure from Scene I of this tragedy. As my father used to say, "Do as I say, not as I do." We did have a deadline, and I did narrow down the location of the problem in a way that reassured me somewhat that it would not jump up to bite the new code in the rear end. And even in its breech the law is confirmed: we do need to address the underlying problem which I have some confidence I now understand because it still presents problems for the software but it will go away only when bigger problems are solved and they are much bigger so I am keeping my sights set on them.

The Road to qooxdoo Part III: Why It Rocks

2008-12-29T02:23:00.009-05:00

In the episodes one and two we learned all about how a Lisp God and Interwebby know-nothing found a way to support a client needing an enterprise Ajax Web application, said way turning out to be qooxdoo after solid efforts to make jQuery, Dojo, and YUI fill the same bill. But we did not learn very much detail about qooxdoo, we just learned that it comes with lots of doc, lots of examples, a solid community, and that the engineering is first rate. I may not have mentioned this, but it also was dramatically faster/smoother than the others on my acid test: scrolling a datagrid.

Okay, that is a lot as a general recommendation. I mean we did not learn much detail. Until now. Here are four Big Things that really stand out:

1. HTML? CSS? Never heard of 'em.
First comes a shocker. The qooxdoo user codes no HTML or CSS and needs know nothing about those two disappointments. Just as my Lisp compiler reads my Lisp code and spits out optimized native machine code, qooxdoo takes your qooxdoo/Javascript and produces the HTML and CSS to drive the browser. Upside: qooxdoo developers write less code and never have to worry about browser variability again.

To a Web noob like me there is a second big win here: I do not have to learn HTML, CSS, or the vagaries of the different Web browsers my client might want supported. I am a bit of a rarity in my ignorance of those things but soon I should have a lot of company now that qooxdoo has set the bar as high as it has on insulating the web app developer from lower level technology.

By the way, in case you are wondering: yes, there is a widget (two, actually) that will accept classic HTML.

2. I do declare. Not!
The second thing that hits us is a consequence of the first: we are going to be writing a lot more Javascript than we would be with libraries such as Dojo where authors still write HTML/CSS. With declarative tools we sit in Lisp or Ruby or PHP on the server generating HTML. Not qooxdoo (but see next) and not YUI.

I am not at all happy about this but I will live and if it ever bothers me enough I will spend a weekend exploring the qooxdoo markup contrib called QxTransformer. Right now I need to catch up on a backlog of functionality the client has had to do without because of the six weeks it took me to identify qooxdoo as The One.

3. OO is not the Grail, but OO rocks
Stendhal said it best in "On Love" in the chapter on Infatuation: (paraphrasing poorly) we do the object of our desire two injustices, first setting them on a pedestal unreasonably high and then, when they inevitably disappoint, setting them too low. OO is not the Grail, it is just a profound advance in serious application development. All by way of introduction of another stunner: qooxdoo's slightly more substantial object model layered atop the core Javascript OO capabilities.

To be honest this struck me as a bit of a yawn until I decided to start carving up my existing code into distinct qooxdoo classes. This is a very big application and I needed code re-use almost immediately and doing it in HTML with Javascript glue was...well, before qooxdoo I was pretty much starting to build my own abstraction layer atop the core tools--there was no way around it.

But now that I have used qooxdoo's OO to partition my rapidly growing code base I have more than once had that wonderful sense of getting something for nothing after doing a major refactoring (you know, taking all the pieces and throwing them up in the air) and having it Just Work after two or three tweaks. Furthermore, I will not bore you with the details but I have not even started leveraging the full capabilities of qooxdoo's object layer: even more benefit will be reaped moving forward because of some nifty reactive capabilities.

This came as a pleasant surprise. I do more OO in more ways than you can imagine in part because I normally program with Lisp's CLOS which is to normal OO what normal OO is to binary arithmetic but by the time I stumbled onto qooxdoo I had been doing Javascript, HTML, and CSS for a couple of months under a lot of pressure so I never had the time to build a sensible OO abstraction atop Javascripts primitive little OO pretensions. Thus Javascript had endowed me with a bit of child mind and I was able to experience the mundane code structuring win of OO as if for the first time. It rocks.

3. Layout the Wazoo
I may have lied. This may have been the first stunner I felt in my gut. The others I knew would be huge if they panned out, but this was the first one I saw in action working as beautifully as had the other frameworks left me hanging. With other frameworks I could find no way to get built-in layout schemes to use the full window real estate available. That cuts two ways: I either ended up with unused whitespace (and with our app we need all we can get) or my application panels grew so large that the browser itself put up scroll bars, meaning I had to window-scroll my scroll panes around. No, that was not a candidate for production release. Note that to a large degree this is the same as point #1: a framework is not hiding HTML/CSS from me if I am still victim to surprising browser decisions, such as oh gosh that is big let's use a scroll bar here.

How good is qooxdoo at layout? When I popped open the Firebug debugger in FireFox, where by default it begins by seizing application window real estate, the qooxdoo layout engine handled the downsizing impeccably, right down to adjusting scroll bars to accurately reflect the new dimensions.

4. Message In A Bottle

This feature is huge in a small way, by which I mean I could probably roll my own version of qooxdoo's Message mechanism in a an hour but--wait for it--I did not have to. They did. That is huge, the huge thing being that the qooxdoo team really has set out to create a compleat Web application development environment and with such a beast comes pleasant surprises such as their Message mechanism which allows a nice decoupling of widgets: one widget can dispatch "Make it so" and anyone responsible for "it" knows to roll up their sleeves simply by having subscribed to "Make it so".

Beyond the Message mechanism we have the Event mechanism. The documentation of every widget lists the events it might generate and what data the event will carry, and in toto we have an engine with an event model dozens of times richer than pure DOM.

5. I know, I said "4"

Number five is an umbrella win, too detailed to list: as I whined above, this Lisp God is now writing an awful lot of Javascript leveraging more and more qooxdoo all the time. Again and again your highly critical correspondent is finding nothing but good news in qooxdoo.

Yes, I could make a bug report every other day if I had the time, but a long time ago I figured out that I am not looking for a perfect library, I am looking for a powerful library that is fundamentally sound (so anything broken can be readily fixed) and actively maintained (so things actually get fixed).

Summary

qooxdoo: fast, powerful, complete, rich, well-engineered, and it hides browsers, HTML, and CSS. Great doc, loads of examples, fine community. It's all good.

The Road to qooxdoo - Part II

2008-12-13T23:24:00.006-05:00

Last time, we learned how one self-appointed Lisp God with no knowledge of web programming ended up programming the web and then soon enough embarked on a six week grand tour of Javascript frameworks in search of a platform suitable for an enterprise web application and where his tour ended: qooxdoo. That episode ended with a promise of more information on qooxdoo, but first a note on the methodology: there was none.

My first experiment with web programming had thanks to the marvels of Lisp in general and AllegroCL's web tools in particular mushroomed into a full-blown web application aided here and there by jQuery widgets before we slowed down enough to think, Damn, we could benefit from a full-blown Web application framework.

So I eyeballed the usual suspects and ran them through googlefight and picked the most promising one and used that to redo what I had accomplished to date. Which was: tab controls, select boxes, data grids as views into server-side data stores, tree views, text input, tool tips, pushbuttons, pop-up windows, and probably one or two things I am forgetting.

How did I assess the libraries? We needed a professional look, good performance, and maximum productivity. Sorry if that seems blindingly obvious. The productivity assessment has some beef: I looked for lots of documentation and tons of examples describing a rich set of widgets at once high-level, feature-rich, and yet highly authorable which may not be a word but it means I get to override stuff without resorting to back doors.

I considered also two metaqualities: on-line support and internals code quality. Having the Lisp HyperSpec an F1 away is just a little faster than asking for the same information on comp.lang.lisp. ie, Community matters. As for code quality, the good thing about open source is not that it is free, it is that I can use the source to understand things at 3am when the community might be quiet; I can add print statements to debug my code; and maybe I can find and fix a bug in the library. But if the code is of low quality good luck with anything more than the print statement insertion.

It was a rough six weeks. Three times in six weeks I logged onto support groups and said, "Hi, this is my first day..." as I rebuilt a relatively feature-rich web app three times while staggering up to speed three times on three environments. All while the client is getting absolutely no new functionality. I feel bad all the while because they need this stuff now. I am letting them down and there is the strong possibility they will be letting me go: I had been brought on to do the Web front-end. I like learning, but not so much with a gun to my head and a time bomb ticking.

Are we having fun yet? Well, yes. The last framework we tried was qooxdoo and it has worked out great. I can now give the client pretty much what they want fast and looking good once they pony up for a graphic artist because I just saw this impressive example of what can be done with their "appearance" framework. But I ain't no designer.

Funny thing, though: qooxdoo like YUI and Dojo came up short during the evaluation. It had no example of the widget I needed most, a fairly standard widget at that: a datagrid backed by a remote store. Bad sign. Second, qooxdoo did no better on googlefight than does Lisp, and that is pretty bad. Finally, qooxdoo like YUI lacks a declarative programming model [though the declarative QxTransformer has been revived since my original investigation] which means I cannot just sit in my Lisp IDE catching XHRs and tossing back HTML/CSS. This would put a non-trivial dent in my productivity.

So how did qooxdoo prevail? What can I tell you? Bullet charts are useless, expert systems work great as long as expert humans interpret their output, and resumes tell us nothing about the work we will get out of people.

qooxdoo did have a contributed example of a remote data store which pretty much worked and when I looked at the qooxdoo internals to learn how to make it work better I found very good code and rather quickly had the datagrid I needed. Otherwise qooxdoo did have loads of examples and documentation and decent community (I say "decent", you'll say "great"--my standard is comp.lang.lisp) and then we get to the bottom line.

While resurrecting my app for the third time with qooxdoo, yes, I worked just as hard as I did using Dojo and YUI. But the effort was different. With qooxdoo I was struggling to learn a very nice framework that needs an index: I was slaving for six hours to find the simple six lines of code I had to write. With YUI I was struggling with a framework that did not work all that well and scared me more and more the more I looked inside it. With Dojo the struggle was with a good framework but the struggle was such that I suspected the struggle would never diminish and probably increase exponentially as the application grew in complexity.

As for the declarative thing, hey, I am a programmer, I can write code. And Javascript has a lot of Lisp soul, I am not stuck with something daft like PHP or Python. Sure, having to spend more time in a JS IDE means a ten percent productivity hit but ten percent we can do. The terror we developers dread is the tool nightmare death by a thousand cuts two steps forward two steps back rope drag death.

I will blog on rope drag sometime soon, but first let's get specific about qooxdoo. In Part III of my Road to qooxdoo.

The Road to qooxdoo

2008-12-12T20:19:00.010-05:00

[The title of this rather boring entry derives from a fun little survey I kicked off years ago.]

OK, why should you listen to a Lisper on Javascript frameworks? Because we use Lisp so obviously we have better instincts when it comes to developing applications. In fact, this road would not have ended so well had not Lars, a fellow Lisper developing a killer new Lisp Web programming framework called Symbolic Web (SW) brushed aside my enthusiasm for Dojo and mentioned he found qooxdoo interesting, although for now SW uses a little jQuery. "A little" because the whole idea of Symbolic Web is to do most of your programming in Lisp which of course is the only way to go but SW is still emerging from the sea and I had a requirement now (well, six weeks ago, but there is major new release of SW just out I need to look into but as this Road will reveal I am not likely to change horses now because of what I found in qooxdoo (not that SW will not eventually catch up)). Gasp.

The second question might be how a Lisp god and Interwebby know-nothing ended up doing heads-down intense Interweb development. Well, it was the Lisp credentials. A former client using mostly Lisp was having great luck with a Ruby/Rails front end but was looking for more speed and hoped having a Lisp image serve the pages directly would make for better response.

What happened next was pretty neat, and again Lisp conquers all. I told the client I would look at what they had and what they needed for a few days to decide if even I could help them, because I can do a lot of things but one thing I cannot do is exaggerate how little I knew about Web programming and even static HTML eight weeks ago.

A glance at the existing screens and Rails code had me thinking, "No way", but I was curious anyway about even getting an application to drive a web site so I fired up a tutorial on Web app development using tools from my Lisp vendor of choice, Franz.

How good are Lisp, Franz, the tutorial, and the WebActions tool? In the three days I was supposed to be deciding whether I could help the client I managed to execute a respectable fraction of the first component they needed. Sorry, no screenshots, NDA and all that.

So now there I am with a contract and a browser toolbar sagging from the weight of bookmarks to sites on HTML, CSS, and for the hell of it jQuery though I was not sure why but everyone was jumping up and down about it. And then I needed it.

Well, not jQuery. I needed a razzle-dazzle spreadsheet-like grid presenting data from an arbitrarily large table of data held on the server. Now at this point I had already rolled my own Ajax paging mechanism, but something superficial but nonetheless important for that this being an important application was holding me back: my HTML looked like crap. Meta-holding me back was my sense from surfing the Web that getting good results out of HTML/CSS was a black art requiring years of practice which certainly leaves me out. Enter FlexiGrid. A URL which as I write responds "Please contact the billing/support department as soon as possible."

Which brings me to why I dumped jQuery. What I found was small, yes, and add-ons, yes, but a hodge-podge of add-ons does not a professional development environment make. By the time I punted I had contribs from three people and while each was lovely alone together they looked like, well, a great decorator on a multiple-personality binge. And beyond appearance there came a surprising result: FlexiGrid did not offer a "row selected" event. Meta-worse was the author's response. "Right, no row-selected event." Ooops. Next up was jqGrid and that was fine but I still had Crazed Decorator Syndrome going and now I am looking for the next widget I need and a light went on: we are building an application here, not a Web page/site. It is wonderful that jQuery is small, but small is exactly the wrong thing for developing a sophisticated application with a coherent look and feel never mind coherent engineering underneath so widgets can play well together. Enter Dojo. But first...

In case you are wondering where the client went, they are encouraging my search for a pre-fab solution. Probably seeing the gorgeous FlexiGrid widget compared to my ugly efforts did not hurt, but the client has good instincts anyway (they use Lisp, right?) so they are leaning towards my original assessment that I am overmatched if I am going to try to turn out a hot application on short experience with HTML and CSS even though at this point I have wondered aloud to them how much HTML/CSS I might have mastered in the same time I was spending beating my head against pre-fab, this being the classic failed deal we make with the devil when we reach for 4GL: yeah, wow, a complete browser screen in five minutes. Gee, could we get the first sub-item appearing on the same row with the item? No. The client wants it. It cannot be done. That was Datatrieve on VAX/VMS twenty-five years ago, but nothing has changed. 4GLs get their apparent power by making decisions for me which works only if I (and my client) are not making any decisions.

But no, the client says "find a framework" and I decide to try again with Dojo. Again?

I had looked at Dojo before. The word on Dojo was incredibly good. Of course that word was from their Web site. When I tried to confirm by eyeballing the doc I could not find it. I mean, there were some great links, but all the pages they led to were blank. Next! But now I was back, encouraged by the news that OpenLaszlo had just placed a bet on Dojo. IBM as well, but OpenLaszlo is more important because I know they are geniuses, they have a dataflow hack like mine.

Oooh-ooh! Another reason Dojo looked good was a benchmark I had found showing the thing was fast. I have no idea if the benchmark is valid, but the pictures are gorgeous so I trust it. So why did I dump Dojo?

Well, OK, I did not really. It came in second. The question is why did I keep looking such that I found the eventual winner. Easy: too frickin hard to get to work. The documentation is awful. They know this and are addressing it but did I mention we need this now? Overall Dojo fit the bill of being a Compleat Solution with solid backing and a great future, but after banging my head against the wall over something (ah, it was the six hours I spent trying to figure out the "build" mechanism for a compressed production release of my JS culminating in watching a blurry video that finally showed how to do it) I went for a better way surf (as in there's gotta be) and saw YUI touted as a serious solution.

A quick glance at the enormous volume of well-organized documentation and exhaustive examples and the client and I agreed at once to give it a try. And I will tell you right now YUI has the best graphic designers, because their stuff looks hot. Dojo is a close second there, too. YUI is a complete solution with a big backer, gorgeous style, and more doc than you can imagine. How did it get dumped?

Stuff was not working well. Even with all the doc and examples it was hard to get working. When I dug into the source to see why things were not working, I did not like what I saw. And the death knell: YUI abandons the declarative model. Why is this so bad? Hellooooo, Lisp? Remember?

Up thru Dojo I was programming in Lisp 90% of the time, doing just enough JS to get information back to the Lisp server application where HTML could be generated and sent back, even that Lispily thanks to HTML generating macros. And YUI was going to make me give that up and be as hard to figure out as Dojo and I suspect worse under the hood. So regretfully I informed the client my battles with Dojo would be resuming.

So how does this road end up at qooxdoo?

The weekend was coming up. Weekends are mine, so I decide to follow-up on Lars's hint about qooxdoo. qooxdoo is the ultimate threat to YUI because qooxdoo also says Just Code Javascript, and if I am forced to do that by YUI I may as well do it in qooxdoo where the code is as good as YUI's is bad. (I looked at about fifty lines of it for five minutes, I should know.) Anyway....

qooxdoo is amazing. They need help on the graphic design (never look directly at their tab control, use a mirror) but other than that the engineering is terrific. [My bad: they simply leave anything more than the utilitarian look up to the developer, witness this before/after case study.] The documentation and examples are also abundant here, and the support is great. They even do dataflow! Yes, I wish I could spend all my time in Lisp sending over HTML, but two things on that.

One, there is a side-project that brings a declarative model back into qooxdoo. QxTransformer. I have no idea where that stands, but I did see a note on-line where a developer spoke of it in the past tense. [Now resurrected.]

Two, hey, I can write some Javascript. I did ten years of C before I did fifteen years of Lisp, Javascript is not a problem. And it even has lambda, something this guy still does not understand. Lisp/markup would be better, but when dealing with mission-critical stuff (the client is doing valuable data crunching but does not win until they make it accessible) any team can roll up its sleeves and crank out the work even if they have to do without the brilliance of Lisp.

This entry is getting long and I am getting tired and my bartenders are starting to worry about me so I will leave a detailed praise singing of qooxdoo to my next entry, but allow me a bit of a rant. What a mission-critical effort cannot stand is fundamentally flawed solutions that seem to offer a fast track but in fact bleed developers to death. This can be jQuery with an incomplete solution forcing me to try to build an application out of mom and pop contribs or YUI with fundamentally bad code or some other framework that might be just plain slow.

As a developer I am not looking not to work, I am looking to have my work yield quality results predictably. I might spend four hours trying to figure out how to do something in qooxdoo but once I do it turns out to be insanely simple and work beautifully. Moral: their documentation needs an index! But I am keeping my big yap shut because documentation is a bitch and they have done a ton of it and I do not document my stuff so...I keep my big yap shut. But the point is that my work now yields results reliably and predictably and this will accelerate as I learn the framework and even learn better how to navigate the documentation.

qooxdoo not only fits the bill of being a complete solution well documented with many examples but it also has high-quality code under the hood, which both gives me confidence that I have found the right tool and not incidentally serves as useful further documentation. More next time.

Why Lisp Packages are so Easy (Hard)

2008-12-07T12:38:00.001-05:00

Slobodan Blazeski wrote:
> May somebody please explain me with simple words why cl packages are
> hard to use, 'couse I really don't get it?
> The best is probably to hear from somebody who just learned lisp or
> somebody who teaches lisp (do such people exist anymore?) and his/her
> students has problems with packages.

...or a good teacher, defined as someone who remembers what it was like before they knew what they were teaching.

The first problem is one of those immune system deals. Packages have a lot of the same proteins as, say, C dot aitch definition files so the recognition systems of programmers coming to Lisp from other languages (you know, everyone) does its damndest to parse them as dot aitches.

In fact, packages have nothing to with anything but symbols, neither the functions nor global values they might name. More expansively: packages are about mapping string names to first class Lisp symbol objects and dividing this up into discrete mappings called packages so the same string "Hi, Mom!" can map to multiple |Hi, Mom!| symbols in multiple packages.

The second problem is that symbols take us straight to black-belt Lisp in which one must understand the Lisp reader and fancy things like read-time and compile-time and all that. Example:

I remember getting messed over because some symbol was mysteriously appearing the base CL-USER package. I could not for the life of me see where it was being introduced. I turned the problem over to a few trees worth of monkeys and one of them came up with this rule:

"Never use a normal symbol like |Hi, Mom!| in a defpackage form coded in a source file with (in-package :cl-user) at the top which would likely be the default even without that."

Instead, use a string like "Hi, Mom!" or an uninterned symbol like #:|Hi, Mom!|.

Strings are dicey because you might want to export 'banana and be silly enough to code "banana". Lisp will take you seriously and create |banana| instead of BANANA, case sensitivity being what it is except when it is not when Lisp folds 'banana to 'BANANA which is what it does unless you are in Modern (case sensitive) mode.

Did I mention "black-belt"? The fundamental idea is simple, but noobs get thrown off mostly by the non-simular simularity with C and then in the debugging stage by case sensitivity seeming to come and go at random and the whole read-time compile-time thang.

hth, kxo

A Beginners Guide to ASDF (Ha!)

2008-12-07T11:17:00.003-05:00

OK, there is at least one thing about which Lispers are not so smug: ASDF (Another System Definition Facility), the de factor standard for the Lisp library equivalent of Makefiles, and my post today was prompted by ANFOBA (Another Noob Effed Over By ASDF) washing ashore on comp.lang.lisp, and how I almost responded there:

Francogrex wrote:

Hi, I am struggling to understand how to work with asdf.

I have been using it for ten years and I am about 50-50 now on getting it to work. I am assured it does a great job in working with a multitude of Lisps/OSes, and I can assure you it is a disaster at working with programmers.

Come back here and post console output freely, you will need help, especially because one of its great triumphs is useless error messages.

Hints:

(1) It will work. You will have to promise it your first-born, but it will work.

(2) No, PB was not joking: you have to separately push locations onto some magical special variable. Leave off a slash (or add a slash, I forget) and ASDF will take your head off, so don't do that.

(3) No, PB was not joking, the syntax is:

(asdf:operate 'asdf:load-op :cl-pdf)

Why is it not (asdf:load-system :cl-pdf)? In my experience, extremely smart programmers think it astoundingly brilliant when "one function does everything!!!!", remembering the ridiculous op-code first parameter left as an exercise.

Since you asked, this is because extremely smart programmers are tone deaf to programmer productivity; they are so smart they do not even notice when they have created something unuseable, they just brain through it.

(4) ASDF has a featurebug: it loads in the wrong order in order to make McCLIM work. Don't ask. If the author of the .ASD file retreated to the desert for forty days to slave over the dependencies this will not be a problem. Otherwise do what I do: take a sledgehammer to the beast:

(asdf:operate 'asdf:load-op :cl-pdf :force t)

That abandons the whole point of having a build system, but it gets ASDF to work. When for the love of god I am just trying to build this frickin thing!!! that is a nice tradeoff of compile-time vs tracking down and short-sheeting the author of ASDF.

(5) On success ASDF emits more messages than an IBM OS/360 core dump. This is so any errors will scroll well out of view and anyway stand out like a healthy thumb from the surrounding...um, other healthy thumbs. Anyway, do what I do: ignore all messages other than a backtrace and Just Try the Hello World(tm).

If you get a backtrace, Just Post It to c.l.lisp.

(6) If hello world fails, f*ck! But at least now you are a Real Lisp Programmer(tm), this is what we live with.

(7) Help Is On the Way(tm): http://mudballs.com/

hth, kenny

The Foisting of An Infinite State Machine

2008-11-18T21:08:00.005-05:00

"I will not allow you to foist this convoluted scheme on the bank!", Sam yelled.

This was fun in a lot of ways. First of all come on this is a bank, three or four full floors of programmers all enthralled by nothing more technical than getting through the day unfired and back on the train home to their families and here is this aging cardpunch nutjob throwback not only noticing Someone Else's Code but caring?!

Second, I did not make that up. Sam actually used the word "foist" without trying. I use SAT words all the time but I am usually trying, you can see my eyebrows arch a little as my language organ reaches for the thesaurus. Nothing shabby about "convoluted", but "to foist"? And Sam tossed that off in earnest in anger in the heat of diatribe sans affectation and it is not clear to me I have ever before or since heard the word used in vitro. Which brings me to the third fun bit.

Sam was Jewish. Classic Orthodox home-by-sundown-on-Friday Brooklyn Jewish and of course he would be good with words. Right, get all PC on me and denounce me but what can you say?, he used "foist" in a sentence on the fly and I think I win on this. So there he is with this classic Billy Crystal Princess Bride accent yelling at me for foisting convoluted code on the bank.

"I will not allow this!" he cried. "I will stop you!"

Oh, please, tho Sam was a sweetheart. Yeah, "was". I heard he passed, Lord keep him, a beautiful soul. I could have choked him in that moment but at least he was attacking me for Code Quality. What was my sin?

I had been tossed a throw-away job of persuading two applications to exchange information reliably when either app was free to collapse in a heap at any time and I was not to lose any information. Nowadays we just pull packages off the shelf for that but this was a while ago. Oddly enough in my first IT job ever I had had to do the same and it was fun because there was nothing theoretical about it, the two applications did even during development reliably disappear left and right allllll the time. Decnet, PDP-11, RSTS/E, 1981... you understand.

What was different was that somewhere in the intervening ten years I had gotten this crazy idea of creating a domain-specific language to make it easier to develop a software Algebra tutor so I had gotten a book on compiler design because I did not realize I could Just Use Lisp. I do not like reading so I did not get very far but fortunately one of the earliest chapters was on parsing so I ended up learning about finite state machines. Oh, sorry.

This post came up because I am doing some Web2.0 programming these days from Lisp and I wanted to send over Lisp sexprs and since Javascript for some reason lacks a Lisp reader I was going to have to parse it myself and after ten minutes of trying to do it without a finite state machine I realize, damn, I should code up a finite state machine.

I will not in this space try to explain what is a finte state machine but it is really simple and powerful and no matter what level programmer you are you need to have it in your, well, toolbox. I think the example I learned on was a pocket calculater when folks still had those. Your initial state is "initial". Duh. Then what happens? Oh, I got "/". Bzzt! Or just ignore it and my next state is still "initial". Now I get a "1". Great! Save that, my new state is...I dunno..."got a digit". We're on a roll. But wait, what other inputs could I get? A "+". Ummm...well, it seems no different, but we'll say "got a plus sign", and indeed when we are done one of the cool things that can happen is that as we fill in the table we may well discover that the row for the state (did I mention that rows were for states and columns were for inputs? Sorry) ..the row for the state "got a plus sign" might be no different from the row "initial" and we first go "whoa, enlightenment" and then collapse the two rows into one. I digress.

But the idea is that we just break it down into cases. A fun example is that we now know whether a "+" is addition or a plus sign (or an error) without really trying, because we have aggrssively exploded the analysis into all these different states like "just got a digit" or "just got a left parens" or...yeah, it goes on, but guess what? Here comes the name! "Finite State Machine"! There are only so many!!! You may think there are a kabillion but after ten you are done, maybe after twenty. And you can collapse inputs (1-9 become "positive digit") and so you have a finite number there. Do you now have 150 cases to handle? Yeah, and each one you can code in your sleep. The alternative is one hyperglutionous mass of conditionals, tests, and branches that will fail on the one millionth execution because of a first time path execution.

Back to Sam.

I have realized that the reliable exchange between two autonomous systems permitted to fail at any point will best be solved using my old friend the finite state machine and I have enthusiastically shared this with Sam. Somehow I am now being denounced for using an algorithm learned from the top book in compiler design. Specifically, I am being denounced for foisting. I will be stopped. The bank will be saved from my convoluted scheme.

Ooooookayyyyyyy.

Well, what can one anachronistic lost in the Torah yamaha wearing card punching goto coding nut job do? Would you believe he could force a code review? Nay, a defense. Sam forced a frickin defense of about thirty lines of the best simplest most powerful correct code that bank had ever seen. Mind you Sam is also the man who produced my favorite line of code ever, using the Vax Basic statement modifier mechanism:

return if flag

That's right, an action at a distance code teleportation straight out of the middle of seventy line for loop using a flag he had named.... flag. You can imagine my reaction. And this guy is challenging my code?

If you are not yet suitably horrified, understand that novice programmers were regularly crashing production runs because no one anywhere was looking at their code and they were going to call a meeting of like twelve programmers to review my foistation of convolution upon the bank? Sorry, my ire is showing. Chase cut to.

Code review/defense: twenty minutes, here it is, any fucking questions?

Yes that was pretty much the tone of my presentation -- I like to consider people skills one of my strengths. And then Sam spoke and you will see why I loved him. At pain of repetition, probably no one else on those four floors would have even listened to me as I bragged on my FSM, but go to the mattresses to stop what he thought was bad code? One in a million. And one in ten million in my experience would have said what he said.

"OK," Sam he said. "That looks fine."

Of course now I am pissed off because I am just getting my turbines spun up for a full bore antler to antler knock down drag out kickin and a gougin in the mud and the blood and the beer round fifteen thrilla in manila finish and Sam is saying...fine?

"That is simple," Sam continued, gesturing towards my whiteboard diagram. "This infinite state machine idea is complicated, but the way it works is simple."

The beauty of that being......no. I would only ruin it.

RIP, Sam. RIP.

URBBR #2: How Bad Is This Book?

2008-09-09T14:17:00.006-04:00

[The following has been... well, only the names have been changed to protect the endangered toads.]

We have a One-Paragraph Test for fiction. Not a complicated test, it involves reading the first paragraph and then deciding. If We are really undecided We can read a second paragraph, but that is a Bad Sign.

For non-fiction We look at the blurbs the author managed to suck out of unwilling blurbers. Well, OK, We do not really want the blurbs to be from blurbers, We want them to be reviews and We know they have cherry-picked the review sentences so everything gets discounted about an order of magnitude.

And there are two such reductions applied, the second for the source of the review. Kirkus Review of Books... well, just quoting them gets a book back onto the shelf. In the wrong section, upside-down, binder in.

Here is the Gold Standard for non-fiction back covers (hmm, the URL goes to the whole book, be clever and click "Back Cover"): The Glory of Their Times. Yes, you need to read this book, and no I do not give a rat's ass if you like baseball just read it.

We turn now to a book I have been asked to review, which will go unmentioned for the same reason I restacked it binder-in:

"This work strikes a balance between the pure functional aspects of Blub and the object-oriented and imperative features that make it so useful in practice, enable .NET integration, and make large-scale data processing possible."

The clever unwilling blurber always manages to say something nice by talking about something else, in this case the language .... instead of the book for 90% of the blurb. What does he say about the book?

It strikes a balance. Wow. How big a put-down is that? Note that the reviewer did not say "a perfect balance" or even "a good balance". And what the Hell does balance buy me anyway?

The good news is that We cannot discount that 90%, Our micrometer does not go that low. But We better check the author of the blurb.

—John Doe, PhD, Researcher, Blubber Ltd.

Oh. The people selling Blub.

The neat thing is that We now know the book is awful and that John was slowest to hide under his desk when the marketing people came around looking for a recommendation.

Unread Book Review: Zen Style Programming, Nils Holm

2008-09-09T06:19:00.006-04:00

On comp.lang.lisp, Nils M Holm wrote:

> I am happy to announce that free pre-release copies of my latest book
>
> "zen style programming"
>
> are now available.
>
> Blurb
>
> The primary purpose of programs is to be understood by fellow human
> beings,

Nonsense! Nils has been reading Knuth or something. The purpose of programs is to model other processes so the most important virtue of a language is the ease of building programs with it. Yes, those programs should tend to be clear but that just requires powerful constructs, long names, and support for the functional paradigm.

Besides, "revise" means "rewrite from scratch anyway because the way this was done is all cocked up" so we do not generally have to understand programs to work on them.

> hence programming languages should have certain properties:

Bzzt! The source of obfuscation is the programmer, not the programming language.

> * Small size and uniformity

Hand-waving! Nils has been reading Graham or something. Small size as a requirement does not follow even if one were to exalt human readability to Prime Directiveness. You could just have good names for functions, like pathname-sans-file. And an editor that knows how to get to a hyperspec.

   * Unambiguousness
   * A high degree of abstraction
   * Achitecture neutrality

The first part of this book introduces the concept of functional programming and describes a purely symbolic language that fulfills these requirements. This language is a minimalisti variant of Scheme whose only data types are symbols and ordered pairs. Nothing else is required to describe algorithms for solving a variety of different problems.

The second part of the book shows how to apply the techniques of  symbolic programming to some problems of varying complexity. Topics discussed in this part range from simple functions for sorting or permuting lists to ``lazy'' data structures, regular expression matching, formal language translation, and declarative programming.

The third part, finally, shows how to implement the abstraction layer that is necessary for solving problems in an abstract way on a concrete computer. It reproduces the complete and heavily annotated source code for an interpreter of symbolic LISP and provides an example of clear and readable C code.

This book contains the full source code to a source-to-source compiler, a meta-circular interpreter, a logic programming system, and to the language that is used to implement all of these.

Damn, that was a lot of work. I should be nicer to Nils. Here's the book, anyway. Let us see what it says:

"The first points are no-brainers."

Come on, Nils, we talked about the hand-waving.

"If a language is too complex..."

Well, by the meaning of the word "too" I think you are on safe ground for the rest of that sentence, but it begs the question of how too is too? Nils needs to make his case or judge someone else's case, not both.

"...programmers will have to look up things in the manual perpetually instead of concentrating on the actual problem."

So now we need programs to be understandable by people who do not know the programming language? I think that was a design goal for COBOL. Anyway, that does not give you a language with a small instruction set, it gives you a language with long, well-chosen names. Like COBOL.

As for looking things up, I want to meet the guy who looks up delete-duplicates when the status bar is saying:

delete-duplicates sequence &key from-end test test-not start end key

Let's see if there are any more self-serving rationalizations....

"The class of problems that can be solved by programs is much smaller. It typically involves very clearly defined tasks like
-- find permutations of a set;
-- find factors of an integer;
-- represent an infinite sequence;
-- translate formal language A to language B;
-- find a pattern in a sequence of characters;
-- solve a system of assertions.
Of course these tasks have to be defined"

Nils must be a professor of computer science. Programs exist to model real-world entities of arbitrary complexity, tho good luck with the human mind. OTOH, simulations of car crashes are admitted into evidence by finicky courts and some people even model human private Algebra tutors. Unfortunately those things are a ton of work which is why you will not see academics doing that.

Now about the title. Zen zen zen, what an affectation! Zen this, zen that. I am going to write a book on The Zen of Ordering Take-Out Pizza.

What is zen about zenlisp? Never justified, never even addressed. Does the author think zen means small because it is a small word? I like that, actually. I tried googling for synonyms: oops. I think they would be pleased down at the monastery.

So much for the name. How about the language itself? Raison d'etre?

Zenlisp is similar to Scheme, but simpler.

That sounds like being rescued from a desert island and then dropped off in the middle of the Sahara. No, the only reason ZenLisp exists is because Lisp is a venus flytrap of a blackhole delivered by John McCarthy an alien from outer space sent to stop good programmers from Actually Programming. Apparently Grace Hopper was just a few lines of code away from anti-gravity and time travel so They sent John down with Lisp so any good programmer would get so excited about the language itself they would stop programming time-travel applications, take out their CodeWarrior CDs, and start implementing a Lisp using (wait for it) C.

The ones who really want to avoid programming will then write a book about their language.

So I wonder what Nils will do now.

AA, BB, CC, and DD

2008-07-25T03:00:00.004-04:00

I am not making it up. Those were the datanames in the code I had inherited from the Wunderkind. So this will be the most boring stating the obvious war story I ever write but that is not why I am writing it. I am writing it because the Lisp subset of the human race is at it again running factual red lights with their prior conviction feet to the pedal.

Yes, kiddies, we revisit today The Unbearable Impenetrability of the Lisper. The cool thing being that Arc is again involved albeit peripherally in this latest train wreck of human comprehension.

Back then the Maddening Crowd was utterly fascinated that I liked Arc which I did not but they were not to be denied, the best part being those people who responded to my objection to the crowd's misperception by saying yeah I saw you were flabbergasted by everyone thinking you liked Arc so tell me, you like Arc?

No I am not making that up. Twice.

This time it was my scrimshaw-ready datanames, sample below. Some months after my exploration of Arc I had a laugh as I whipped up a DSL for my Algebra software and found myself approaching Arcitude in the brevity of my names. I posted something to comp.lang.lisp inadvertently loosing The Hounds of Lisp Density. A sample:

(hard

(dsb (b x) (rp 2 (rv))

(m/ (m^ b (m* (r2 7) x))(m^ (xqv b) (xqv x))))

(dsb (b x) (rp 2 (rv))

(m* (m^ b (ms* (r+ 7) x))(m^ (xqv b) (ms* (r+ 3) (xqv x)))))

(dsb (b x) (rp 2 (rv))

(m*eo (ms^ b (r+ 7))

(m^ (xqv b) (ms* (r+ 3) (xqv x)))))

(dsb (b x) (rp 2 (rv))

(m/eo (ms^ b (r+ 7))

(m^ (xqv b) (ms* (r+ 3) (xqv x)))))

(dsb (n d) (rv 2)

(w (k (r+ 12))

(m/ (m^ k n) (m^ k d))))

dsb is short for the Lisp destructuring-bind.
m/ is short for make-fraction.
m*eo is short for make-product reordering the factors randomly (either-order)
etc etc

I gave a tip of the hat to Arc and explained that I had done this before (in C) and that this code (to generate randomly many varieties of Algebra problems) was a known PITA and even in C I had used the C preprocessor to likewise make the coding manageable.

Enter the Savages of Comp.lang.lisp.To a geek they lectured me on the importance of nice long meaningful names, or as His Sulzberbergerness edified me a ways back what Confucius called The Rectification of Names which is not quite the same thing but I can never resist name-dropping either Jay or Confucius.

I responded to the jackals nipping at my heels that yeah I know but in this case with vast incessant repetition of a small collection of opcodes that recognition would not be an issue and that it was much better to diminish the low information content (cue Shannon) of long names (what exactly does make-product add to m* in a context where a leading m is used only for makers?) and learn a dozen opcodes which were mnemonically and predictably built anyway from atoms such as M and * and EO.

The universal response was that longer names were better. My universal response was that these special circumstances flipped the arrow on that otherwise sage rule.

The universal response was that longer names were better.

Trying again, my universal response was that I agreed, but would anyone like to address the salience of the special circumstances I had suggested were germaine, perhaps explaining how they were not special enough or too special or the wrong damn color?

They all responded that longer names were better and I really started to enjoy things at that point. I was reminded of this desert spider that had a routine for burying any captured wasp and it involved positoning the wasp up just so and then going to dig a hole and then dragging in the dead wasp and these researchers would move the wasp a little while the spider was digging so the spider would be thrown off and start again by repositioning the wasp and no matter how many times they moved the wasp while the spider was digging the spider would just start right over readjusting the wasp and then digging the hole.

The topper was I myself was a staunch proponent of good long names. Tilton's Law of Programming:

Spend more time on the names you choose than on the algorithm.

As for the specific quality of length, we have Tilton's Rule of Abbreviation:

Abbreviate no name less than seven characters long and then only if a good abbreviation is no more than half as long. Rounding down.

Which brings us to AA, BB, CC, and DD. I was working as a body shop consultant in easily my most Tall Building job ever. One day this new guy came on board, totally not GQ, scrawny, smart, energetic. I had no idea he was a first round draft pick, destined for greatness. An employee, by the way. Tom. He dives in and starts churning out a front-end application, learning the HLL and OS and tools all at once, a man after my heart.

At one point he mentions to me a problem. He wants users to be able to make discontinuous menu jumps, say, go sideways without backing up to the menu above (yeah, this was the good old days of modal interfaces) and the programming language would not go sideways. ie, He was using HLL recursion to handle nested menus.

I suggested the obvious: no, you cannot call sideways in a structured language, you have to return from the called function with a "message" always checked by the caller to see if the user should be taken somewhere else. Tom yelled Great! and tore off to code it up.

A couple of weeks later I have inherited the system. Tom was smarter than I: as soon as he had chalked up the win he told his boss to "get some consultant" to maintain it. Moi. So there I am working on the first RFE and I am perusing the code trying to figure out how the hell it works and I find myself slowed a bit by the data names that pretty much controlled everything: AA, BB, CC, and DD. Come on, you think I could make up names that bad?

So over I wander to Tom's desk, clear my throat, Tom looks up.

"Tom," I said. "I was looking at the code. AA? BB? CC? DD?"

Tom burst out laughing.

"Those are just temporary variables!" he protested, laughing even more.

"Un-hunh," I replied. "And they control the entire program flow."

"Well, change them if you like."

As I said, Tom was a smart cookie, he was wiping his hands of the whole deal.

Now it turns out that one of the things I like to do when working on OPC (Other People's Code) is to stare at it and stare at it and when I see some crucial variable playing a big part in things and its name is getting in my way I pick a better name and do a global change, rinse, repeat until the damn code makes sense. The only reason I had gone to talk to Tom was the same reason we pay to see a two-headed sheep at the carnival.

Two hours into the renaming I had come up with decent names for AA, BB, CC, and DD (that last one turned out to be three different variables) and I was making the global changes eyeballing each as I stepped through the source and after one such change forgive me I will never in my life remember the specifics but imagine you have just made the substitution and are now looking at a line of code that says:

total-weight = total-weight + this-length

I did not feel completely comfortable with that edit so I called Tom over. Did I mention he was a smart guy? Two seconds into explaining how I had gotten to that point he yells out, "Great! I gave up on finding that bug!"

Smart guy. Tough bug? He just moved on and chalked up the win. Let some consultant take over the code and run into the bug and think they introduced it.

But there you have it. A bug so hard to find that a very smart programmer gave up on finding it and someone who did not even know the bug existed changed a few datanames and the bug positively jumped off the page. Explaining the corollary to Tilton's Law:

If the names are right the algorithm will write itself.

Now if only some walking fencepost from c.l.l will post a comment saying... well, that would ruin it, would it not?

Now Batting, Dave. Why is Mariano Rivera not worried?

2008-07-18T15:16:00.004-04:00

The venerable Mr. Roberts draws on his profound mastery of things algorithmic as he intones:

I'm actually quite conscious of cons-ing, but I also believe that obsessing about cons-ing leads to premature optimization and far greater numbers of bugs.

We are fortunate to have the benefit of his wisdom and it occurs to me that we can keep talking past each other enjoyably for weeks like this so i will not suggest we try to establish a concrete example where he would code append and I would code nconc and ruin all the fun. But it is a good question especially if we agree (we do) that as a rule the functional paradigm utterly Rocks the Casbah so how on earth am I going to get a list back from some function I call and not be free to nconc it? The moral being I am right (will it never end?): if we try to find a point of Actual Disagreement(tm) we will fail so let us not.

Getting back to the fun abstract theoretical conflict, the problem is a policy of copying which leads straight to the solar power objection aka death by a thousand cuts which in turn leads us back to the simple question of why we do not just code the right operator at every turn? Genius is in the details which means we are always on our game bringing me to Mariano Rivera.

For those who do not know Mariano plays baseball and is starting to be called the best ever at his position, namely "closer" which means we won the first eight innings and do not want to lose the game please do the rest and he does. Even in the seventh game of the World Series that he lost every hitter broke their bat but the three balls involved known in the game rather morbidly as "dying quails" found their way to the grass. Hard to feel bad about shattering three bats. I digress, Mo is good.

Genius. Details. The great Nconc vs Append War, Roberts-Tilton III, nothing like this since Ali-Frazier, Affirmed and Alydar.

It was Just Another Game. The Yanks had won, Mo had saved. They always collar some poor sod and keep him from his shower and car service home to make those of us who pay to watch baseball on cable TV feel better and tonight poor Mariano's number (42!) came up and by the grace of God I pulled my face out of a carton of cold noodles with seame sauce just in time to witness a perfect moment in sports broadcasting and as well a master class in excellence any time any where.

The announcer on the field for the interview went for a home run question, something way more astute than the usual "How did it feel getting the big out?" and he came up with a beaut:

"Billy Bob is a rookie and you have never faced him before, there must not be much of a book on him," the announcer observed. "Was it tougher facing an unknown quantity?".

"No," Mariano replied. "I went out during batting practice to watch him hit."

Mariano's disclosure that as a Hall of Fame shoe-in and multi-millionaire perennial All-Star participant he had looked down at the lineup and seen an unfamiliar name and broken off his clubhouse routine to go watch some ridiculous kid take batting practice had no small effect on our seasoned, professional, expert baseball announcer:

"You did that?!!!!!!". He blurted out, hopping a little in the air, lurching towards Mariano. "You went out to watch him take batting practice?!!"

Now it was Mariano's turn to be amazed, leaning back a little to create a margin of safety.

"Of course," he replied. "I get paid a lot of money."

Doodleoodleoodleoodleoodleoooo. Game over. The announcer (and I, I confess) had been seriously schooled.

We sit at home and watch these athletes precisely because what they do is so wonderful to us and we endow them in our minds with magical and wonderful qualities because at an early age we were robbed of angels and dragons and Santa Claus and even monsters in the closet and trolls beneath the bridge and then every once in a while we learn that the miracle strikeout we saw with the bases loaded happened because a multi-millionaire punched the time clock, checked his glove for broken laces, sharpened his spikes, and checked the opposing team's roster for any names he did not know.

I say learn the damn language.

Fear No Evil II, or Why Solar Power Sucks

2008-07-18T02:55:00.004-04:00

Just a quick note cuz I remembered something that answers Dave's well-reasoned comment:

My own personal rule on this is to use functional programming whenever possible and let the GC deal with it.

Not much of an excerpt, but it says it all: modern Lisp GCs truly Rock the Casbah, and I myself am on record as denouncing the flipside of this blogcoin, consophobia. Actually, there may be a problem, Dave says "whenever" possible and perhaps then we agree!

But "whenever possible" has a ring to it like "oh, you know, all the time" and I actually had a good reason for not doing that, I just forgot it. Has to do with solar power.

The reason solar power sucks even though there is an overwhelming amount of it is that it is diffuse. Yes, enough sunlight falls on Kansas in a month (I am making this up) to power Earth for a year, but just try to find a solar panel as big as Kansas, never mind deal with the Kansans or the inefficiency of the transmission to Australia. I digress.

The problem with a "Damn the consing! Full molasses ahead!" policy is that when ones application drags to a halt there will not be a bottleneck to fix. There will just be hundreds of places one was too lazy to think about whether one could use the right construct. And correcting them all will be as painful, tedious, dreary, mind-numbing (sound like fun yet?) and thus as bug prone as it would be easy and fun whenever reaching for a list manipulation tool to challenge oneself with, gee, can I cut in the afterburner? Stand back, GC! Captain Destructo is in da house!

That is a practical objection. My ethical objection (viz, hey, learn the language) is not all that far behind: jeez, I am building this list right here in front of my eyes, why am I not using a destructive operation for the next step? Do these same folks all have a copy-n-sort function in their personal toolkits that begins with a copy-list on the sequence to be sorted?

No, they do not. Why not? Hello.

My apps periodically get bogged down. I wait until they do and then dive in for what I like to call Speed Week after a regular TV extravaganza on vehicular racing. Normally this turns up interesting algorithmic gaffes I do not feel too bad about but one time the smoking gun was the most innocuous little needlessly copying statement you can imagine write at the center of the heaviest hitter bit of code you can imagine. This is pretty much the opposite of the solar power argument -- there was just one little guy to track down -- but without the code profiler offered by AllegroC I am not sure how long it would have taken to find that.

And now I come back to the main point: er, exactly how hard is it to know when one can use destructive list operations? So why not use them? Only one excuse comes to mind (fear) but always copying is not even a clear win (ie, it can totally suck at times) so.... learn the language?

If I was going to cut the Copying Cowards some slack I would say, shucks, not everybody writes insanely intense tight loops. But then I remember a point I made in FNO/1: we get noobs all the time struggling with slow code because of excess copying. Perhaps complex applications and tight loops are orthogonal?

I say learn the damn language.

Chomsky! Pinker! Get In Here!

2008-07-14T02:25:00.005-04:00

[OK, OK, it is a Lisp blog but if I can refute Chomsky's entire body of work on language acquisition in one post, why not?]

These academic PhD geniuses have a problem. They are geniuses. They think they can figure everything out by thinking, like Feynman fixing the neighbor's radio at the age of three or something close to that by thinking. Great story, buy the book. Forget which.

Chomsky and Pinker merely repeat the error of the AI crowd who sniffed at NI (I just made that up, it means "natural intelligence") and said, "NI?! NI?! We don't need no stinkin NI!". They said that because they were geniuses they knew that what NI did with pretty much organic chemistry they could stomp with (drum roll, please) reason. Ooooohhhh, excuuuuse me!

I guess I can leave Pinker out of this, he did not add anything to Chomsky, he got drugged in only because a correspondent mentioned his "work" on blank slates. I love that "work": Oh, let me lean back and draw on my pipe and I have a PhD so if I think babies learn something too fast they learn them too fast, dammit! As if. And believe it or not I am coming to my point. But first a science trick, a prelude to my witnessing of the refutation of Chomsky. It has no bearing on anything else but it involves a cute girl and something I could not believe when I saw.

There I am in a sports bar/shore bar celebrating a great week almost delivering my world-changing Algebra software when I see a lovely lass drop her drink. Three or four feet, to the floor. It was a plastic cup, which helped a lot come to think of it, because a glass glass being more rigid would not have absorbed the energy and there would have been a wicked rebound preventing what happened from happening: the drink landed upside down and just stood there, trapping half the drink inside. The bad news being there was no elegant way of finishing said drink, but that is not my point. Did I have a point?

Well, if I had to have a point, it would be that no one else was all that impressed. Clearly I need to spend more time in bars late at night watching drinks being dropped and I do not think it is possible to spend more time in bars late at night than do I so I guess I need to find bars with clumsier people, good luck on that.

Oh, and the other useless and irrelevant to the demolition of Chomsky/Pinker thing I learned is that it takes just one unauthorized speech on world affairs (well, he was mumbling pretty badly, I am not sure it was on world affairs) over the unattended band microphone to get escorted unceremoniously out the front door at the sports bar in question. Right, Chomsky.

So I look up and I see a skateboard competition, lucky enough to be watching the winner's winning run, and lucky enough for my brain to clear long enough to understand what I am seeing, a Johnsonian kicking of the rock I refute you thus Dr Berkeley of Chomsky/Pinker by some sixteen year-old if that. What did he do, and why did my brain have to clear? Two good questions.

What he did was give himself a push across a flat floor in a reasonably small room (no gigantic ramps and concomitant acceleration/stored momentum) and upon reaching a ramp at the same time push off in such a way that the skateboard (unattached to his feet for those who do not know) elevated with him to a height just above a rail (er, just an elevated horizontal pipe mebbe a foot off the floor) such that he could land on the skateboard in midair just as the tip of the skateboard landed on the rail and using his/their forward momentum proceed to slide along the rail for a couple of yards before again levitating off to land somewhere else to commence another trick.

And before the anti-blank-slate morons can scream "Oh that is easy, I have the algorithm right here" allow me to break their hearts and add that as he did the initial elevation springing the board into the air with him (a yawner in these competitons) he took the trouble to pop his deck into a barrel roll (or two, who knows at that speed?) before regaining it to land the rail toe-slide. If the MIT geniuses think that is easy lemme just say that I have spent years in Central Park watching the same people practice on skateboards for years and they still cannot jump over a small backback and land on the skateboard on the other side.

So... these people have skateboard organs? Or do you think learning to barrel-roll a jump into a rail toe-slide is more important to them than is mastering language to an infant so their motivation is stronger? This would be a reasonable belief if one has never been in the presence of an infant who will join in any conversation it hears when it neither understands nor can generate one intelligible syllable.

Oh, and why did my brain have to clear? Because what I saw was a commonplace: inconceivable performance easily explained by sufficient practice. If a recent blog had not been on my mind I would not have given the skateboard performance a second thought, nor the motorcycle-flipping organ manifested in the next story on ESPN.

Come to think of it, this is about why Lisp wins: we simply are not as smart as we think we are. I like to say that I cannot write good code but I do know when I have written crappy code. Recognizing is easier than generating. Hell, crappy is just one bit, right? But how do we produce a huge stream of uncrappy bits? Good luck.

What we want then is an agile language that lets us create awful code really fast and fix it as fast as we identify the problems. Pretty soon our software organs cut in and we are creating not-so-awful code up front. Even better.

Now I just have to figure out an elegant way to finish a drink upside down on a barroom floor.

Buzz off, Charlotte

2008-07-09T02:39:00.003-04:00

Omigod. I just made up that title because of the subject of this post and not five minutes ago I ended up daydreaming about a girlfriend I had back in college named Charlotte and as you will see there is no connection. Scary.

Anyway, it is 2:38 am and I am in the middle of a trivial but unwelcome re-de-refactoring with a scant few days to go before a self-imposed but jeez I gotta get this frickin thing out the door it is the middle of July deadline and it is an educational app and how far is it from the middle of July to frickin September?!!!!!!! I digress.

It is 2:38 am and a spider I swear to god no more than 2mm across is dangling in front of my flat planel trying to figure out this code (the last bit of the last line, to be precise):

(defun alert (s oks)

(tk-format-now (conc$ "tk_dialog .oops {} {"

(format nil "I thought you were solving for \"~c\"? I do not see it."

(key-char sfvar))

(format nil "} {} {~s} {~s}" oks oks))))

Charlotte (look it up furriners, E B White) is freaking out over the double occurrence of the variable OKS. I know because the little hussy has lowered herself I swear to God ten feet from the A-frame ceiling of my ranch rental all the way down to within a foot of my desktop and stopped. How still is the air in here? She was swaying no more than another 2mm. I digress. But how does a 2mm spider pack ten feet of gossamer into its, well, gossamer pack?! I don't care how thin it is, it is stll ten feet!

Charlotte was both deeply concerned and a sufferer of tunnel vision. She ascended to the top of the flat panel (I have it angled down to my slouched position a few inches above desktop), gained a foothold or five, and then inched...no, mmed over and redescended to get a better look at the first occurrence of OKS, just to be sure. It is possible she then pirouetted on her thread to ask why I did not check the CLHS on FORMAT to find out how to back up one argument and re-use it, but I cannot be sure. If my eyes were better, I could tell you.

Well, I am in a hurry dammit and you know what I think I have the strength to type oks a second time and no I do not think the redundancy will lead to the loss of a satellite but if a damn spider complains I know enough to listen... ah, there it is ~:*, back up one arg.

Now buzz off, Ch... hey, where'd she go?

It's the shoes, Jeff! It's the shoes!

2008-07-03T02:07:00.004-04:00

My buddy Jeff wondered aloud on comp.lang.lisp why We Lisp Gods were so much better than the rest of you at adapting to alien environments, aka other languages. Without coming down in favor of any, he offered several possible explanations:

We are just smarter than you un-Lisping dopes
We are accustomed to learning other languages because the food is OK in soup kitchens but the decor? Omigod!
We are older and you can't teach an old dog new... hang on.

Based on recent copious experience, you can safely forget #1. Lispers are precisely as dense as other programmers, and if anything more so: as the title suggests, we are being carried by the shoes. Anyone both smart and using Lisp is named Graham or Morris and either has or split over forty big ones, and no, a thousand smackolas is not "big".

#2 is a fine answer but is boring and knocks the legs out the upcoming argument so let us declare it false. That was easy

#3 is part #2 and part what I joked: old may be wise but it sure ain't adaptable, it already knows The Right Way, it aint listenin.

That leaves the title of this entry, a reference to one (or more?) humorous Spike Lee ads for Nike sneakers in which he insists the sneakers explain Michael Jordan's Unbearable Lightness, to which we see his Airness reacting with feet-on-ground dubiosity. IIRC, but I think I do.

But in this case, I think for once Chomsky might find a home. Chomsky decided that because he Chomsky could not understand how infants picked up language that there must be a magical device at play, a language organ in the brain dedicated to language acquisiton just as the spleen is dedicated to damn I wish I had not slept through biology.

Chomsky had a tiny little problem with his nonsense in that when one looks at human languages -- any of which can be picked up by any infant raised in its midst, whatever that infant's genetic tree -- well, the structural variation is infinite. I remember one universality they found: adjectives tend to appear very close to the nouns modified. Well, shazaam!

So the Chomsker has a problem: this super-duper language organ has to be hardwired not to learn any particular syntax, it has to be hardwired to learn every syntax. Ok, that makes no sense, shall we try again?........... um............. er....................... I got it! It is hard wired to learn an abstract syntax! If only we could find one!!! Chomsky failed except for that bit about modifiers miraculously always appearing close in sentences to the thing being modified. Yeah, how come no language ever came up with "The little giant slowly balloon over drifted the mansion red white."? Because the language organ rejected all those! Where was I?

Ah, flexible language acquisition, computer or otherwise. No, it is not the brain, it is the shoes. Basketball no, programming yes. No, Michael did not reach new heights because his sneakers opened new doors for him, but yes: expose someone to Lisp and (as many before me have observed) they become a better programmer.

Chomsky saw the infinite variability of actual syntaxes as a severe objection to his hypothesis, and he was right to do so. Indeed, look at it from the other direction and we see the light: forget infants' ease of language acquisition, look at the difference between those who know one language and those who know five. Something happens after one has learned a bunch of languages -- acquisition of the next becomes a snap. No, I do not have that backwards. There is a small effect of "hey, anyone who has learned five just digs languages and does because they are easy for them", but I learn natural languages insanely easy and get a huge kick out of stumbling thru a conversation in French or Mandarin and learning a new language is still hard for me because circumstances never led me to learn more than what I needed to get out of high school.

No, what is going on is the same as any learning process: the more we do the more the brain wires up metaskill at that activity. Babies are not born with a language organ, they just happen to have nice blank slates and speech is to them obviously something utterly compelling to master, it seems to control these big creatures moving about before them. Nothing like insane motivation for apt learning, eh Dr. Chomsky? But later on in life when a second language is an amusement? Acquisition is brutal. Where's the language organ now? Withered? Why does it come back after I have learned four the hard way? Ooops.

Was I supposed to be answering Jeff's question? My bad. Chomsky was wrong, McCarthy was right: a language in which code can be data and verse visa is one in which the programmer is stripped of structure and stricture and as Sartre warned us is not free to be not free. In mastering any art true freedom comes only within the constraints of the form, but Lisp offers us no such handholding. Some brilliant wag once observed that Lisp gives us a million ways to do anything but (1- million) of them are wrong -- a working Lisper like one who has learned the hard way five natural languages learns a metalanguage of programming that would delight and disappoint Chomsky, because the learned metalanguage is just one more demonstration of the brain's ability to find patterns in whatever it is immersed in long enough. ie, Organ, schmorgan, neural nets simply rock.

Lispers are not smarter, they just fell into a pool and learned to swim -- ah, good analogy, nothing to hold them up, had to make do in a medium that would not support them, they had to add energy to the system to stay afloat, something like that. Lisp not only supports every paradigm, worse, it even allows any paradigm you can invent one as I thought I did with Cells until I looked up the prior art, but that does not change the fact that while working on a little GUI geometry problem a new paradigm jumped into my code... deal with this kind of language and... well, imagine sneakers that really did have wings. Wear those for a year and then climb back into ordinary sneakers...I'll give you a minute to think. What happens?

Disappointment? Some. Depression? No. End of flying? Never. You have tasted flight, you know what is possible, and you want to be there again. Now the analogy breaks down, because guess what? Lisp is just another language. When you do not have it, you can Greenspun it. When subjected to a toad of a language you grouse for fun but you also have code to write and you shrug off the downgrade and fly with the sneakers you have.

In my first programming job I astounded the team so badly they kinda fired me, gave me only the maximum raise and blew me off when I asked for more because in nine months I had finished every system they had in development. At my going away they said they did not go out on a limb for me to exceed the maximum because they knew I would be leaving anyway. They did not know how much I loved the company bowling league or softball beer league. What was my secret in this Cobol shop? I had been programming in Basic on an Apple II.

One day I came upon a senior guy (OK,they were all senior, I had been there three months at the time) staring at my code as he tried cloning it to his assigned, well, clone. Concerned, I paused and asked if there was a problem. He waved me away, saying, No, I just have never seen Cobol like this. I understood. I knew even as I did it that I had been programming Cobol in Basic. I think I have that backwards. I remember asking them -- I was just learning Cobol at the time -- if a subscripted variable could be a subscript and I remember a very nice twenty-year Cobol veteran asking me very gently if I might not be making things a little too complicated. Nah, just trying to do Basic in these Cobol sneakers.

Lisp then is a substitute in itself for learning five natural languages, because it itself is not Chomsky's language organ, it is the universal syntax Chomsky had to identify for his imagined language organ to have been hardwired to learn. Lisp is the programming language embodiment of Sartre's curse: we are not free to be not free. Program this language and one will first use the wrong language feature at the wrong time (hang out on comp.lang.lisp to witness the hijinx of Lisp noobs trying to do things "The Lisp Way") and eventually get to the point where one can look at problems and see the paradigms they bring with them aided always by your language which is being utterly useless at telling you what paradigm to use, it could care less.

ie, It's the shoes.

30Day Death March: If They Can't Take a Joke, F**k 'Em

2008-05-30T16:52:00.005-04:00

The title? Part one is someone's idea on how to distract programmers from working so as to make it even less likely we will get anything done in thirty days but at least we'll have company. Part two is how we pull it off and is what we used to say about the customer on my summer job in the automobile brake repair shop when we felt our work was a little sketchy.

I poked around on the feed and can confirm what someone said: I totally killed myself over the years by thinking certain edge features were indispensible. In my head an imagined reviewer was going batsh*t over something that could be better, like a Consumer's Report automobile editor knocking off two stars from a sport sedan review because their test football linebackers found access to the rear seat problematic. Did you notice it was a sports sedan?! The linebackers can take the frickin bus, OK?!

Even knowing this I did it again this time around (this is version two of something I did in the last century) and I blame Lisp because it made it so damn easy. At first. I failed to notice quickly enough the ensuing death by a thousand cuts for which I thus had signed up. That sentence reminds me of diagramming sentences in the ninth grade when I should have known I would be a social failure when the English teacher and I were doing high fives over the corner cases while the other kids in the class were silently writing me off as... I digress, just watch out for Lisp, it is a two-edged sword.

The crazy thing is that I also know damn well and will tell anyone who asks that this is one of those 80-20 or 90-10 deals, pick yer own damn ratio, OK?: almost all of the value is in the first feature. [kick kick kick] That was for me, not you. Who are you, anyway?

One caveat: sitting in my cubicle above cloud level one day on one of five floors dedicated to IT drones I was stunned to overhear an intelligent abstract discussion of technology. Someone had a theory. I love a good theory. This one said, Don't ship too soon. The point was and my experience had already confirmed that shipping costs. Period. Suddenly one is working on all sorts of crap that does not move the product forward. Once one ships being live costs. This microISV stuff is about one-person operations so guess who is dealing with the cost/burden of being live? And guess another thing: how completely ineradicable are first impressions? Can I write a survey question guaranteed to elicit unbiased responder input or not?

Oops, what am I? The Anti-30day-Christ? OK, let me finish on a positive note. True story.

"These drawings are wonderful," marvelled aloud the pre-school parent to the children's teacher. "You have a great gift for art instruction."

"No," the teacher demurred. "I just know when to take their drawings away from them."

We few, we happy few Actually Writing Applications have as our mission should we choose to accept to take from ourselves our own applications.

"We Can Live With the Way You Handled That"

2008-03-22T17:45:00.006-04:00

Tilton's Lemma had come under attack and the title of this entry was the happy result. You forgot Tilton's Lemma? Here ya go:

Refinements to requirements cannot vary fundamentally from the original.

The point is that careful design is worth the effort so that software can be successfully rolled out even though users naturally produce RFE after RFE during user acceptance testing. No matter what they come up with the software will adapt to it with a minimum of refurbishing.

So what was the attack? We must begin with the launch meeting of the relevant project, where my conduct was such I rarely again got asked to meetings, but as a tease I will share that this was one of the most profound God is speaking to you experiences I have had in software development, one of those moments where one senses forces unknown at play.

I was being given a nice new standalone task perfect for a cowboy contractor such as myself who was good with code but not with teams. My manager Frank would be perfect for such a loose cannon, one who cared not about how loud I snored when I catnapped or at what hour I showed up or how badly my lunch reeked as I ate it over the keyboard but only if the code worked so he could sleep at night.

Just one problem. I walk into the launch meeting and sit down having no idea what is happening. After thirty minutes of sitting quietly at one corner of the table having nothing to do with any of the discussion I have deduced that a foreign branch of the bank has accounting software that will be shut down and covered by the software at our central location and that I will be writing the interface to take the feed from an off-the-shelf humongoid accounting package and feed it into the central system in such a way that it replicates their existing system.

I have grown increasingly intrigued at this prospect as the meeting drags on because our group has more than a few people assigned to supporting the massive off-the-shelf system (shall we call it Isis?) and it seemed worthy of months of apprenticeship. On top of that I certainly had no knowledge of the foreign branch's system. Best of all I knew less than nothing about accounting. Okay, no, best of all was that no one was talking to me at the meeting so I developed this crazy idea that they were just going to turn to me at the end and say, Need anything else, Ken?

Then Tony the Senior AVP turned to me.

"Need anything else, Ken?"

Left to my own devices for the first 99% of the meeting I had had time not only to deduce its ending but also prepare an answer. My fans from comp.lang.lisp can tell you His Kennyness likes nothing better than a good analogy and it had occurred to Himself that the developing situation resonated strongly with an image that formed spontaneously in his mind of a couple with a dog on a leash standing on some corner in the middle of Manhattan deciding how best to get to the Botanical Gardens in Brooklyn and after considerable discussion and perhaps even a consultation with maps street, bus, and subway settling on a means and path of transveyance and turning to the dog and saying, "Lead on, Fido!"

Hence my prepared answer, before not one but two AVPs and the users on conference call from Canada:

"Woof! Woof!", I offered, and looking back I regret only not having thought to wiggle my ass in my chair to signify enthusiastic tail wagging.

When it became clear that the import of my barking had eluded those present and had at least stunned Canada into silence I expanded on it by observing that I knew neither the sending application nor the application I was meant to replicate nor the entire subject of accounting and concluded by wondering aloud if the guy outside tending the falafel cart might not be a better fit since he at least knew how to make change.

The Canadians had heretofore been utterly thrilled at the idea of having their accounting taken over by our central gang that could not shoot straight (I'll wait while those of you who missed the irony go back for a look) and now they had White Fang as the contractor assigned to the task, confirming their worst nightmares.

They agreed to provide a thorough specification and the AVP and I ever after when meeting in the hallway would bark at each other in greeting and at his going away dinner -- well, after the second round of shots it was pretty much non-stop howling, thank god for private rooms, we should have had one. I digress.

Remember: this is about users not being able to change requirements discontinuously. So now we know how I ended up with the most beautiful requirements from which I have ever worked. Requirements? Bah, this was pseudo code. Just brilliant. They identified a dozen kind of transactions/situations/circumstances I could expect from Isis and they specified exactly how each should be handled.

And this "exactly" was not just exact. There was this tremendous regularity in how they specified the handing of each case. Each was expressed cleanly as so many debit this, credit that, debit this subtransactions. These subtransactions just begged to be viewed as microcode for the larger instructions specifying how each case was to be handled.

So that is the code I wrote, after effusively praising Team Canada to themselves and anyone else I could find. The high-level code read Isis transactions, identified the high-level case, and invoked its handler, each handler dispatching one or more microcode subtask handlers. We are talking easy work and soon we sail into user acceptance done thoroughly for a couple of months because I mentioned they did not trust us. I also know it was a long time because by the time the call came through with the title of this entry I had forgotten about the entire issue to which they referred, which was this:

After a week my bugs had been rooted out and IRs were being handled by letting them know how they had screwed up the test. Frank was happy, I was snoring, life was good, we had time to buzz about this new write once run anywhere language called Java that would let browsers run applets.

Then well into the test when the IRs had slowed to a drip I got a call from Canada. I had blown a transaction. Lemme see it... OK, sorry, that is what the spec says. No, these have to be handled this way, the debit goes over here and credit goes over there when X is Y.

Now as usually happens in these deals I still did not know anything about accounting but some of the words had taken on glimmers of meaning and I asked, But isn't this the same as whatever? And then I threw around some terms and said that from my poor understanding what they were suggesting seemed wrong. Nope, that is how we do it. OK. Bye. Bye.

And now comes the good part. I invoked Tilton's Lemma -- not because Kenny could not see the legitimacy of their accounting, though that encouraged me, but because of something else I found to be strangely compelling: my code would not do what they wanted.

I tried to see how I could patch in the exception and there was no way. I would have needed action at a distance and Einstein had already spoken on that. What was needed was for the microcode to behave differently based on information (X and Y) available only at the top level in which the higher level transactions were being identified and their handlers invoked. The only way to give Canada what they wanted was either to add a parameter to everything in the call chain (which I would have called woof-woof, I think) or even more elegantly to set a global variable at the higher level that this one microcode handler could watch out for. Ewwww.

Now please do not think I am posing as a person of integrity or principle or anything because I am the most miserably pragmatic soul you can hope to meet, but I got back on the phone to Canada.

"Marcia, Kenny. My code does not want to do what you want, eh? Your spec is pretty clear on all these things and just from the sound of it it seems the way I am handling it now is right anyway. I can kludge things up to get what you want, but I will hate myself in the morning."

Marcia said she would talk to Bernie, her VP, and get back to me. She did not do so for so long that I forgot about it and to her credit she did not even have to make the call but she did.

"Kenny, Marcia. I spoke to Bernie. The way you are handling that transaction will be OK. We can live with the way you are doing it."

Tilton's Lemma lived on. Somehow clean design (starting I freely acknowledge with a great spec) had flushed out a faulty business practice. The users had indeed offered an RFE discontinuous with all their other stated requirements, but it turned out to be such an egregious error that they agreed to fix it. (To their credit again!)

Why does this war story give me goose bumps? Yes, I need to get out more. And OK, maybe it is not such a big deal. The careful architecture forced on us by having to program these godawful machines makes it easy for programmers to see inconsistencies the business experts miss.

I just get a kick out of it -- a sense of God or Plato or the Tao throwing their weight around as long as we take design seriously and hop on the phone over such things.

TLOP: The Worst Thing You Can Say About Software Is That It Works

2008-03-22T12:11:00.006-04:00

I walked out of my cubicle, turned left, went up one maze turn, turned into the next cubicle. This was clearly a while ago, before email and even before I had reached the point where I would phone a guy ten feet away in case they were not there so I could leave voice mail.

I wanted to tell Mike that I had enhanced the julian date function he had shared with me so he could replace his with mine instead of the system growing a multiplicity of julian date functions.

Mike was another contractor on some godforsakenly dreary application being developed at cruising altitude in some megabank, a great guy always joking well and laughing about everything including his obsession with the real estate scheme infomercials he apparently watched non-stop in his free time.

I told Mike I had snazzed his function up nicely, generalizing it or adding some whiz-bang feature or something, and mentioned he could now modify his program to use the new version.

Something I need to get across to My Cherished Reader is the utter and profound honesty of Mike's reaction. Indeed, sufficient googling may well surface Mike's blog entry detailing in turn his astonishment and horror at what I had proposed. For when I finished there was a brief moment of incomprehension and then a dawning and his eyes grew wide and his jaw dropped and then a huge smile of wonder spread across his face, his palms rose in supplication for any possible justification of my madness and at last words came to him.

"But...but...," he stammered for effect. "My code already works!"

It was a grand moment, two solid technologists sincere in their craft standing face to face with not one scintilla of common ground on the simplest of questions.

Mike was reeling at the idea that a working module should be touched. He repeated his encomium ("It works!") and then with no offense intended and with solid good-feeling camaraderie just started laughing at me.

Meanwhile I knew without doubt that swapping in my version would likely take one minute (what's that? about sixty seconds?) and that application systems should be kept as simple as possible (and so not have two julian date functions (can you say Y2K?)).

I shrugged -- it was a small matter -- and retraced my path back to my cubicle before the pheromone trail could grow cold. Well, this is my blog, so Mike was wrong. Tilton's Law:

The worst thing you can say about software is that it works.

I think I can prove that with geometry: if a pile of code does not work it is not software, we'll talk about its merit when it works, OK? Therefore to say software works is to say nothing. Therefore anything substantive one can say about software is better than to say it works. QED. (That last bit is the geometry.)

So what else besides that it works can you say about software, especially about applications written in tall buildings? Well, unless one is banking on the relatively high probability that the project will get canceled before ever it sees light of day, the software will eventually be asked to do new things or to do old things differently.

So the software must not be brittle and must not have any more built-in assumptions than necessary and where one builds in the assumptions it would be nice if they were abstracted out of the core so that new requirements can be handled as gracefully as possible.

Note that new and changed requirements come along not just after a successful deployment but also during development, perhaps even more so because only when users see software do they realize what they wanted. They should do a study: I bet requirements change more just implementing the software than they do ever after. Brittle, inflexible software can be so hard to change during development that projects never make it out the door no matter how well the software "works" on the original requirements.

Reminds me of another war story. Schedule "B".

The law on tax shelters had just been changed eliminating the bogus write-offs those things produced and some financial establishment needed some contractor to write the software to generate statements for clients detailing exactly how badly they would be hosed by the new law. Apparently they had tried it in-house and the guy assigned to it had quit after working on it for a couple of months to no avail. Jobs were everywhere back then in the Eighties, why not?

God forbid I should digress: it was a fun interview, really nice group and nice manager. The best point was when he asked His Kennyness what was His greatest weakness. Neat question!

Lemme see...no...no...no... I was looking at the floor going through coding, debugging, DB design, and it was taking a while which I thought was pretty funny and I am sure they were all enjoying it but I really wanted to come up with something and then it came to me.

"Testing!" I almost yelled. "I am terrible at testing!"

Whew! Reminds me of the time I had a little team (this was before people realized I should just be chained to an out-of-the-way workstation and hosed down and thrown food as needed on big projects) and was assigned a junior nooby who asked me the same question about three times and who hated me because apparently Hell hath no fury like a woman gently suggested she might take notes, which was great (that she hated me) because she was so useless we just had her testing my code and believe me no software has ever been subjected to such merciless, sustained, and deliberate abuse as was mine. Every twenty or thirty minutes we would hear this triumphant rejoicing "Ha!" explode from her cubicle as she found another bug and imagined another dagger plunged between my ribs when in fact I was loving the whole thing because to me testing is just God and Abraham and Isaac all over again. Where was I? Schedule "B".

Anyway, the task went very well. The manager (another Mike) gave me a very clear spec and I did my usual structured architectural blah-blah-blah and the users were very nice and called me every once and a while with RFEs. Bringing me to my point.

Mike was a great guy but also button-downed and into process and told me not to listen to the users, make them ask him and he would tell them, No. So I was always very nice with the users and said I welcomed their thoughtful consideration and input and could barely restrain myself from just dashing it off right now if I could just put them on hold for a second and I will indeed start looking at the matter while it would help very much if they could just run this great idea past Mike, he's a bit of a stickler for protocol, terrible bother I know, hate to ask this of you.

OK, they would say.

Then I would put down the phone and look at the code to see what was needed. Oh. Scratch, scratch, poke. Done.

Then I would go into Mike's office and tell him the users had asked for X and he would make a face and I would say I had already made the change if that would affect his decision.

Mind you this did not happen because I consciously planned for change, I just worried about things besides whether the code worked. Simple architectural tidiness went straight to the business bottom line: the users got exactly what they needed at no extra cost to the enterprise. How does simple clean design make RFEs easy to handle? Tilton's Lemma:

Refinements to requirements cannot vary fundamentally from the original.

It is possible for users to say, Oh my, you have the runners running the bases clockwise? We wanted them going counterclockwise. It is not possible for the users to come back and say, We wanted them going first to second base, then over to first, then to third, and then home. Some fundamental force of nature constrains how we can screw up and keeps it within a metalevel such that refinements to requirements will never break code all that badly.

[Keep your eyes open for the "We Can Live With the Way You Handled That" mind-bending illustration of the power of clean design.]

The flipside war story was in a different tall building working on a system so badly designed that after a certain point it pretty much stopped evolving at all.

I walked up to my manager and said I had an idea on how to improve something I was working on for her so that it could be used for other things.

"No, no, no," she said. "I just need this one thing working before Thursday's meeting so I can score one on the users. They are out to get us."

Ah, systems development as an act of war. The system they had developed was such a horror that the only thing that ever moved it forward was a team of very well-paid outside contractors who managed against all odds to add new functionality, pausing from their toils every several days when one of them would take a design two-by-four across his skull from said system and lean back and poll the others on how badly each felt the system hindered them, an order of magnitude generally being settled on as no exaggeration however often it usually is.

The IT department ended up with a bunker mentality lobbing back hand grenade code releases to suppress fire from users strafing us with gratuitous RFEs as the organization staggered towards collapse and acquisition, and it being a financial institution to no small degree dependent on how well it processed information I cannot help thinking this system was to blame.

But it did work.

Oh, sorry. Lisp? Well, the more powerful the language the more time one has for that much easier a job of designing an application as opposed to banging out whatever source code passes the unit tests. And when it comes to separating specifics out into a configuration area that drives a more universal engine... can you say DSL? Sher ya can.