A Bear's Life

News from the Mobile Front

2009-08-30T23:56:00.001-06:00

Someone recently asked me where I lived. This is a interesting question, since for the last year I have not really had a permanent residence. I pretty much live out 5 suitcases and a plastic Tupperware tub. After this gig is done in Portland I plan on dropping that to 4 suitcases and a laptop bag.

Since I live alone and travel alone I can travel pretty light. Currently I have a extended stay hotel in Portland that I live in and rent by the month. My current contract ends in May of 2010. At the time I will decide if I am staying in Portland or moving on.

My work is now pretty much contract work. Go in take a gig for 3 months to a year and move on. I spent 10 years in Boise putting down roots, and I have no desire to do that again, so I am planning on staying mobile. I also like the nature of contract work. I can be very technical, solve the problem and move on. No need for the politics of a company or the drama. I enjoy the technical challenges of developing software, not so much the politics.

As for my collections, when I lived in Boise I had a extensive CD, DVD and Book collection. Around 5000 hard bound books, over 1000 music cd’s and over 1000 movies. Now that I have pretty much gone mobile and all these collections are digital.

I currently have over 40,000 digital books. This includes books in EPUB and Kindle (mobi) format. Both public domain works and books I have purchased. As I mentioned before I really like http://oreilly.com/ for technical books since the have them in a variety of formats, but I also get a lot of technical books from APRESS as PDF’s and Amazon.com for my Kindle. I try not to buy any physical copies, since those are hard to carry around :-). Most of my older science fictions I get from Baen books. Other books not available in EPUB format I but from Amazon.com and have them on my Kindle DX. My current electronic book collection is a little over 15 gig in size.

I have over 12,000 tracks of mp3’s I buy from Amazon.com. Most of these are classic rock, but I admit I have some pretty weird albums in my collection :-). I have my album collection stored on my two laptops, my 512 gig passport and finally the main copy is on my IPOD classic (120 gig). My current music, audio book and podcast collection is about 100 gig in size.

Movies I get through either net flicks or purchase them from amazons video on demand service. My digital movie collection on Amazon.com is now about the same size as my old collection used to be. Most of these I leave on the Amazon servers. Only a few have I downloaded.

My software is stored as ISO images. All my software from Microsoft I get from my TechNet subscription and store it as ISO images. I then mount this using Magic Disc (a very cool utility that gives me a ton of virtual CD’s). If the software comes on a CD then where possible I take a CD and copy it to an ISO image and then mount the ISO image. I store the CD’s in my CD case. These are my biggest headache currently. I am trying to get all my software digitally so I can get rid of the CD’s. Sadly the biggest issue is the software CD’s for portability. Sigh…

Email and document storage is all courtesy of Google apps. I have several gigs of email on Gmail and am quite happy with it. Everything else is stored on my laptops and my passport. This combination makes it very easy to have all my primary files available digitally. Digital pictures I keep on a a 16gig SD card and then stored on the passport and the IPOD.

So there you have it. Over a year and most of my life is nothing but bits stored on local drives or in the cloud. Kind of scary when I put it that way :-).

Sony Ebook Reader or a book shelf of half a million plus…

2009-07-12T22:58:00.003-06:00

I recently picked up a new gig in Portland at just about the same time as my Kindle died. With the death of my Kindle I was looking for a new ebook reader. The kindle2 was out but I really liked the specs on the KindleDX. Problem was the DX was not going to be out for a while. To aggravate the situation I now have a 45 minute commute into town on the train.

Now don’t get me wrong, the train is a great way to go. No traffic and I get some exercise walking back and forth to the train station (2 miles a day). The problem is a 45 minute ride leaves me bored to tears. My lifestyle is such that I no longer want to collect a ton of hardcover or paper back books. I have over 5000 of these in storage in Idaho and don’t need anymore. The kindle worked great for me because all my books were digital and I could carry them around in my hand.

I really wanted the DX, but those are all on back order and anyone who knows me knows I hate waiting for toys. I decided to get a Sony PR700 series ebook reader for the short term. I will still probably get a DX so I can read the Wall Street Journal on the way to work and because it is easier to read my PDF’s on.

I have started to collect a lot of ebooks and wanted to switch to a standard format that I keep my main library in. I have standardized on the epub format because it is a open standard that has no DRM. This means I can easily play with books in that format using .NET since an epub is simply a zip file containg a standard set of files a content in html. These files include:

A xml file containing the bibliographic data for the publication (title, author, publisher, etc…) in Dublin Core format
A xml file containing the table of contents
The contents of the publication in a series of html files

Gutenberg

OReilly is publishing their books in EPub, PDF and mobi (Kindle) format. So if you want to read Programming WCF Services on your Kindle, Sony or even IPhone it is easy enough to do.

Finally the relevance to the title above. You can download any of Google’s Library of EPUB books easily without owning a Sony reader (say you want to read them with Stanza on your IPHONE). To do this:

Download the Sony Reader from http://ebookstore.sony.com/google-ebooks/
Register with the site
Click on the Unearth a Classic link on the far right
Search Google books for books of interest
Download those you want

You may also get free science fictions in EPUB/Stanza format from Baen books, and there are other source all over the net. Almost all of Project Gutenberg is now available in EPUB format. I will be posting a mass downloading utility for Gutenburg in a later post that allows you to download all 38,000 plus epub books from project Gutenburg.

So there you go. EBooks have more then arrived, they have over run the competition :-). Believe me my 40K+ ebook collection is much more portable then my 5000 book collection was (and is easier to search).

The only drawback I have discovered so far is no EBook reader on the market today can handle the collection I have acquired. So like any good developer I am writing my own. More to come on that later.

Darrel

LINDUG: The LinkedIn .NET Users group

2009-05-16T15:19:00.001-06:00

If you do not make your regular user group meetings, or don’t have a user group in your area, you may want to join the Linked .NET Users Group on www.linkedin.com. It is a virtual .NET Users group of 24,000+ members. They sponsor webinars and have some decent discussions.

Some upcoming webinars include

Scott Guthrie talks shop with developers, May 27, 11:30 AM PDT - http://events.linkedin.com/LIDNUG-ScottGu-talks-shop-developers/pub/60571
Sara Ford talks about Open Source with CodePlex, June 21, 9:00 PM PDT - http://events.linkedin.com/LIDNUG-Visual-Studio-Tips-Tricks-by-Sara/pub/65637

For that matter you can always join the Elegant Code group on linked in. If nothing else it keeps David out of trouble :-).

C Style operators in SQL Server 2008

2009-04-17T08:51:00.001-06:00

I started off my programming career developing C applications. This was primarily because I did my work on Mini computers running UNIX. Almost unheard of at the time :-). For the first 14 years from 1982 to 1996 I developed my applications in C. During that same time frame SQL came along for relational databases. Now I had SQL, C and Embedded SQL for C. Throw in curses for developing screen applications and what more could a developer want? :-).

In late 1996 I began developing applications in Visual Basic 4, and found events. Very cool. But I still missed some of the features I had grown to love in C.

Recently I noticed the section on compound operators in SQL Server 2008. This is interesting since these are the first new operators added to SQL Server in at least the last 8 years. SQL server 2005 and SQL server 2000 both had the same list of operators. Now Microsoft has added some new ones. In addition adding the BackSlash operator is just a nice little bonus (now if I could just get my ternary operator (?:)).

I guess a good thing about starting my career developing C has been that all the new languages (including SQL server) are going back and picking up some of the concepts from C. K&R should be proud :-).

New expression operations in SQL Server 2008 Tranasact-SQL

+= (Add EQUALS) (Transact-SQL)

-= (Subtract EQUALS) (Transact-SQL)

*= (Multiply EQUALS) (Transact-SQL)

/= (Divide EQUALS) (Transact-SQL)

\ (Backslash) (Transact-SQL)

%= (Modulo EQUALS) (Transact-SQL)

&= (Bitwise AND EQUALS) (Transact-SQL)

|= (Bitwise OR EQUALS) (Transact-SQL)

^= (Bitwise Exclusive OR EQUALS) (Transact-SQL)

~ (Bitwise NOT) (Transact-SQL)

Some Themes and a Little Crimson and Gray

2009-04-02T20:42:00.001-06:00

I have been working on themes for WPF and Silverlight applications. I found this one for SilverLight and WPF. It is called shiny red. Gotta love that. Nice bit of crimson and gray.

As some of you know I am a bit of a WSU fan, so I love this theme. They also have a couple of purple ones. Needless to say I deleted those. Pure evil they were… :-).

On Further Review…

2009-03-19T01:32:00.001-06:00

Never mind. Microsoft has a CTP of the 2.0 version of the Open XML SDK that does this much better. Check out http://www.microsoft.com/downloads/thankyou.aspx?familyId=c6e744e5-36e9-45f5-8d8c-331df206e0d0&displayLang=en

Reading a Office 2007 docx file using C# and SharpZipLib

2009-03-18T21:27:00.001-06:00

I found myself needing to read a office 2007 docx file to get the xml of the document., so I went and got the ECMA spec for OfficeOpenXML. That is a heavy 1500 page heavy read :-). So I decided to keep it handy and just write some code to see how bad it would be. So far it has not been as bad as I thought it would be so I thought I would post some code on it. It has been a while since I have posted anything, so here goes.

The docx file is actually just a zip archive of a bunch of files. The trick here is to read the zip and pull out the content. I decided I wanted to load the data to a class hierarchy so I could do some Linq to objects work with it.

So I will start this with some simple code. First the DocumentPart.cs class holds the document content and some information from it’s zip file. The document content is stored as a linq to XML XDocument..

    1 using System;    2 using System.Xml.Linq;    3     4 namespace OfficeOpenXML.Package    5 {    6     /// <summary>    7     /// A document part in the OfficeOpenXML Package    8     /// </summary>    9     public class DocumentPart   10     {   11         public string Name {get; set; }   12         public string Comment { get; set; }   13         public long CompressedSize { get; set; }   14         public DateTime EntryDate { get; set; }   15         public long Size { get; set; }   16         public XDocument Content { get; set; }   17     }   18 }

It is pretty basic so far (love those automatic properties).

Now how do you read the zip file?  I use the SharpZipLib to read the zip file and then store each member of the zip archive into a dictionary of DocumentPart with the filename as the Dictionary key. 
 
    1 using System;    2 using System.Collections.Generic;    3 using System.IO;    4 using System.Text;    5 using System.Xml.Linq;    6 using ICSharpCode.SharpZipLib.Zip;    7     8 namespace OfficeOpenXML.Package    9 {   10   public class Parts   11   {   12     public Dictionary<string, DocumentPart> DocumentParts { get; set; }   13     public string FilePath { get; set; }   14    15     public void OpenPackage(string filePath)   16     {   17         ZipEntry Entry;   18         XDocument contents;   19         StringBuilder XMLDocument;   20         byte[] Buffer = new Byte[8192];   21         int bytesRead;   22    23         using (ZipInputStream Package =   24           new ZipInputStream(new StreamReader(filePath).BaseStream))   25         {   26           while ((Entry = Package.GetNextEntry()) != null)   27           {   28             XMLDocument = new StringBuilder();   29             while ((bytesRead = Package.Read(Buffer, 0, Buffer.Length)) != 0)   30             {   31               XMLDocument.Append(   32                 ASCIIEncoding.ASCII.GetString(Buffer, 0, bytesRead));   33             }   34             contents = XDocument.Parse(XMLDocument.ToString());   35             DocumentParts.Add(Entry.Name, new DocumentPart()    36               { Name=Entry.Name, Size=Entry.Size,    37                 Comment=Entry.Comment,    38                 CompressedSize=Entry.CompressedSize,    39                 EntryDate=Entry.DateTime, Content=contents });   40           }   41         }   42     }   43    44     /// <summary>   45     /// Construct the class   46     /// </summary>   47     /// <param name="filePath">The office Open XML document</param>   48     public Parts(string filePath)   49     {   50         FilePath = FilePath;   51         DocumentParts = new Dictionary<string, DocumentPart>();   52     }   53   }   54 }

I use the using statement (on line 23) to handle the opening the docx file and then process the zip archive using the GetNextEntry.  The inner while loop (at line 28) reads the content of the entry into a string.  Finally while the DocumentParts.Add() adds the document dictionary.  

A simple NUnit test (not exhaustive by any means) is: 

    1 using NUnit.Framework;    2     3 namespace OfficeOpenXML.UnitTests.Package    4 {    5     [TestFixture]    6     public class PartTests    7     {    8         [Test]    9         public void OpenPackageTest()   10         {   11             OfficeOpenXML.Package.Parts p = new OfficeOpenXML.Package.Parts();   12             p.OpenPackage("../../TestData/AbilitiesandConditions.docx");   13             Assert.IsTrue(p.DocumentParts.ContainsKey("[Content_Types].xml"), "No Content_Types?");   14         }   15     }   16 }

In my next post I will show how to use the information from the [Content_Types].xml entry and the docsprops/app.xml and docprops/core.xml files to create an object that has information about the document being read (using some Linq to XML to populate the Class).   

Darrel

Red, Green, Re-factor

2009-03-11T19:30:00.001-06:00

I find my self getting more and more into the test driven development paradigm. I am working on some fairly heavy OO code with a great requirement specification (it is actually and ISO standard).

The application is growing pretty much via the unit tests. Write the test, then make sure the code fits. It is amazing how many times I am not sure how a piece of code is going to work so I just put together the unit test to fit the standard and then code the class. I can then re-factor as needed.

I am using the speech API’s to drive the interface so the speech api simply outputs a string of text that I can use to execute the program commands. I wanted the program to be able to be both driven by speech commands and through the standard GUI approach. Since I am working on a compiler, interpreter I have simply added the speech grammar as another grammar in the compiler. .I will probably do the same to parse the command line options.

Makes for some interesting code :-).

Not posting much lately

2009-03-10T16:31:00.001-06:00

I have been working on several projects. They are starting to come to completion and I hope to post some more on them at the beginning of April. I have also been busy with contracts and life in general, but I will get back to some posts in a few weeks.

Human Costs of a Bad Economy

2008-10-13T22:20:00.001-06:00

I have been traveling around the country for the past 90 days through the Midwest, the east coast and the south and staying in a lot of hotels. Most recently my hotel in Daytona Beach.

Now my hotel is right on the ocean. In fact my balcony faces the ocean and the pool. All in all not a bad place that I got for $32.50 a night. Now Daytona Beach is a resort town. All most all of the business here comes from the tourists. With the state of the economy there are virtually no tourists around, the hotels and restaurants are getting almost no business. Ditto for the tourist shops. You can almost watch them close all up and down Atlantic Boulevard.

As a geek I make pretty good money. As a matter of fact I probably make more then any of the cooks, house keepers or shopkeepers around here. it is sad when you hear some have cut back on hours to less then 20 hours a week. I mean they already don't make that much and now they make half of that and most of them are not that well off.

You can only hope it gets better soon for all of us.

Sunny Florida

2008-10-03T13:05:00.001-06:00

Well, I am the second elgant coder to be outside of the Boise, Id area. I have moved to Florida and am currently camped in a hotel on Ormond Beach.

Although I enjoyed my 10 years in Boise it was time to move on. After being riffed off from Micron I was pretty much at ends as to what was keeping me in Boise. I had originally moved there in 1997 to be around my mom. After she died in 2003 there was little keeping me in Boise except for momentum. I had no real reason to change. Micron's layoffs gave me a good reason to think about moving on.

I have always wanted to see a shuttle launch and wanted to be around a ocean again. So with those two criteria Florida was really the only choice. I spent two months driving across America seeing family I had not caught up with in years. In addition I spent some time just seeing the country. Something I have wanted to do for years. I traveled 9000 miles in 60 days and went from one end of the country to the other. A great trip for me and the cat (although I am not sure the cat would agree :-)). I will be posting some blog entries about the trip over the next few weeks.

For now I am planning on staying on Florida for a while. I will start looking for full time work next week. I am excited about the change and looking forward to some new challenges.

The Tablet PC

2008-07-12T04:27:00.001-06:00

I recently took a tumble (anyone who knows me will tell you I am clumsy) and hurt my right arm. Needless to say this is not good if you and a computer geek. Fortunately at the same time I was trading out my Dell 1720 for a new dell lattitude XT. So I am actually writing this blog entry. Literally! I am using the handwriting recognition features of the tablet PC.

So far I am very impressed. The tablet does a better job reading my handwriting they some of my friends do (hi joe!). It is interesting that it reads my writing better then my printing. For some reason all those years I actually thought my printing was easier to read then my writing. Guess I was wrong. I am pretty sure I actually can write faster than I can type. This as is kind of sad knowing ] have made my living writing computer programs for the last 30 years.

I will have to bring up my visual studio 2008 environment and see how easy it is to code this way. l should point out I am doing this all on Vista Ultimate. I did not like Tablet XP any where as much as Vista.

Steve and Bill On-Stage

2008-06-28T23:57:00.001-06:00

In case you have not seen any of these videos of Bill Gates and Steve Jobs on Stage talking about our industry past, present and future. A fascinating discussion.

Paraphrasing a classic comment from Gates. We want a computer in every home. We never thought about the fact that we would have to be a BIG company to do that...

Gates is the Rockefeller of our times

2008-06-23T23:55:00.001-06:00

Interesting article on MSN on Bill Gates and how his foundation is changing philanthropy today. He will give away more money then Rockefeller did at the beginning of the century. How will this change education and medicine in the next few years? For example the Gates foundation is:

"..a foundation whose combined assets will one day exceed the budgets of all but 30 percent of the countries in the world"

Talk about having to learn how to scale :-). I it will be interesting to see how he takes on this challenge.

Am I asking for too much?

2008-04-15T04:56:00.005-06:00

I am currently participating in a interesting thread on Matt Berther’s bog on my podcast. I commented on his post Blasting open source because I felt my podcast was not meant to pick on up source projects although part of it could be interpreted that way. So since then what I have been explaining is why I don’t see any innovation happening in software today.

Although I like .NET and Java I think there has been no real innovation in software in the last 30 years. The primary things I see as the backbones of what we use computers for are:

Databases - CODASYL defined the standard for network databases in 1969 (RDMBS were first described by Codd in the 1960’s and 70’s at IBM)
Word Processing - The Unix Concepts of text formatting for publishing came around in the 1970’s (as a matter of fact UNIX was first designed as a text editing and text formatting system). Although UNIX text processing is much more akin to HTML then WYSIWYG editors like MS Word
Spreadsheets - First patented in 1971, VisiCalc was actually being distributed for the Apple II in 1979
General Programming via high level Languages -
a) Interpreted Languages (If you want the full deal Smalltalk was around in the 1970’s. Object oriented, dynamically typed and reflective)
b) Compiled Languages (Cobol was created in 1959 and C in 1972)
WIMP Interfaces - (Pioneered by the Xerox Alto in 1973)
The Internet which started in 1969.

So if you take as a baseline of 1980 we have had these systems around for at least 28 years. Most have been around longer. Sadly I have been programming for 30 years so I have played with some of these at one time or another in my career.

Do I expect too much? I can’t help but see the feature set of these products though and say that on the whole the pace of innovation in computers disappoints me. Those advances we do see to have picked up come about more because of hardware advances then software advances. As Brooks says in No Silver Bullet

…the anomaly is not that software progress is so slow, but that computer hardware progress is so fast.

As I see it we continue to attack the accidents of software engineering. I quote from Brooks

If we examine the three steps in software technology development that have been most fruitful in the past, we discover that each attacked a different major difficulty in building software, but that those difficulties have been accidental, not essential, difficulties.

The three that he describes include:

• High Level Languages. Now this would include java and C#.
• Time Sharing.
Now this would be the entire concept or personal computers, laptops, PDA’s etc…
essentially ways to get the answer immediately.
• Unified frameworks. Here
he talks about UNIX, but you can also include in this the .NET framework and the
Java class libraries.

So my contention is that one of the reasons software innovation is so slow is we spend too much time reinventing what we have had since the 1970’s and not focusing on advancing the art of computer science. Am I wrong? So many of the things we see in the Open Source and Commercial World are clones of each other. Are we as developers taking the simple way out? Sure we can design a new enhanced WIMP interface, but that is because we already know most of the requirements except for that little tweak we are going to make? It is fun to do these kinds of projects because we don’t have to think about the hard stuff like requirements analysis, test cases and design. We just set down and code another build system, or IOC Container or OS.

We continue to attack the accidents because the essentials are so difficult. I was doing some reading on the System.Speech namespace in .NET and Natural Language Processing in general. One of the big things required to get these to work is to put together grammars for what we want the computer to do once it has recognized the words. The problem is almost no one is building the grammars. Why is that? Because it is hard. It would require us to step outside of the fun and simple things we do and take on some difficult tasks. The same thing could be said for coming up with systems that use common component architecture. How much simple would things be if we started building components and plugging them into applications. As David said in my Podcast, I want to drag and drop a CRM solution.

What does it take to get to these? We have to give up on some of the fun things we do and start attacking the essential problems in software development. Am I expecting too much? Are we as Software Engineers unable to tackle our discipline as a engineering science and start innovating again?

** Portions of this entry were posted a a comment on Matt’s blog ***

Founding Brothers

2008-04-14T18:00:00.001-06:00

A friend of mine, Chuck Steffens, has convinced me to start reading Founding Brothers: The Revolutionary Generation by Joseph J. Ellis. I must say this is a fascinating look at American history. Specifically the time of the American Revolution and the founding of the republic. I will do a book review in the next few days, but so far (up to chapter 3) it is a great read.

I downloaded it to my Kindle last night at midnight and started reading it. The only thing I can say about the kindle is I am glad they don't have a ton of Computer Books available on it yet. I would go broke every time I went to amazons web site. :-). The science fiction and historical stuff us costing me a fortune. On the plus side I have finally started to read Winston Churchill's series on World War II after having owned the books for years.

Google App Engine

2008-04-08T01:13:00.001-06:00

I just ran into a reference for this. It looks pretty cool. Develop your application offline and then deploy it to the Google servers. The man page is at http://code.google.com/appengine/.

I have been meaning to play with Python and Iron Python. I have just never had a reason beyond curiosity. Now it looks like it might be worth learning.

Mono 1.9 is Released

2008-04-03T03:28:00.001-06:00

Mono 1.9 is released. This is the last version before 2.0 which supports the entire 2.0 version of .NET on Linux. This version also supports some C# 3.0 features including Linq to XML and Linq to Objects. Very Cool
http://www.go-mono.com/mono-downloads/download.html

They have also released the Mono Develop IDE. Full support in an IDE very similar to Visual Studio.
http://www.monodevelop.com/Main_Page

Checkout the screen capture with their integration with NUnit

http://www.monodevelop.com/Image:Nunitmain.png

It is all about Portability...

2008-04-01T09:01:00.001-06:00

As cell phones get more and more powerful, you wonder how soon they replace laptops. Primary thing missing are good keyboards, mice, decent size monitors, better performance and memory. The Windows CE development environment is there already. So let's take a look at these one at a time...

Now I can get a 8 gig memory card from SanDisk for about $50. This kind of makes me wonder. How much document memory etc do you need?
Now for keyboards I can get any number of virtual keyboards although the support looks a bit shaky.
For Mice can I use any standard Bluetooth mouse?
Now here comes a projection system. Is this a possible replacement for the monitor?

Mind you, none of these are that great now, but with cell phone providers adding more and more content to the phone and revenue from voice communications stabilizing you will probably see cell phone providers going after any source of revenue they can find. Of course I don't think I will see Visual Studio there anytime soon, but who knows?

Accidental Properties of Software: Programmer Productivity

2008-03-30T01:54:00.001-06:00

While researching for these articles I ran across this blog entry describing something the author refers to as Yannis's Law: Programmer Productivity Doubles Every 6 Years. The whole premise of the article is to dispute Brooks statement that there will be no order of magnitude improvement in the essential properties of software development. The author goes on to prove that programmers are seeing a 11x improvement in software development since 1986. There are two places this assessment is wrong. First Brooks was looking for an order of Magnitude improvement in 10 years (by 1996) not 21 years (2007). Second this is an attack on the accidental property of software (it's construction), not the essence of software, the mental crafting of the conceptual construct.

I do not dispute there are a great many cool tools and languages out there to increase programmer productivity, but we should all have seen the breakdown of the Software Development Life Cycle by now. How much of the SDLC is related to software construction? If we say it is 30% (which is probably a bit high) we are still looking at over 70% of the project time that is not being addressed by this improvement. As we reduce the construction time to zero (which would be magic :-)), we are still only attacking the accidental properties of software. What are we going to do about the other 70%? We have still not addressed the essential property.

Attacks on the Accidental and Essential Properties of Software

2008-03-29T16:08:00.001-06:00

Over the next several months I will be posting articles that discuss various aspects of software and the state of the industry. These will be titled as either attacks on the essential properties of software or the accidental properties of software as defined by Brooks in my post I first learned about Agile Development….

Brooks went on to provide a clearer definition of Accidental and Essential Properties in No Silver Bullet Refired (Chapter 17 of the Mythical Man Month). In particular Brooks uses Dorothy Sayers definition of creativity:

instead I follow the English dramatist, detective story writer, and theologian Dorothy Sayers in seeing all creative activity to consist of (1) the formulation of the conceptual constructs, (2) implementation in real media, and (3) interactivity with users in real uses.[6] The part of software building I called essence is the mental crafting of the conceptual construct; the part I called accident is its implementation process.

The goal is to try to discuss the progress towards a silver bullet for software development. These are my thoughts on progress in the methodologies and tools to make developing software easier. These are things we are doing as an industry to attack the Essential properties of Software. There are also that are just white noise interesting, fun and useful, but in the end just attacks on the accidental properties of software and have no real impact on our ability to quickly create quality software that attacks the essence of building software, or as Brooks says:

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.

If this is true, building software will always be hard. There is inherently no silver bullet.

So lets look around for what we have done in the last twenty years to attack the essential properties of software and progress towards the silver bullet of software (from the essay No Silver Bullet written in 1986).

Taking the Library Digital...

2008-03-17T07:12:00.002-06:00

I have been collecting books for over 40 years. So now I have about 400 Technical books. As for the Fiction I have almost 5000 volumes of those. Ran out of wall space in the house and the garage last year.

One of the reasons I picked up my kindle was to reduce the amount of paper I was throwing away. I went to getting a online subscription of the Wall Street Journal. Downloads everyday and no paper to throw away. I also have started picking up the classics off project gutenburg, new ebooks from baen and fictionwise. Up to 400 or so books on the kindle.

I am currently reading some fiction,
· Fatal Revenant by Stephen Donaldson
· Myth-Gotten Gains (Myth Adventures) by Robert Asprin and Jody Lynn Nye

I have started buying my technical books as pdf's so I am reading
· Pro LINQ: Language Integrated Query in C# 2008
· Client-Side Reporting with Visual Studio in C#
· Pro C# 2008 and the .NET 3.5 Platform, Fourth Edition

I also switched to getting my Music (over 10,000 tracks) as MP3 downloads from amazon and My dvd's through Amazon Unbox to save space. The only disadvantage I see so far to going digital on this stuff is I am using over 100GB for Music, 450GB for ~700 movies and TV shows and ~5GB for electronic books. I am trying to find out if I can put a 8GB Sd card my kindle now and I think I will try to pickup a MS Home Server from HP to supplement my 1TB World books.

Now I just need something like this http://www.engadget.com/2008/03/05/buffalos-linkstation-mini-packs-1tb-into-entirely-too-small-an/ to carry around some of my digital collection with my laptop.

I first learned about Agile Development...

2008-03-15T01:28:00.001-06:00

From the essay No Silver Bullet written in 1986. Many of the principles described in this essay apply to the last 60+ years of Software development from the days of the ENIAC. For example the concepts for constructing software are based on a comment from 1958!

Warning, What follows is a fairly long discussion of things I have learned while developing software for the last 30 years. I started writing software when I was a senior in high school in 1978. I am still writing software and designing systems today. Most of this blog entry is based on an essay written by Fredrick P. Brooks in 1986. He also wrote the The Mythical Month in 1975 and it is still a great read. Believe me most of what Brooks says in the Mythical Man Month applied to my first 8 years of programming as much as it applies today. Brooks original Mythical Man Month is based on developing an OS at IBM in the 1960's but ignore that and look at the principals. There have been other authors who have expanded on what Brook's had to say in the original 15 chapters of this book but Brooks said it first. Why do you think the primary law of project management is called Brooks's Law?

Two additional essays from Brooks, No Silver Bullet and Silver Bullet Re-fired were added to the 25th Anniversary addition of the Mythical Man Month. No silver Bullet was written in 1986 (over 20 years ago and 8 years after I started writing software). This is the essay I want to talk about now. All quotes from No Silver Bullet are in Bold Itallics.,

...the anomaly is not that software progress is so slow, but that computer hardware progress is so fast.

Essential Properties of Software

The essence of a software entity is a construct of interlocking concepts,

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.

If this is true, building software will always be hard. There is inherently no silver bullet.

Software is also essentially complex. When a software architect models a system they must take this into account so that the Software Models do not obscure this essential property.

The complexity of software is an essential property, not an accidental one. Hence, descriptions of a software entity that abstract away its complexity often abstract away its essence. For three centuries, mathematics and the physical sciences made great strides by constructing simplified models of complex phenomena, deriving properties from the models, and verifying those properties by experiment. This paradigm worked because the complexities ignored in the models were not the essential properties of the phenomena. It does not work when the complexities are the essence.

So what are some of the essential properties of software?

First software must conform to business processes and institutions. This adds to the complexity of software because the requirements of conformity are based on Business processes that do not always make sense or are not as clearly defined as the principles of other disciplines (such as physics). So much of the complexity a software designer faces is a arbitrary complexity...

Much of the complexity that he must master is arbitrary complexity, forced without rhyme or reason by the many human institutions and systems to which his interfaces must conform.

...in all cases, much complexity comes from conformation to other interfaces; this complexity cannot be simplified out by any redesign of the software alone.

Many large Enterprise Systems are just now beginning to try to tackle this. Business Process reengineering targets the business process itself, not the software that implements the business process. Essentiality though this is just an attempt to do better requirements analysis and help to reduce the arbitrary complexity of software. This complexity is not addressed by re-factoring code. It is a essential property of the software.

Another essential property of successful software is that it changes:

All successful software gets changed. Two processes are at work. First, as a software product is found to be useful, people try it in new cases at the edge of or beyond the original domain. The pressures for extended function come chiefly from users who like the basic function and invent new uses for it.

Brooks goes onto describe another essential property of software. It is invisible

As soon as we attempt to diagram software structure, we find it to constitute not one, but several, general directed graphs superimposed one upon another. The several graphs may represent the flow of control, the flow of data, patterns of dependency, time sequence, name-space relationships.

Many Enterprise Architecture tools demonstrate this truth. In fact the entire premise of UML demonstrates this. We have Class Diagrams, Component Diagrams, Sequence diagrams, etc.. all designed to help us try to visualize software. These all succeed and fail to some degree because invisibility is a essential property of software.

Accidental Properties of Software

Brooks goes on to describe technology advances that attack the accidental part of software development

If we examine the three steps in software technology development that have been most fruitful in the past, we discover that each attacked a different major difficulty in building software, but that those difficulties have been accidental, not essential, difficulties.

The three that he describes include:

High Level Languages. Now this would include java and C#.
Time Sharing. Now this would be the entire concept or personal computers, laptops, PDA's etc... essentially ways to get the answer immediately.
Unified frameworks. Here he talks about UNIX, but you can also include in this the .NET framework and the Java class libraries.

I find this section really amusing

Operating systems, loudly decried in the 1960's for their memory and cycle costs, have proved to be an excellent form in which to use some of the MIPS and cheap memory bytes of the past hardware surge.

How often do you hear the cry that windows is bloated and overgrown but UNIX is lean and mean. This battle has been going on for over 45 years with no end in site. For all of you who continue in these debates know that the next 45 years will probably continue in the same way. We will simply substitute a different OS name and battle on...

Hopes for the Silver Bullet

This is a interesting section. See how many of these concepts seem familiar

High Level Languages advances. We have all blogged about the neat new features in C#. So 25 years ago it was the neat new features of Ada...

Ada not only reflects evolutionary improvements in language concepts, but indeed embodies features to encourage modern design and modularization. Perhaps the Ada philosophy is more of an advance than the Ada language, for it is the philosophy of modularization, of abstract data types, of hierarchical structuring.

Object Oriented Design. Have you seen an order of magnitude change by adopting C# or .Net in general?

An order-of-magnitude gain can be made by object-oriented programming only if the unnecessary type-specification underbrush still in our programming language is itself nine-tenths of the work involved in designing a program product. I doubt it.

Artificial Intelligence. It is interesting that Brooks breaks out this section into two separate sections. His comments on speech recognition and image recognition are very true today. One of the essential problems in speech recognition is not recording what you say, but understanding what you mean.

The techniques used for speech recognition seem to have little in common with those used for image recognition, and both are different from those used in expert systems. I have a hard time seeing how image recognition, for example, will make any appreciable difference in programming practice. The same problem is true of speech recognition. The hard thing about building software is deciding what one wants to say, not saying it. No facilitation of expression can give more than marginal gains.

Expert systems. The most advanced part of the artificial intelligence art, and the most widely applied, is the technology for building expert systems. The next section discusses why mentoring is so important for Architects and Senior Software Developers today since we have yet to build an expert system to do this for us.

The essential prerequisite for building an expert system is to have an expert. The most powerful contribution by expert systems will surely be to put at the service of the inexperienced programmer the experience and accumulated wisdom of the best programmers. This is no small contribution. The gap between the best software engineering practice and the average practice is very wide_perhaps wider than in any other engineering discipline.

Automatic Programming. What we could term as code generation today still has not achieved the goal of being able to state the problem and have the computer just solve it for you. Business Process Execution Languages are the latest attempt at this. But I still agree with Brooks when he says this about code generators

It is hard to see how such techniques generalize to the wider world of the ordinary software system, where cases with such neat properties are the exception. It is hard even to imagine how this breakthrough in generalization could occur.

Graphical Programming. Brooks has some interesting comments on software diagrams (Read UML and generating code from UML diagrams). I believe many of these apply just as easily to software developed today. How man times have you generated the class diagram from the existing code instead of the other way around?

Nothing even convincing, much less exciting, has yet emerged from such efforts. I am persuaded that nothing will.

Brooks goes on to say:

In the pitiful, multipage, connection-boxed form to which the flowchart has today been elaborated, it has proved to be useless as a design tool--programmers draw flowcharts after, not before, writing the programs they describe

Program Verification. In the simplest form this is a way to have the tests match the requirements (as David put it). In a broader form it is to verify program program correction overall. To many times we take this as Unit testing and think if we have 100% code coverage on our Unit Tests we are golden. Scott Hanselman has a two great podcasts that address this myth here and here. The root of this is as Brooks says,

More seriously, even perfect program verification can only establish that a program meets its specification. The hardest part of the software task is arriving at a complete and consistent specification, and much of the essence of building a program is in fact the debugging of the specification.

Environments and tools. Surely integrated development environments are the answer to better programs, right? Although I would not want to give up Visual Studio, the interesting thing is Brooks saw those developments coming 20 years ago and still did not see them as that important even today integrated debuggers are now decried as less useful then unit tests by agile developers.

One's instinctive reaction is that the big-payoff problems--hierarchical file systems, uniform file formats to make possible uniform program interfaces, and generalized tools--were the first attacked, and have been solved. Language-specific smart editors are developments not yet widely used in practice, but the most they promise is freedom from syntactic errors and simple semantic errors.

So what about the latest and greatest in workstations. That new laptop with 4 gig of memory, solid state disks, fast networking, 64 bit development environments...

What gains are to be expected for the software art from the certain and rapid increase in the power and memory capacity of the individual workstation? Well, how many MIPS can one use fruitfully? The composition and editing of programs and documents is fully supported by today's speeds. Compiling could stand a boost, but a factor of 10 in machine speed would surely leave thinktime the dominant activity in the programmer's day. Indeed, it appears to be so now.

More powerful workstations we surely welcome. Magical enhancements from them we cannot expect.

Attacks on the Essence of the Problem

So in the end we are left with an early form of the discussion on Team Velocity

If, as I believe, the conceptual components of the task are now taking most of the time, then no amount of activity on the task components that are merely the expression of the concepts can give large productivity gains.

Hence we must consider those attacks that address the essence of the software problem, the formulation of these complex conceptual structures. Fortunately, some of these attacks are very promising.

Here is a shocker for how to swiftly deploy systems.

Buy versus build. The most radical possible solution for constructing software is not to construct it at all.

...Any such product is cheaper to buy than to build afresh. Even at a cost of one hundred thousand dollars, a purchased piece of software is costing only about as much as one programmer year. And delivery is immediate!

The development of the mass market is, I believe, the most profound long-run trend in software engineering. The cost of software has always been development cost, not replication cost. Sharing that cost among even a few users radically cuts the per-user cost. Another way of looking at it is that the use of n copies of a software system effectively multiplies the productivity of its developers by n. That is an enhancement of the productivity of the discipline and of the nation.

And finally a initial discussion of the Knowledge worker of today

I believe the single most powerful software-productivity strategy for many organizations today is to equip the computer-naive intellectual workers who are on the firing line with personal computers and good generalized writing, drawing, file, and spreadsheet programs and then to turn them loose.

How about a early discussion on Iterative development. Software construction concepts based on a comment in 1958?

The hardest single part of building a software system is deciding precisely what to build.

...Therefore, the most important function that the software builder performs for the client is the iterative extraction and refinement of the product requirements. For the truth is, the client does not know what he wants. The client usually does not know what questions must be answered, and he has almost never thought of the problem in the detail necessary for specification.

...Much of present-day software-acquisition procedure rests upon the assumption that one can specify a satisfactory system in advance, get bids for its construction, have it built, and install it. I think this assumption is fundamentally wrong, and that many software-acquisition problems spring from that fallacy. Hence, they cannot be fixed without fundamental revision--revision that provides for iterative development and specification of prototypes and products.

...Incremental development--grow, don't build, software. I still remember the jolt I felt in 1958 when I first heard a friend talk about building a program, as opposed to writing one. In a flash he broadened my whole view of the software process. The metaphor shift was powerful, and accurate. Today we understand how like other building processes the construction of software is, and we freely use other elements of the metaphor, such as specifications, assembly of components, and scaffolding.

So instead we should grow software systems organically

Let us turn nature and study complexity in living things, instead of just the dead works of man. Here we find constructs whose complexities thrill us with awe. The brain alone is intricate beyond mapping, powerful beyond imitation, rich in diversity, self-protecting, and self renewing. The secret is that it is grown, not built.

So it must be with our software-systems. Some years ago Harlan Mills proposed that any software system should be grown by incremental development. [10] That is, the system should first be made to run, even if it does nothing useful except call the proper set of dummy subprograms. Then, bit by bit, it should be fleshed out, with the subprograms in turn being developed--into actions or calls to empty stubs in the level below.

I have seen most dramatic results since I began urging this technique on the project builders in my Software Engineering Laboratory class. Nothing in the past decade has so radically changed my own practice, or its effectiveness. The approach necessitates top-down design, for it is a top-down growing of the software. It allows easy backtracking. It lends itself to early prototypes. Each added function and new provision for more complex data or circumstances grows organically out of what is already there.

The morale effects are startling. Enthusiasm jumps when there is a running system, even a simple one. Efforts redouble when the first picture from a new graphics software system appears on the screen, even if it is only a rectangle. One always has, at every stage in the process, a working system. I find that teams can grow much more complex entities in four months than they can build.

As someone who has been developing software for over 30 years and never gone the management track, the next comment is very telling. I became a Fellow at Micron in less then 5 years and still did not see the dedication to the technical team he mentions here.

Hence, although I strongly support the technology-transfer and curriculum development efforts now under way, I think the most important single effort we can mount is to develop ways to grow great designers.

No software organization can ignore this challenge. Good managers, scarce though they be, are no scarcer than good designers. Great designers and great managers are both very rare. Most organizations spend considerable effort in finding and cultivating the management prospects; I know of none that spends equal effort in finding and developing the great designers upon whom the technical excellence of the products will ultimately depend.

My first proposal is that each software organization must determine and proclaim that great designers are as important to its success as great managers are, and that they can be expected to be similarly nurtured and rewarded. Not only salary, but the perquisites of recognition--office size, furnishings, personal technical equipment, travel funds, staff support--must be fully equivalent.

Conclusion

So in the last 30 years of software development we have only begun to addresses the essence of software development.

Software is Complex
Successful Software is ever changing
Software is invisible

So as you continue to develop software look at what you are doing. Attack the essence of the problem, not the accident.

Buy vs. Build where appropriate
Empower your users to allow them to solve problems themselves
Use Iterative Development Processes
Build strong technical teams and designers

100 times the current broadband speeds

2008-03-13T21:56:00.001-06:00

My friend Herb sent me a link to this article, Scientists Claim Breakthrough in Broadband Speeds. Pretty scary when you think about it. My broadband network at home then my gigabit wired network..

You always wonder what the connected world will be like in the future. Extremely high speed broadband networks could be the start. How soon before that hits the Wireless networks? My built in cellular modem on my laptop could soon be faster then my current wired broadband connections.

Maybe Ericsson is right?

C# 3.0 automatic properties...

2008-03-12T18:34:00.001-06:00

I am a firm believer in encapsulation so I do not like to have public fields instead of public property gets and sets in my code. So here is how I used to write classes in C# 2.0.

public class Person {    
       private string _firstName;     
       private string _lastName;     
       private int _age;     
    
       public string FirstName {     
              get {     
                     return _firstName;     
              }     
              set {     
                     _firstName = value;     
              }     
       }     
    
       public string LastName {     
    
              get {     
                     return _lastName;     
               }     
              set {     
                     _lastName = value;     
              }     
       }     
    
       public int Age {     
    
              get {     
                     return _age;     
               }     
              set {     
                     _age = value;     
              }     
           }      
}

Now C# 3.0 has automatic properties so I can just write...

public class Person { public string FirstName { get; set; } public string LastName { get; set; } public int Age { get; set; } }

Man is that cool! Great for the lazy coder in all of us. For more details see Scott Guthrie's blog post