Thoughtful Programmer

How to make Soduko boring - Follow the Process

2009-11-23T10:44:00.002-06:00

I spend a lot of time travelling to and from client sites, and when I travel I often pass the time by playing Soduko on my iPhone (I use a very good version from Mighty Good Games). Once apon a time I relied on the paper versions in the airline's magazines - but the iPhone app is now my norm.

One incentive for playing Soduko was to keep my little grey cells from getting lazy - but it also a fun way to pass the time, so I was rather surprised last year when my friend Roberta told me that she had given up the game after just a few months of playing. According to Roberta - who is a math teacher - the game wasn't challenging any more. She'd figured it out, and the puzzles were no longer puzzling.

My first reaction to Roberta's statement was typical for me - I just assumed that the Soduko puzzles she had access to weren't "hard enough". Many, many times I would finish a flight with an unfinished puzzle from the American Way Magazine staring me in the face. I certainly hadn't "figured out" Soduko.

Then in March I saw an article in USA Today about J. F. Crook and a Soduko "solution". Crook is a mathematician, but his "solution" isn't a mathematical proof - it's a series of steps. I'm a process guy - so when you say "series of steps" I say "process". Crook had published a process definition for solving any Soduko puzzle.

I read Crook's process definition, and it wasn't that far from recommendations that I has seen earlier. Soduko puzzles are grids of 81 squares, nine across and nine down. Some boxes have a number filled in; the rest are blank. Players must fill in the blank squares with numbers between 1 and 9 without repeating any numbers in a row, column or the nine interior 3-by-3 boxes of the puzzle.

Crook's Sudoku Solving Process can be summed up as follows: Mark all the squares in the puzzle that could be a "1" - Based on "what could be a 1" apply some simple rules to "solve" a few squares. Repeat this process for numbers 2 thru 9. Crook admits that you still may find some sqaures that might be one of two numbers, but you just "guess" a number for that square and proceed until your guess is either confirmed or invalidated.

The problem with Crook's Process is that it's really tedious. While solving a Sudoku puzzle you'll often have insights or spot patterns that lead you to solutions for a whole bunch of squares at one time. Consequently, when I read Crook's Process I said to myself "That's nice" and went on solving my puzzles the same way I always had.

My method for Soduko was to "solve" the obvious squares first, then scan the whole puzzle looking for the less-obvious relationships between the squares. Using that rather random process I could solve many "Moderate" puzzles in ten to fifteen minutes, but I was also frequently left with puzzles that I just couldn't crack.

When I switched from paper Soduko to iPhone Soduko I initially followed my old-but-not-reliable process... but the iPhone apps features opened my eyes to "the better way".

With paper and pencil it's really messy to follow Crook's Process. There's a lot of erasing that goes on, and it's really easy to make mistakes. With the software version I can mark/unmark squares to my heart's content without reducing the legibility of the puzzle. The software version also immediately flags my mistakes, so I don't multiply those mistakes by contiuing on with bad data driving my subsequent decisions.

A little bit of software can make it a lot easier to follow a process - Who would have imagined that? :-)

Even with my new found ability to make notes on the puzzle - I still stuck to my old process for the most part. I's use my process to solve as much as I could, and then resort to Crook's process... and sure enough I would still get stuck with puzzles that I just couldn't solve. This didn't happen often, but it happened.

I should mention that up until this time I was limiting myself to the "Moderate" puzzles on my iPhone. It's very unsatisfying to have an unsolved puzzle when my plane lands, so I stuck to the relatively easy ones for the most part.

On a recent flight I decided to tackle an "Expert" Soduko puzzle - and it was a bear. I was so intimidated that I followed Crook's Process. I diligently marked all the "1's", then the "2's", etc. and sure enough in 30 minutes I had solved the puzzle. I tried again with a new puzzle, and sure enough, in 30 minutes I had solved that one too.

Since that time I have improved a bit, I'd say my average is about 25 minutes, but sure enough I can solve any puzzle as long as I keep to the process. Just like Roberta, the puzzles aren't really puzzling anymore. Give me 30 minutes and I will solve the puzzle.

The trade-off here is that I won't be able to solve any puzzle in less time. The process is tedious and it's not easy to speed it up. I sacrifice "brilliant flashes of insight" for dependable, predictable, and plodding. The process makes Soduko boring.

As always I have to turn this experience into an analogy for the domain of Managed Business Processes: I could have followed Crook's Soduko Solving Process all along - but a little bit of software really made the process easier to follow. Software made each step of the process easier, but it didn't force me to stick to the process - If the software had managed the process (forced me to stick to the process) I would have achieved the goal of consistently solving Soduko puzzles sooner.

You might be saying - "Yeah, but it also made it boring".

That's certainly something to consider when introducing a Managed Business Process into your organization... We don't want to turn our process participants into mindless drones. We don't want to lessen the importance of creativity and insight - but we do need to make things more predictable and insure success. Finding the balance between predictable process and bored employees in your organization is a puzzle that won't ever get boring.

Tips for the Business Process Developer - Process Data Perspectives

2009-11-20T15:49:00.012-06:00

Those of us who build Managed Business Process solutions for a living have to spend a lot of time thinking about data. Our BPM suites make it a snap to define Process Flow and Routing, but we still have to make sure that each Participant in the process has access to all the data that they need to accomplish their tasks.

I always insist that folks implement their process flow first. Build a place-holder for each Activity in the process, connect them up with flow lines and any necessary Decision Gateways, and add in just enough process data and build just enough user interface to make sure that you can play back each step through every path in the process with your business folks.

Building this much of your process first has huge paybacks - There's nothing like stepping through a process to validate whether or not the business folks agree that you've got the process right. Validate the process requirements first - worry about everything else later.

Once you have this first playback of the process done you've got to turn your attention to defining the rest of the process data... and suddenly it's not as clear how to proceed. Where do you start?

The data aspects of a Process encompasses many things that you might not consider when developing other types of applications. There are several common process data patterns to consider - processes gather, transport and transform data across activities that are performed by multiple participants. Data models often dynamically evolve and transform as the process proceeds.

I've had several recent discussions with Fahad Osmani and some of my other Lombardi colleagues about how to effectively develop process data models. We've all implemented a number of managed business processes - We've all thoroughly mastered the craft of implementing processes - But we haven't really formalized our approach for designing process data models. Knowing in your gut what to do is great, but formalizing that into something that you can share with others is better.

Here's what we've come up with so far:

In every business process you've got a variety of actors with a variety of perspectives that you have to deal with - we've abstracted this into the following list:

1. Process Owners
2. Process Analysts
3. Process Participants
4. Process Builders

We've found it best to design a process data model from each perspective in this order, and then consolidate those models into a consistent unified model.

The Owner's Perspective of Process Data:

Start with the owner's perspective of process data - the folks with overarching concerns to make sure that the outcome of a process is achieved. If you're working with a Bank on a Loan Application process, these are the managers of the Loan Department who are held accountable for efficient and effective handling of the loan applications.

Owners have a business perspective on the process data. They've got the big picture business view of the process payload - What are the basic business objects that the process deals with? What data is being gathered? How is the data transformed during the process? What happens to the data when the process ends?

Owners often see process payload data as starting points and outcomes. Their major concern is with the state of the business data "now". They care about the state of the business objects when the process starts, the state of the business objects at key milestones, and the state of the business objects when the process finishes. The owner's data model provides that information.

Beyond the payload data, owners also perceive data in terms of process rules - What data determines which activities to perform? What data determines when a task is due? What data determines who should perform a task? The owner's data model answers these questions.

The Analyst's Perspective of Process Data:

Process Analysts are the folks who look at historical process metrics to figure out if the process can be tweaked to work better. In some cases the Owner may also be the Analyst - but the Analyst perspective really is different. In a Loan Application process, the analysts are the folks who evaluate the effectiveness of the process itself in the hopes it can be improved in the future.

Analysts deal with understanding the past to improve the future.

Analysts view the process data as snapshots in an album. Their view of data is intricately linked to the process diagram - The snapshots are meaningless to them unless they know where the snapshots were taken.

Analysts care about how long things take and why some things take longer than other things. Analyst care about "rework" (tasks that have to be redone). They don't really care much about what's happening now - they care about what happened over some time period. To meet the analyst's needs, process context data (and time stamps) are stored at specific points in the process. Making sure that the right snapshot is taken at the right time is their major concern. The analyst's data model works in conjunction with "snapshot events" in the process flow to capture the information the analyst needs.

The Participant's Perspective of Process Data:

Participants in the process are the folks who are tasked to perform specific tasks. Sticking with a Loan Application process, these would include the Loan Officers who review and make approval decisions about loan applications.

Participants care about now. Their perspective of process data is immediate. What forms do they need to fill out? What fields are on each form? What are the acceptable values for each field on a form? Participants view data from a task level - What information do they need to consult and what information do they need to gather and modify to accomplish their task.

Some information is passed to the participant when they are assigned a task, and some information they look up - but it's all information that is necessary to accomplish a task. Making sure that they can access data and modify the data that they need is their major concern.

One special caveat when considering the Participant's view of data is to ask whether or not any of the data that the Participant changes must be "immediately" seen by others. Data visibility requirements like this are critically important to the Process Builders.

The Builder's Perspective of Process Data:

Finally we get down to the Process Builder's view of the process data - that's us. We really can't do our work until we understand what the Owners, Analysts and Participants need.

Builders care about the actual structure of the data. What data are we going to pass around and how are we going to pass the data around? What data needs to be stored an retrieved and how are we going to store and retrieve the data? How are we going to meet the data visibility requirements?

The Owner's perspective helps the Builder define the major business objects that the process will deal with. The owner's data model also identifies the data that is used to determine which paths are taken, how long each task should take, and who should perform each task.

The Analyst's perspective helps the Builder define the data structures for the snapshots that are taken throughout the process - and informs the Builder where in the process the snapshot needs to be taken.

The Participant's perspective helps the Builder define the process payload (the data that is passed to and from tasks) and it helps them define the data system architecture that is required to retrieve and modify data within activities.

The Payoff:

The end result of this multi-perspective approach is a model for process data that just plain works.

It takes a bit more up-front effort, but you are less likely to get blind-sided by unanticipated data requirements popping up late in your development cycle. The data requirements may change, and any requirement change may still require substantial rework - but odds are the rework will be much easier for you to accomplish.

Clearly understanding requirements from the perspectives of your the process actors always pays off - and this axiom is doubly true when data is involved.

Open Source + User Builders = Great Products

2009-11-11T07:37:00.006-06:00

I read this story over at npr.org this morning:

"Before Jonathan Kuniholm had a tour of duty in Iraq, he worked for Tackle Design, an industrial design, research and development firm. After that tour, he was missing part of his right arm — which he lost when his Marine patrol was ambushed near Haditha.

When Kuniholm returned to his design shop, he brought along three prosthetic arms given to him at Walter Reed Medical Center — the same body-operated hook many veterans have used since World War I, a shorter utility prosthetic, and a new, state-of-the-art myoelectric arm. Each one had its drawbacks — and when Kuniholm and his Tackle Design colleagues disassembled them, they quickly concluded that they could improve on the designs. They founded the Open Prosthetics Project, an open-source collaboration that makes its innovations available to anyone. And Kuniholm signed on with Revolutionizing Prosthetics, an initiative of the Defense Advanced Research Projects Agency, or DARPA."

There's an important lesson here for every company that builds products:

Users are a better judge of what your product should do than your developers.
Users who have the skills to build your product themselves will improve your product if given the chance.
If you don't let your User/Builders improve your product, your product won't be as good as it should be.

Viva Las Lunas!

2009-07-23T09:53:00.006-05:00

It's taken me a few days to figure out what I wanted to write about the 40th anniversary of the moon landing... I was 12 years old - and the moon race played a very prominent role in the world view of that nerdy 12 year old.

I remember an evening shortly before "One small step" - I don't remember if they'd already landed or if it was between the touch-down landing and Neil Armstrong's famous descent of the ladder - but I was outside with my father and looking up at the moon. Tom, my father, was talking with his friend "Mr. Hahn" about the landing, and "Mr. Hahn" asked me if I was going to watch it on TV that evening.

For some reason, my response was "No". Perhaps I was just being a contrarian pre-teen, but I definitely remember him reminding me what a special occasion this was. Humans had been standing on the earth looking up at the moon in the sky for thousands of years, but never before had someone been standing on the moon looking back at the earth.

"Mr. Hahn" was a rather profane man, born into the wild-catter-oil-patch culture of West Texas and not predisposed to poetry - but he shook me out of whatever funk I was in and I went home and watched Neil and Buzz with rapt attention.

What amazes me to this day is the undeniable fact that at 12 I was already jaded by the space program before the moon's first footprint. I've been going through my stash of memorabilia weeding things out lately, and I came across a folder bulging with NASA photos and articles and sketches of my own plans for space ships. There's quite a lot of "Star Trek" along with the NASA glossy photos - obviously I was hooked on space.

When I was 9 years old, we were all going to live and work in space. The first astronauts were super-human test pilots - but that made sense. They were blazing the trail that we were all going to follow. Space was not going to be the sole domain of multi-Phd toting eggheads - "Normal people" would be there doing all the things that "normal people" do.

NASA was an agency of innovation. They were constantly unveiling new rockets, capsules, rocket planes, etc. It was always new and exciting.

Perhaps the Apollo One disaster was the beginning of my disenchantment. The inferno that killed those 3 heroes was due to sloppy workmanship, not technical complexity. Even as a 10 year old I was sickened. How could anybody associated with the space program be sloppy?

After the last moon flight it just got worse.

Skylab was a disaster on launch - Solar panels ripping off during launch due to poor construction. The shuttle - supposedly cheap and reliable - just a ticking time bomb of compromises that had more to do with keeping the money flowing to contractors than to accomplishing anything worthwhile.

NASA (the agency) has no luster left to lose - not the people. The people are great. The agency NASA should be scuttled and its responsibilities parceled out to more focussed agencies: NOAA, DARPA, DOE, etc.

My space heroes are now Elon Musk and Robert Bigelow. These guys understand that interest in space has little to do with science - it's about adventure. If it's not exciting, then why go?

NASA's plans for a moon base are about as exciting as the Antarctic Scott Base. How many people do you know who are dying to visit Scott Base? (How many have even heard of it?)

I know many people who would love to take a cruise to Antarctica to see the fjords, glaciers and penguins... but who wants to go sit in an over sized high tech igloo? Compare NASA's 2009 vision of a moon base to their 1970's vision for a space settlement and you'll see how boring the future has become.

Vegas baby... the moon doesn't need a "base", it needs a "strip". Vegas is a city that never should have been - a gaudy exciting bubble of sex, sin and gambling in the middle of absolutely nothing. If Vegas didn't cater to hedonism, there wouldn't be a Vegas.

Musk and Bigelow understand that. They're serious businessmen, and they're following the money that's available now (COTS, etc.), but I think they know that the NASA gravy-train and communication satellites are only going to take things so far. The moon's going to remain forlorn until Cirque du Sole has a lunar venue where they can perform.

40 years ago something truly amazing happened on the moon. Let's hope that 40 years from now we don't have to look back 80 years for inspiration.

Get the Words (and Notation) Right

2009-07-22T17:31:00.009-05:00

Dan North (the chief evangelist for Behavior Driven Development) loves to say:

"Get the Words Right"

In the context of creating software if you don't know what somebody wants you to build - then you can't build it for them... That statement is true for lots of contexts but let's stick to software for the moment.

If the person who wants something built uses terms that you don't fully understand, then you'll have to translate those terms into something that you do fully understand. Translation is never perfect, so you're bound to screw up something. When you are ready to explain what you're building to the client (to confirm that you are in fact building the right thing) if you use terms that the client doesn't fully understand, then they'll have to translate your words - and they are likely to misunderstand what you're building.

Sometimes a 3rd party will be on hand who understands both the client's and the builder's terms (I do that a lot)... but then you are dependent on the translation skills of the 3rd party. The good news is that both you and the client can blame the 3rd party, but your software still stinks.

Dan's solution to that problem is to spend more time up front agreeing on a common vocabulary between builder and client... with a definite prejudice towards adopting the client's vocabulary whenever possible.

Seems pretty much like common sense, but it's very hard to actually do it.

Many programmers just don't really want to understand their client's businesses... they're excited about programming, not business. Business concerns are mundane and boring compared to the challenges that they face when creating software... in their opinion. Many business people are just as bored by technical details as programmers are by business details.

It's hard - but in truth it's the easiest way to succeed. When everyone is on the same page building projects are actually fun... and the "buildings" are really fun to "live in".

In the Business Process software domain, we have a common notation for expressing the "flow" aspects of processes - BPMN. BPMN is relatively simple to learn and it has proved to be a very effective means for Business and Programmer to communicate about the "flow" aspects of their business processes. Unfortunately, BPMN isn't good at all when communicating all of the other aspects of a business process (data requirements, routing rules, etc.) but it's a good start.

BPMN is Notation rather than words... but Dan's point is hugely relevant - you have to get the Notation right.

I've worked with BPMN for a few years, and I think the Notation has elements that just aren't right. They don't convey the meaning of the symbol to most Business people.

Here's my least favorite symbol - the Multiple Instance Activity:

When you see this symbol, you are supposed to know that many copies (instances) of this Activity will be performed at the same time. You "know" this because there are those two little parallel lines on the Activity - They're parallel to each other, so of course that means there are parallel Activities taking place.

I am not a Business person, but I doubt that many would describe Activities-That-Happen-At-The-Same-Time as parallel Activities. If someone out there knows of a study that proves me wrong, please tell me.

I think a better symbol would be something like the following:

With this symbol everyone can see at a glance that Multiple-Things-Are-Happening (hopefully at once). I'd also add some indicator of the number of "Things"... although often that's only determined when the process is running.

I suspect that the current Multi-Instance symbol was chosen because it is easy to draw by hand. Fair enough... but I find that most of my hand-drawing is done on white boards, and I almost always draw something that looks a lot more like my proposed symbol.

Fortunately, with the current state of BPMN editors it is usually pretty easy to swap out the "official" notation's symbols for your own... There are much bigger problems to solve in the realm of conveying Business Requirements to Programmers than this little icon - so please don't waste any time lobbying for it to change. I'm just using it to illustrate the point:

Words Matter - Notations Matter - Diagrams Matter

Pictures are worth a thousand words, so take the time to get the picture right before you start building your next project.

The Toilet Paper Replenishment Process

2009-07-14T22:55:00.005-05:00

Don Norman has an excellent blog posting about the poor design of many Toilet Paper Dispensers - focusing mainly on the poor algorithms that we simple humans use when given a choice of which roll to use from a multi-roll dispenser... Multi-roll TP dispensers prove Don's thesis that many every day things just aren't designed well. I had read Don's book and seen his blog long ago, but it came back to mind when I encountered the following in a bathroom in Stanford University's Terman Engineering Building...

No less than six partial rolls of toilet paper perched on top of a standard two roll toilet paper dispenser.

Obviously Don is right about the design of the dispenser itself... Given the choice of which roll to use we'll pull sheets off each. But obviously there's something more at work here: The process for refilling the dispenser is suspect.

I suspect the process is something like the following:

Check the dispenser each day. If either of the rolls is less than 1/4 full
replace it with a full roll and place the partial roll on top of the dispenser.

The objective of this process is to make sure that there's always plenty of toilet paper available... Obviously you don't want to run out.

My guess is that the partial roll is left on the dispenser in the hopes that folks will use it - This is much better than throwing it away, and given frugal college students I'm surprised that the partials aren't disappearing.

The flaw in the process seems pretty simple... "Enough Toilet Paper on Hand" doesn't take into account the partial rolls. Instead of basing replacement on the size of the rolls installed in the dispenser, it really ought to be based on the total amount of tissue on hand - the rolls in the dispensers plus the partials.

At first glance this appears to be a classic example of mis-stating the objective of the process. The janitor who refills the dispenser has been told something like "No roll on the dispenser should be less than 1/4 full". The real objective should probably be something like "There should always be the equivalent of a full roll on hand".

Having restated the objective: What if you ended up an "equivalent full roll" made up of sixteen partial rolls? That certainly wouldn't be acceptable to most folks.

Returning to Don Norman's point - the design of the dispenser itself is really at the heart of this "process" problem. The process for refilling the dispenser is overly complex due to the poor design of the dispenser itself. Try as we might to give proper instructions to the janitor, success is going to be highly dependent on the janitor's judgement on whether or not enough toilet paper is on hand.

What can we learn from this example?

If a process that should be simple isn't, then look beyond the process definition: It may be better to buy a new dispenser than to figure out how to refill the one that you have.

UPDATE:

Prompted by the comment of a good friend and Stanford Alum I delved a bit deeper into the conundrum and sadly discovered that this is both a Process and a Design problem... In another bathroom at Stanford I discovered the following shocking sight:

The superior design of this multi-roll dispenser has been thwarted by the installation of two such dispensers... The point of the design was to eliminate choice (thus simplifying the refill process) - introducing a second dispenser nixes that advantage... and as you can see the "refill process" is still needlessly removing partial rolls.

Such a thing would never have happened at my alma matter (Rice University) ;-)

Changing planes while in the air

2009-07-09T22:34:00.003-05:00

I fly a lot... way more than I ever thought I would... and I can say with some confidence that it's sometimes much harder to get from point A to point B than you thought it would be.

My home bases are Austin Texas and Santa Fe New Mexico, and neither is an airline "hub". Airline hubs are those wonderful airports where you can fly to and from a lot of places (non stop)... Neither Austin nor Santa Fe, as I just said, are one of those wonderful places.

Virtually every trip that I take requires changing planes. I land, scurry to another gate, and take off again. "Scurry" is often a nice way of saying "run flat out as fast as I can". For those of you who don't fly often, you may not realize that most flights board 30 minutes prior to departure, so if you think you have an hour between flights, it's really only 30 minutes... even a minor flight delay can turn "plenty of time" into a missed connection or lost baggage (if you are brave enough to check your bags).

Each trip that I take is (in a sense) a process. I fly from home to an airline hub, then I fly from that hub to another airport. On rare occasions I have to fly from the second airport to a third... You get the idea. On each flight, I am sitting along with a couple of hundred other people who are each taking part in their own similar processes. As long as all of the planes that we'll use are on time (the plane that we are on along with all the planes that we'll connect to) all of our processes will run smoothly. If the plane that we are on has mechanical difficulties and has to return to the airport, or if weather or some other obstacle keeps us from landing where we expected to, then our processes are all in a heap of trouble.

From a process design perspective the events that I have described are exceptions. We don't "expect" these things to happen, but we have to handle them if they occur. Some airlines have processes in place that work really well... other airlines... well... let's just say that I've slept on a bench in O'Hare airport and leave it at that.

Exception handling is actually pretty well understood in the realm of Process Design. With BPMN notation you can attach event handlers to Activities and model "what needs to happen" when something out-of-the-ordinary happens. Proper event handling requires a lot of thought... but in a sense it's a well understood aspect of programming that isn't specific to process engineering... and it's not really what I want to blog about today.

Let's go back to flying... This example is admittedly kind of absurd, but imagine that the airline changes their schedule while I am in the air. Let's say that the itinerary that I am on is now supposed to go from Austin to Denver to Chicago instead of from Austin to Dallas to Chicago. From now on, whenever I want to go to Chicago I will change planes in Denver instead of Dallas.

What I've described here isn't a process exception - it's a process change. The airline has changed the process of getting from Austin to Chicago.

For somebody starting a new trip, it's obvious that they will follow the steps of the new process definition. For folks who are already in the air on the first leg of the trip, there are two options:

Continue the following the "old" process steps until the process finishes - this depends on the airline keeping a flight from Dallas to Chicago until everyone in the air gets there.
Switch to the new process while "in-flight" - this depends on the airline re-routing the plane mid-flight to land at Denver instead of Dallas - or perhaps jumping from one plane to another as they pass each other.

Option number one is really safe. The old process is well understood and it's very clear where everyone will end up... we know we have enough gase to get to Dallas. Unfortunately continuing an old process until it is complete isn't always a viable option. Some processes take a really long time to finish, and it's unreasonable to carry on as if nothing has happened (sometimes for years) - the airline can't keep an unfilled plane waiting at Dallas.

Option number two - metaphorically changing planes while in the air - can be very risky to your in-flight process instances - the planes in the air may not have enough gas to get to the new destination, and it's quite likely that some of the passengers on the flight planned to get off the plane in Dallas. Let's not even think about jumping from one plane to another.

It's this ability to handle in-flight process changes that makes the implementation of a really good process manager difficult. Creating an "engine" that can execute a process definition is pretty straight forward... it's mostly a state machine.

Switching from one process definition to another while "in flight" can be a heap more complex...

Process improvement is a key goal of BPM adoption... When you figure out how to improve a process you want to be able to implement that change as soon as you can... but process change requires more that modelling the new process... you also really need to model the process of getting instances from the old process definition to the new.

For every possible step in the original process definition, you need to define the corresponding step in the new process.

If the correlation isn't exact (one process definition has steps that don't really exist in the other), then you must tell the process manager what to do. In some cases, an in-flight instance may need to "go back" and redo a "previous" step in the new process... or the in-flight process may have to perform some "transformation" steps before merging with the new process flow.

Unfortunately, BPM tools don't generally have explicit support for "migration" processes... it's generally left to the practitioner to figure out what to do with in-flight instances, and sometimes there's a rather painful transition when a new process definition is deployed. I'm sure this will change as BPM adoption increases, but in the mean time you'll have to build your own parachutes in case something goes wrong.

People Manage the Process?

2009-06-25T13:51:00.004-05:00

Gerhard Basson has posted a very nice article Process-oriented Systems Paradigm for the Process Age over at the BPM Institute web site. I agree with almost everything that Gerhard says... but as usual one little phrase pops out and spawns a blog of my own.

Gerhard asserts:

"processes do not manage people – people manage processes"

I understand what Gerhard means... very clever word-smithing... but I squirmed when I read it. I know that I am splitting hairs here, but "manage" just isn't the right verb for the phrase:

"people verb processes"

That aside, I vehemently agree (as opposed to vehemently disagree) with Gerhard when he says:

"People will bypass and reject any system that does not help them perform work in a natural way."

It's always about the people... From the conception of the process to the implementation of the process to the running of the process to the improvement of the process... it's always about the people.

If we don't adapt our systems for the People of the Process Age we're not going to get very far.

Programming Language Sociology

2009-06-24T09:55:00.006-05:00

Bruce Eckel shares his reminiscences about the dawn of C++ and Java in his blog entry: Why? Language Archaeology.

This is great stuff... Insight into the "why" C++ and Java are the way they are (from Bruce's perspective). My take-aways from this essay are the following:

"Things" often make no sense because the requirements that dictated those "things" are no longer requirements
People skills can be just as important as Technical skills

I really appreciate Bruce admitting that his opinions about Java are colored by his dealings with Gosling... Social aspects impact Technical aspects. How you feel about someone impacts your evaluation of their work. That is a hugely important thing for us to remember.

I was an early adopter of C++ and loved it... but I loved Java more. Was that due to Java's technical superiority?

Nope.

I loved Java more because it coincided with the Web... and the Web allowed me to collaborate with other Java developers in a way that I had never been able to collaborate with other C++ developers. The social aspect of this collaboration is why Java trumps C++ on my "fondness scale"... Java had (and still has) a much stronger sense of community than C++.

The stereotype of the programmer nerd with poor social skills had a lot of basis in fact... but in my experience programmers really crave social interaction within a society where they feel comfortable.

My programming focus is now in the realm of Business Process Management, so the society in which I find myself is much broader than that of my C++ days. In my C++ days I wrote stuff for business people. Now I write stuff with business people. Huge difference.

Successful Business Programming requires a society where both Business and Technical people feel comfortable. That's tough to accomplish because the passions of those two camps rarely coincide...

Next time you're in a conference with both Technical and Business people, sit back and watch:

When the Business folks get into an animated discussion the Technical guys will be checking their Blackberries. When the Technical folks get into an animated discussion, the Business folks will start looking at their watches.

The challenge in business programming today is not about programming language features - it's about the social aspects of application development. We've got to solve the problem of engaging everybody in the development process. We need to fully engage the Business folks to make sure that we build the right things, and we need to fully engage the Technical folks to make sure that the things are built right.

Compared to the real challenges of business programming, agonizing over whether or not to include the "new" keyword in Java is seems positively "quaint". Bruce's recollections are a fond trip down memory lane for me... but lets not get stuck in the past lest our descendants look back on our concerns as "quaint" too.

Good Engineering != Good Design

2009-06-09T08:41:00.004-05:00

This thought-provoking statement grabbed my attention...

"Most tech products—heck, most products in general—aren’t as good as they can be because they’re put together by the people with the technical knowledge required to build them."
Jason Snell, Macworld.com

Jason makes some really good points in this article... Go read it ;-)

Artists (and programmers) hate negative feedback

2009-06-01T09:43:00.006-05:00

If you really want to piss off an artist, just tell them how their painting/sculpture/etc. could be improved.

The best possible reaction that you'll get is a politely disingenuous "Thank you"... More likely you'll get some form of "Mind your own business" delivered with various levels of hostility based on the strength of the artist's self-esteem.

The creation of art is intensely personal... Art is an extension of the artist - created primarily to satisfy the artist's need to create. Art is presented to the public - not shared with the public. The artwork may be sold - but it always belongs to the artist.

What does this have to do with Programming?

For many programmers - Programming is their Art... and that's the root of many of the conflicts that you will have with programmers.

There's an anecdote about Frank Lloyd Wright: The story is that he attended a dinner party in a house that he had designed, and before leaving he had rearranged the furniture back to his original plan. Wright couldn't let go... The house was held hostage to his vision rather than released and allowed to become a home. Wright liked to preach that "Form follows function" - but in this case he totally forgot the true function of a home.

When you have to provide negative feedback to a programmer - focus on conveying the big picture: What do you really need? Instead of focussing on how the current software does not meet your needs, challenge the programmer's inner artist to create something that will fulfill your needs... You'll probably be much happier with the results.

Process Manager Interoperability - Wf-XML

2009-05-08T11:16:00.004-05:00

Scott Francis sent me a snippet from a recent Keith Swenson blog posting on Wf-XML...

I've been looking for a standard like this... Process Folks have been pretty focused on Process Definitions that can be executed on any Process Manager (Process Portability), but in my day-to-day it seems a lot more important to insure that Process Managers can operate with each other...

Specifically, I want to be able to incorporate a Process that is running on one Process Manager as a Sub-Process of a Process that is running on another Process Manager. I want to be able to start the Process, Monitor the Process, and get Information back from the Process when it completes.

Much as we might like a world in which there is only one Process Manager, this just isn't going to happen. Consider the Java app server world... Despite corporate attempts to mandate the use of only one Java app server you still find a mixed bag. Once a WebApplication is developed and deployed on one flavor of Java app server it's seldom migrated to another. It's just not worth the expense or the hassle - even though migration might be relatively straight-forward.

In the Process World we should be able to take full advantage of previously deployed Process Applications regardless of their BPM platform. If an application has been developed with an embedded jBPM Process Manager I should be able to incorporate it in my Lombardi solution. If my Lombardi implemented Process can be used by a wider Pega implemented Process, then that should work too.

Wf-XML is an attempt to make that happen... I haven't had time to delve into the details, but the concept is right on the mark.

College new-hires and the zen of BPM

2009-04-29T10:05:00.002-05:00

I'm supposed to be preparing a one hour lecture for some college new-hires on: "Basic and Advanced BPM Concepts"... but yet I blog instead ;-)

For those of you who know me well, you'll probably think that my biggest concern is that I only have an hour. I tend to go on and on (and on) about topics that I am passionate about, leading a former co-worker to coin the term "johntification" in my honor.

Explaining BPM is what I do - focussing pretty much on what's now being called BPM-tech more than BPM-bus. When I first encountered BPM I felt like I had found the missing link... a paradigm and technology that closed the gaps between what I knew how to do and what my business colleagues really needed.

I was blown-away by BPM because I had experienced the pain of the problems that BPM addresses. This stuff fixed something that I knew needed fixing.

My worry about introducing BPM to college new-hires is that they probably haven't experienced "Process Pains"... at least not from the perspective of one who writes or maintains software, or from the perspective of someone who tries to get the "right" software written.

When I describe the problems that BPM tackles they may say "So what?". They may scoff at the magnitude of the problems - and they probably assume that the solutions that BPM provides have always been around.

With my standard BPM audience I'm fairly assured that heads will begin to nod in recognition of shared pain in thirty seconds or less...

Most in my audience have experienced meetings where a dozen people had to be present to figure out how that incoming *application* finally ended up as an outgoing *disbursement* (*substitute the inputs and outputs of your own business).

Most in my audience have also experienced "Office Heroes" - those harried individual who on a daily basis keep a company running through shear force of will. Whenever anything falls through the cracks... Whenever anything gets lost or derailed... Whenever any critical deadline is in danger of being missed... Office Heroes jump in and save the day. If a truck hits an Office Hero your business will really have to scramble to recover.

BPM helps business people understand how their company really runs, and it helps reduce their company's reliance on Office Heroes. BPM helps IT people provide tools that make it easier to run the company, easing the burden of the Office Heroes.

My college new-hire audience will certainly understand what I tell them about BPM - but will they relate? Can I make BPM relevent to their own experiences, or will I just sound like an old guy droning on about the old days: "When I was your age, computers only had 4K of memory....."

These "kids" are smart... they just haven't had as much experience. Instead of relying on shared experience to grok BPM I'll have to find another approach... When (or if) I figure it out I'll let you know.

Applications as Process Activites

2009-04-17T13:11:00.009-05:00

I return to this topic a lot... BPM solutions automate the transitions between Activities in a Process, but they don't necessarily automate the Activities themselves. Some Activities can be automated, but others require Human interaction or Human judgment to complete. If we are convinced that BPM is a good idea, and we want BPM to be as pervasive as spreadsheets, then we need to make it easier to implement the Human-centric Activities that our Processes need.

Many BPM suites support building Human-centric Activities within the suite itself, but sometimes you'd rather re-purpose an existing application to do your bidding.

Let me walk you through an example from my own wonderful life as a Travelling Guy...

When I get a new assignment there are a lot of things that I have to do. I've got a mental checklist that I follow, but I'd really rather have a Managed Process to keep me from screwing up.

Most of my gigs are somewhere else, and that makes it very important for me to make travel and lodging arrangements. I almost always need a flight and a hotel, and based on the location I might need a rental car.

To perform this Activity I use my company's preferred Travel site - it happens to be Orbitz, but it could just as easily be Expedia, Travelocity or any number of sites. Each of these sites supports the concept of a "Trip" and each lets me make flight, hotel, and car reservations. These travel sites also provide feedback - they'll send me email when a reservation is confirmed or "something" changes.

As I am going through my process of getting ready for a gig, I log on to Orbitz and make the necessary reservations. When I am "done" I will "check off" that task from my list of things to do.
This happens a lot in BPM solutions. An Activity (a task) is assigned to an individual Participant. The Participant gets directions on how to perform the Activity (what it is that they need to do) but they have to use some "external" application in order to actually perform the task (in this example, the Travel site). When they are "done" with the task they "check off" the task and the process continues.

Processes that are implemented like this can be really error prone. The Process Manager has to rely on the Participant's assertion that the task has been performed correctly. You just have to trust the Participant to do the right thing.

If you had the time and if you had the money, then you could build all of the tools that the Participant may need to use in order to complete their tasks. Quite frankly, that would be a huge waste of both your time and your money.

The better approach for all concerned is to encourage the makers of those "external" tools to become Process Aware. Get then to modify their existing applications to be aware of Process Flow.

Let's go back to my "Get Ready for a Gig" Process... Some time after the Process starts I will need to perform the "Make Reservations" Activity. The Process Instance knows the Location of the Assignment, and it knows the Dates of the Assignment (the Process was started by a message that contained this information).

When I run the "Make Reservations" Activity from my Task List I'd like Orbitz to open with the Dates and Location of my Trip pre-populated. After I make my reservations, I would like the site to send (structured) confirmation information directly to the Process Manager.

Based on the information that Orbitz returns, the Process Manager would either move on to the next task, or require me to try again. For example - if Orbitz says that my reserved flight is on the wrong day the Process Manager should tell me.

If you are a Programmer, then I think you can easily envision how you would make the Webapp that implements the Orbitz Travel Site "Process Aware". The trick is in recognizing that there is a beginning and an end to planning for each Trip. Modify the Webapp to support "Dates" and "Location" on the invoking URL - along with a "Return Address". When the Webapp determines that planning for the Trip is complete, use the "Return Address" to send structured information (most likely XML) back the the Process Manager.

For Web Services (automated Activities) we've done this for several years. Service providers (like Orbitz) publish WSDLs that define the Services that we can invoke. It's time to do the same for Activities that cannot be Automated. It's time to standardize interactions with what I've called Human Powered Web Services - Web Applications that can serve as Process Activities.

In my ideal world, all Travel Sites would support "the same" interface for starting a new "Trip" and for returning results. That might be a bit much to hope for... but it's a good target to shoot for. Even if every site had a unique interface, we could build adapters for our Processes - a much better situation than today (from a Process Manager's perspective).

If you are a Developer who builds sites where folks go to "do things", then I hope I've inspired you to think a bit about those "things" as part of a larger Process. It's not a huge leap to build into your site the hooks that a Process Guy like me needs to incorporate your site into my Managed Process. I'll be grateful, and I'll bet the users of your site will be too.

Clarifying your ideas

2009-04-14T13:15:00.009-05:00

I love the Webcomic xkcd by Randall Munroe... Geek humor for sure, but often very insightful...

"You'll never find a programming language that frees you from the need to clarify your ideas"

Amen to that.

Programming is the act of transforming requirements into something that a computer can execute... but before those requirements can be transformed they'd better be clear. You might think that this would be obvious, but you'd be wrong... In way too many cases the "idea" that spawned the software either wasn't clarified or it wasn't communicated clearly.

Programming languages - both textual and graphical - won't substitute for an idea that hasn't been clarified. But "better" programming languages can help to insure that the idea has been implemented properly (or at least make it clear what was actually built).

Business Process and Activity Interfaces

2009-04-08T08:37:00.025-05:00

Business Processes are made up of Business Activities. In a BPM solution the transitions between each Activity are automated, but the Activities themselves may either be automated or they may be performed by Humans (That's why I use the term Managed Business Process instead of Automated Business Process).

Most of the focus in the community of BPM practitioners of late has been on the Business Process Definitions (BPDs) that the Process Managers use to control the transitions between Business Activities. BPDs provide the information that the Process Manager needs to assign the right Activities to the right Participants at the right time. Google BPMN, BPEL or XPDL and you'll find plenty to read.

BPDs are crucial... but they only define the Process Flow. Activities are identified in BPDs, but only the interfaces to the Activities are actually defined (The interface to the Activity defines the information that is passed from the Process to start the Activity and the information that is returned to the Process from the Activity when it completes).

All of the BPM suites that I am aware of generate executable process definitions (in one form or another). Some BPM suites generate "portable" definitions using standards like BPEL and XPDL, but many (including the Lomabardi Teamworks suite that I use) produce proprietary definitions that will only run on their own Process Managers. One of the big drivers between the efforts to make BPMN 2 "executable" is to drive the industry towards "portable" BPDs that can run Process Managers from any vendor. Great idea... but it's going to take some time to see if it gains traction.

Many BPM suites also incorporate tools to implement Business Activities. A Business Activity is really just an application that is tailored to perform a specific task. The Process Manager kicks off the application with some initial information, the application runs (usually gathering information from a Human) and when it completes it passes information back to the Process Manager.

I've had discussions with BPM practitioners who feel that implementing Business Activities should fall outside the realm of a BPM suite. Perhaps I've been spoiled by Teamworks, but I can't imagine creating my BPD in one tool and implementing my Activities in another tool. It's just incredibly nice to be able to drill down from an Activity on a Process Diagram to the underlying implementation of that Activity.

Despite my bias toward integrated Activities, I freely admit that you must be able to implement an Activity outside the limitation of your BMP suite of choice. Some Activities do require a very sophisticated user interface - and in many cases a pre-existing application can be fairly easily adapted to perform an Activity. I've had to resort to "external" Activities on several occasions (for key Activities).

If the implementation of an Activity is beyond a simple Form that the Participant needs to fill out, then you'll have to build some sort of task focused application for the Participant to use to complete the Activity.

BPM suites such as Teamworks allow you to build rather sophisticated task focused applications - but sometimes it makes more sense to go outside the suite and build the application using JSP, ASP, .Net, Ruby - whatever tools make the most sense to get the job done.

The "problem" is the lack of standard interfaces for connecting these "external" task oriented applications with your Process Manager/Process Engine. Most of the good BPM Process Managers provide the necessary interfaces, but they are almost always proprietary interfaces.

In the early days of BPEL, Activities were "just" standard web services. An Activity (web service) that was invoked from one BPEL engine could be invoked by any other.

BPEL (by itself) could only implement processes that could be completely automated. Most Business Processes include Human Activities - so pure BPEL implementations were few and far between.

Unfortuntely we've yet to come up with a standard (or even a de-facto standard) for implementing Human Activities. Some BPM suites (like Intalio) make use of standards (like XForms) to implement the user interfaces for their Activities, but the back-end interfaces between the Process Manager and the Activities are still unique to each vendor. There's no standard for registering an "Activity Application" with a Process Manager There's no standard way for an "Activity Application" to authenticate its user with the Process Manager. There's no standard way for an "Activity Application" to query a Process Manager for Activities that belong to its user.

Until we have these standards, there's really no such thing as a "Portable Activity". I could build a framework that helped me build task oriented applications, but I'd need adapters for any BPM suite that I wanted to interface with.

Hopefully this will change soon. As more BPM suites begin to support "standard" BPDs it will become obvious that we need Portable Activities too.

All Big Brothers Aren't Bad

2009-03-21T16:17:00.018-05:00

Joel Trammell beat me to the punch with his posting: You can't manage an economy you don't measure:

"this problem, in part, was caused by lack of oversight"

I'll try to keep my outrage in check because I'm not an economist or an accountant or a lawyer. I don't really know what happened any more than anyone else (probably less than many people) but Joel's point resonates with me: If somebody had been watching and willing, then these problems could have been detected and dealt with earlier. In retrospect, it's obvious that excessive risks were taken for short term gain, and some folks were just plain crooks.

It's ironic that folks in the Financial Services industry were blind-sided by this crisis. For years they have been at the forefront of analysis and pattern recognition technology, driving many of the advances in the field. Software that could have detected Madoff's ponzi schemes is trivial compared to the sophisticated financial gaming models developed by the Quants. There's really no excuse except for a lack of someone who was willing to look out for everyone else.

At times like these we realize that sometimes a Big Brother can be a wonderful thing to have.

I'll bet that most of you think of George Orwell's Big Brother when you hear the term. Orwell's Big Brother scares the hell out of us - a omnipresent leader spying on everyone - but he's just a one dimensional character from a horror story.

We should also think of the Big Brothers Big Sisters organization... proving the point in real terms that having someone to look out for us can be a very good thing indeed.

But what about privacy?

My personal belief is that real privacy is an illusion in the 21st century. Maybe not for those living on the margins of society, but for someone like me the walls are transparent and thin. If you are curious and persistent you can probably dig up every embarrassing and questionable act that I've ever done. You can figure out where I live, where I shop, where I eat, where I travel. You can dig up former girlfriends, past employers and folks who can't stand me. My life may not be an open book - but opening my book is way too simple.

I'm not all that happy with my lack of privacy, but I am more concerned by the thought that someone can use that publicly available information to pretend to be me (identity theft). As I wrote in my 2004 blog Privately Famous:

"I know that many people fear a National Identity Card, and I can sympathize; Governments are notorious for turning oppressive. I hold the other view; I want a National Identity Card, and I want my DNA and finger-prints on file. Without a trusted identification authority (and I know that nothing is foolproof), it's just too easy for someone else to pretend to be me, and too hard for me to prove that I am me."

The price of Privacy can be way too high if it's carried to ridiculous extremes. Our fear of government prying led us to condone the gutting of regulatory oversight that we just naturally assumed was protecting us.

Forget the financial mess for the moment: Did you ever think that American Peanut Butter would be dangerous in the 21st century? Didn't Teddy Roosevelt institute food safety oversight a hundred years ago?

Regulatory oversight is certainly a potential evil - but it's a necessary evil. With the tools that we have today Big Brother can watch us, but we can also watch him. We have no real privacy, but neither does he.

Technology can help. We can help reduce the "red tape". We can help spot the suspicious. We can help predict the long term. We can help the watchers watch, and we can help watch the watchers.

We can help avoid messes like this in the future.

The Science of Art

2009-03-21T09:15:00.004-05:00

If you've looked at my LinkedIn profile you've probably noticed that I've changed jobs quite a bit over the years. There have been many reason for changing jobs... but usually I've moved on to grow. I've moved on because I found something that interested me more, or something that challenged me more. On a few lucky occasions I've found things that were both more interesting and more challenging.

On other occasions I have moved on because I just couldn't stand it any more. Frustration over some aspect of my working days would build up until it was obvious (to me) that I was wasting my time and my talents. Something about the company was perversely keeping me from doing my job in the way that I knew it really should be done (or so I felt at the time).

I think that all of us who work for a living have felt something like this from time to time, and that's why I found Douglas Bowman's post on his departure from Google to be so poinient. Bowman's artistic design philosophy crashed head first into Google's statistical mantra. Classic case of irreconcilable differences... Make a clean break and part as friends.

Changing jobs is a personal matter... but Bowman raises an issue of wider interest in the community of UI designers and programmers:

"Is UI Design Science or Is It Art?"

The answer is of course both... and everybody has always known that. What's changed is our radically increased ability to apply Science to our designs. Analysis of our designs used to be woefully limited to focus groups or beta testers, but as Stephan Shankland points out: Google's practice of "Choosing color shades and pixel widths on the basis of the behavior of millions of Web page users is a fascinating development to the form-follows-function school of design."

With SaaS (Software as a Service) solutions it's much simpler to roll out a minor design change and "see what happens". It's much easier to employ the Scientific method. It's much easier to test instead of speculate.

Take this new "try it and see" capability with a grain of salt - you'd need Google's millions of users to even notice the effect of subtle changes - but the promise is intriguing.

Bowman's case warns us that there's also a down side if Science is improperly applied. Demanding statistical justifications for artistic design decisions is fraught with peril... You'll drive off your best artists and you'll likely lose out on true innovation.

Inspiration seldom comes from statistical analysis... and inspiration is usually at the heart of all good designs. Statistical analysis can measure the success of a design, and it should be more widely applied. Statistical analysis helps us continually improve a design, but it's not the genesis of good design.

Art first - then Science.

Picking Power

2009-03-15T09:52:00.005-05:00

The Austin American Statesman has a great graphic by Roberto Villalpando in today's paper that compares the energy costs of building a new power plant - I'd love to post a link, but it doesn't seem to be on their online edition. Here's a scan:

The graphic compares the factors associated with Nuclear, Coal, Natural Gas, Wind Farm, Solar and Wood Biomass power plants. As you'd expect, it's all about tradeoffs.

I really applaud the Statesman for publishing this comparison. You might take issue with the details ascribed to each option - but this clearly makes the point that there are pros and cons to every Energy option. There are costs and savings associated with each option. Nothing is cut and dried - each option requires careful consideration. Trite slogans just don't cut it.

Power trip

2009-03-11T19:13:00.009-05:00

MIT has come up with a battery that can fully recharge in seconds.

Wow.

Talk to anybody in the power industry and they'll tell you that the problem is storage, not generation. Wind power is great. Solar cells are great. Solar furnaces are great. Unfortunately they're all undependable because the wind and the Sun aren't necessarily there when you need them.

Response to this sad fact has resulted in bizarre plans like huge Solar Power Satellites orbiting the planet to beam down uninterrupted streams of microwave energy and relatively low tech solutions like the Luddington Pumped Storage Power Plant using a huge water reservoirs and gravity to store power. Solutions like these are impressive, but they just don't work for cars.

Cars need portable power... and today that portable power is stored in gasoline. Burning gas to liberate its power isn't that efficient - so folks are looking for substitutes. The most common substitute that I have heard pitched is the Hydrogen Fuel Cell. Hydrogen was kind of scary to the Coal and Gas Industries, but then someone got the great idea of making Hydrogen from Gas and Coal. Amazing how the same folks keep making money... isn't it?

MIT's new batteries kind of throw a wrench in those plans. If you can fully recharge your car's batteries in a couple of seconds, then why lug around that Gas and Coal to Hydrogen Powered Fuel Cell? Fortunately, you can still charge that battery from the local Gas and Coal burning Power Plant :-)

But I digress... a battery that charges in seconds is an incredible development. Bill Joy identified batteries as the key stumbling block to practical electric cars:

"There's a range of new chemistries coming so that you can imagine, say five to ten years from now, instead of 100 watt-hours per liter we're at today, that a break-out company will have a 500 or thousand watt-hours--a five to 10 times (increase in) the energy density,"

Bill and the rest of us were thinking about batteries that held more power... not batteries that charge faster. Our thought was that a Chevy Volt isn't practical because you have to charge it every 40 miles (or use the gas engine). Give it a 400 mile range and folks will stand in line to buy one (and you can ditch the gas engine).

In retrospect - What were we thinking?

Would you rather have a large (400 mile) gas tank that takes 8 hours to fill or a small (40 mile) gas tank that takes 8 seconds to fill?

There are lots of factors to consider before answering that question... but I suspect that fast beats slow most of the time. If the MIT batteries really work and you can recharge your car in a few seconds, then I think that most folks will accept a battery powered car as practical even if they still have to charge it every 40 miles.

What do you think?

Is that a bad thing?

2009-03-10T07:02:00.015-05:00

Joe Trammell recently blogged an interesting anecdote that concludes with the following moral:

"The first step in trying to improve any process in IT is convincing the people with the power to make decisions that the current process is flawed."

I'd remove the words "in IT" from Joe's statement. In any organization, unless "they" agree that "it" is broken, "it" is not going to get fixed. Joe goes on to say:

"enterprise computing isn't just technical knowledge, it's also social interaction"

Amen!

In my experience, most software development obstacles are cultural. People are different. One person's crisis is another person's acceptable annoyance - I'm not going to help you fix something unless I agree with you that it's broken.

There's no quick fix for this problem, but I think that we're headed in the right direction - BPMN, Business Rules, Business Natural Languages -all attempts to foster better communication between Programmers and Business. All attempts to help each camp understand what the heck the other camp is talking about. All attempts to help each other understand what the problems really are.

We're not there yet. Communication is still a bit awkward and stifled - but we're getting there.

Seeing is Believing - Business Process Visualization

2009-02-21T15:22:00.005-06:00

Time to introduce a new TLA (Three Letter Acronym) to this blog: BPV - Business Process Visualization.

I often sum up basics of BPM (Business Process Management) with the following pitch:

Model your Process
Manage your Process
Monitor your Process
iMprove your Process

For me, the enabling factor for BPM was the Executable Process Diagram. Use tools to graphically depict the Activities and Flows of a Process, and then auto-generate the software necessary to manage the transitions between those Activities.

With any good BPM Suite I can rest assured that the Right People will Execute the Right Activities at the Right Time to keep my Processes Flowing Smoothly.

The BPM Industry has more than met the PFS (Processes Flow Smoothly) challenge - The decision for a company to adopt BPM technology and methodology is pretty much a no-brainer (Which suite to adopt may require some thought, but whether or not to adopt isn't even an issue any more).

So what's next?

In a recent post I attempted to describe a BPMS in terms that might have been familiar to any 19th Century businessman . One of the comments I received really peaked my interest:

Roeland Loggen said...
Maybe your story reflects one of the key issues with current BPM thinking: if the software provides automated concepts of a year 1890 shopfloor, then not much innovation has been created. Since 1890 many many new ideas have been created on people working in processes. The mechanistic view of typical BPM software has not clue of these more modern concepts.
The 20th century knowledge worker's reality and productivity is muh different from Taylor based process thinking... not based on orchestration, central coordination, workflow items, inbox/task screen.... but adhoc, collaboration, changing processes on the fly based on new insights and events....

For the sake of argument: If every 1890 shop floor had an exceptional Process Manager on hand, productivity would have soared... but Roland's right: We can't be satisfied to stop with systems that automate 19th century businesses. It's the 21st Century, and we can do better than that.

The key for doing better is Business Process Visualization (BPV).

In my 19th Century BPMS I discussed someone that I dubbed the "Process Visualizer". This is that person who took the raw metrics that the Process Manager had gathered and turned them into reports, charts and graphs that the Business Owner could use to understand the past and to plan for the future. These reports, charts and graphs are an integral part of BPV, but there's a whole lot more to the full picture.

BPV begins with the Process Diagram. The Process Diagram distills the essence of the problem into a form that Business and IT can share. It's a crucial part of validating that the process is correct and for communicating the requirements from the process owner to programmers who will implement the process application. In most of today's BPM Systems the life cycle of the Process Diagram often goes something like this...

The Business decides to improve their operations, so they hire an Analyst to come in and help them.

The Analyst talks to a lot of people and produces an "As Is" Business Process Diagram. This diagram is often used by the Analyst to reassure the Business that the Analyst really understands what's going on.

Now that the Analyst really understands what's going on, a new "To Be" Business Process Diagram is created. This diagram is used by the Business to capture exactly how they want the new process to operate.

At this point simulations can be run to predict how the new process will behave relative to the old. Simulation is always tricky, but if the assumptions are accurate (hopefully based on historical data), then it's sometimes possible to tune the "To Be" definition to increase the likelihood of success.

Once the Business and the Analyst are happy with the "To Be" diagram, it can be passed on to the Implementers (and it now becomes known as the "As Planned" diagram). The Implementers now morph the diagram into an executable diagram, usually adding way more details than the Business wishes to see.

Slight digression: There's a lot of controversy swirling around BPMN 2.0 over the issue of adding execution semantics to process diagrams. Implementers (in general) want truly executable diagrams but Business folks (in general) don't want to clutter the diagram with non-Business related details. I think there's a simple solution... which I will get to later.

The Implementers work with the Analyst and Business until all are happy and the process application is deployed into production... at this point the Implementer's diagram becomes known as the "As Built" diagram.

Note to all: The "As Built" diagram is a clear differentiation from all pre-BPM Systems. In a BPMS, the "As Built" really is the process that is running in production. Before BPMS the diagram may have been accurate once, but you never really had confidence that the diagram ever matched reality.

In some BPM Systems, the "As Built" diagram is often the end of the line. Once the process application is deployed they don't think about it much until the next time a process change is contemplated... However, in many BPM Systems the "As Built" morphs into a living and breathing "As Running" diagram.

Live and historical data breathes life into the "As Running" diagram. My old friend the Process Visualizer takes the raw data from the Process Manager and animates the "As Running" diagram to give the Business a clear view of what's happening where (and what happened when). The "As Running" diagram gives you your first glimpse of what BPV can achieve.

Thus far the Process Diagram has be limited to a pretty small audience - those involved in designing the process and those involved in implementing the process. BPV should expand the Process Diagrams visibility to a much wider audience.

Go into just about any office and ask a typical worker the following question:

"What Processes do you work on?"

Most likely, the response will be a blank stare.

Now ask the worker:

"What Activities do you perform?"

The blank stare will vanish and you'll get a very accurate litany of tasks.

BPV needs to give each worker the answer to that first question. It's not enough to know what Activities you need to work on - To do your best you really need to know how those Activities fit into a bigger picture. Process knowledge is a bigger picture, and it goes a long way to helping the workers understand why they are doing what they are doing.

A task that seem stupid in isolation may make sense when seen in a process context - or it may really be stupid. In the former case the worker may do a better job, and in the latter case BPV may give the worker the context needed to communicate their frustration.

When folks such as Lombardi's Phil Gilbert talk about BPM on every desk in an enterprise, this is what they are talking about. Process Visualization helps everyone put things in context, and that helps everyone do a better job.

Now for a bit of Devil's advocate: Most workers couldn't make heads or tails out of a BMPN diagram, and if you add all the execution semantics proposed for BPMN 2.0 that percentage is going to drop even further.

Granted, but there's a solution: Different views for different folks.

Returning to my programming roots... I am a huge fan of Aspect Oriented Programming.

Aspect Oriented Programming evolved from a simple fact: Most worthwhile endeavors can be broken down into distinct concerns, and often you can meet the goals sooner by dealing with these concerns (aspects) in isolation. Obviously there are limits to this when concerns overlap, but it's a good place to start.

With Aspects as my inspiration it's easy to see where Process Diagrams need to go: Aspect Oriented Diagrams. If we can identify the aspects of a diagram that the Business Owner is interested in, and those that the Operations Folks are interested in, and those that each Worker is interested in, then we can generate the Views that mean the most to the intended audience. It's all just a matter of Perspective, and BPV enables many, many perspectives. Some views may look like process diagrams, others may look quite different - doing BPV right will employ a lot of artistic scientists.

Here are just a few process views that come to mind:

Process Owner's "40,000 foot" view
Activity Owner's "Escalation logic" view
Participant's "Where does this fit?" view
Operation's "What's happening now?" view
Operation's "What happened?" view
IT's "Capacity planning" view

As Roland said, the future is about "ad hoc, collaboration, changing processes on the fly based on new insights and events"...

BPV's going to help us find those insights.

Grilling Business

2009-02-18T07:29:00.004-06:00

Mike Hugos offers some interesting thoughts in his article: How To Truly Partner With Business.

Mike points out that the traditional requirements gathering process is:

"about as friendly as being grilled by a police detective"

Not exactly the relationship we've all dreamed of, is it?

== != =

2009-02-11T21:51:00.004-06:00

This week I visited a client in the Detroit area, and on the whiteboard in their meeting room I saw the following:

== != =

For those of you who aren't programmers, this is a slam at perhaps one of the worst programming language syntax decisions that has ever been made. You can read the above expression as "Equal equal is not equal to equals". Still lost?

In the C programming language (and derivatives such as C++, Java, JavaScript and C#) the way that you test to see if two variables are equivalent to each other is to use the "==" operator. For example:

if ( a == b ) { doSomething(); }

If you want to assign a value to a a variable you use the assignment operator "=". For example:

a = b;

Believe it or not, even folks who have been programming for years occasionally screw up and use the assignment operator "=" when they meant to use the test for equivalence operator ==. Here's an all too common example:

if ( a = b ) { doSomething;}

The line above sets the value of variable "a" to the value of variable "b". If "b" is not zero, then the function "doSomething()" will be invoked. Perhaps that's the behavior that the programmer desired, but I really doubt it.

According to the wikipedia article comparing the Pascal programming language to C:

"It is common for programmers new to C to accidentally put assignment expressions in conditional statements such as if (a = 10) { ... }, which will always execute because the assignment expression a = 10 has the value 10 ≠ 0 and is therefore considered "true" in C. Also note that a has now got the value 10, which may affect the following code. This kind of mistake cannot happen in Pascal, as assignments are not expressions; using the wrong operator will cause a compile error (and it's not likely that anyone would mistake the := symbol for equality test)."

I have to take issue with this statement... It's not just programmers who are new to C. I made this mistake last week :-)

This mistake is so common that some programming editors will actually flag statements like this to warn the programmer. I've no idea how much time has been lost on this little syntactical monkey wrench, but I think it's probably quite a bit.

Long ago before I learned C I used Pascal. In Pascal the "=" operator is used to test for equality. If you want to set a value you used ":=". When reading this symbol out loud I'd always say "set equal to". For some reason, I don't ever remember confusing the two or knowing anyone else who ever confused the two.

The BASIC programming language is even simpler... the interpreter uses the context of where "=" appears in an expression to determine what it means. IF $A = 0 THEN is interpreted as a comparison. $A = 3 is interpreted as an assignment.

I don't know why the authors of C chose to use "==" and "=" in this way - but unfortunately that decision was inherited by most of the programming languages that are widely used today. Every programmer who includes some JavaScript on a web page might screw up and make this mistake... so just remember:

== != =

19th Century BPMS

2009-02-06T13:28:00.006-06:00

I sometime find it useful to describe a BPMS (Business Process Management System) in terms of things and people that you probably would have found in an office or factory in the 1890s:

At the center of the BPM system is a Process Manager who makes sure that each Process runs smoothly. The Process Manager uses Process Definitions to know what all the steps of each process are, the proper sequence in which each step should be performed and which people should perform each step.

The Process Manager hands out assignments to each worker - updating each worker's Task List to insure that each worker knows what to do and when to do it. The Process Manager also passes along any information from previous Tasks that the workers doing the remaining Tasks may need.

If a worker doesn't finish a Task on schedule, the Process Manager may escalate the Task. Escalation could take many forms - the Task might be reassigned or perhaps the worker's supervisor might be informed - It's all laid out in the Process Definition.

A great Process Manager can manage many instances of many different types of processes simultaneously (that was probably rare in the 19th century).

The workers do the work (just like they did in the 1890s) performing whatever activities are assigned to them by the Process Manager. When a worker finishes a Task the Process Manager must be informed.

Workers in the 1890s were given detailed instructions that told them exactly what to do to accomplish their current Task along with Forms to record any information that the Process Manager or other workers needed. In today's world the workers rely on software to guide them through each task (often in the form wizards). Most BPMS systems run these wizards using Task Engines that can execute any Task Definition, but often custom applications are developed or adapted to perform specific Tasks.

The Process Managers primary job is to keep every process running... But that's not enough.... Are the processes efficient at accomplishing their goals?

Workers want to know how they are doing relative to their peers.
Bosses want to know how well their workers are doing and whether or not the company's goals and obligations are being met.
Owners want to know how today's performance compares with that of last year... and how next year's perfomance might be better.

Everyone needs to be able to Visualize their processes in meaningful ways that can help answer their questions.

The Process Manager records all of the raw data necessary to track each process instance, but often relies on someone else to crunch the numbers and produce the meaningful reports, charts and graphs that help make sense of the processes. I call this someone else the Process Visualizer for lack of a better title. I'm sure there were people in 19th century offices who performed a similar function.

Ultimately it's the Process Visualizer that sets a BPM Sytem apart from a Workflow System. Without visibility into a process you can only guess on what changes might improve it. BPM is about enabling Continuous Process Improvement - so that Visualizer is key.

Their are many vendors of BPM Systems in the world, and many might take issue with some aspects of my analogies - but I think that most will agree with my broad characterizations.

BPM Systems are made up of relatively mundane parts that would sound familiar to most 19th century Businessmen - the concepts are old - it's our technology (and people who know how to use the technology) that is making these BPM concepts transformational.

Just imagine what could have happened in the 1890s if there had been a great Process Manager and Process Visualizer in every office and factory.