Blue Screen Joy - A Software Tester's Journey

The Lucky Thirteen

2008-09-24T17:05:00.002-04:00

Harry McCracken has done a great post on The Thirteen Greatest Error Messages of All Time. This baker's dozen rouge gallery should bring joy to any tester's heart.

Who Tests the Tester?

2008-09-08T14:55:00.003-04:00

Finding the right tester to join your team is tricky business. For one, software testing is not something many people have degrees in. Instead it is something most people learn on the job, which is a bit of a chicken and egg situation in of itself. Secondly, it seems many people only visit the testing profession rather than making a career of it, so the people with the right experience might not be interested in the job.

One thing we do we selecting candidates for an open Test Engineer position is to go give a quiz. In its current form it is only six questions that we give people up to an hour to answer. This provides a good level-set for what kind of tester a person is likely to be. I won't share the exact questions here in case some savvy applicant finds this blog. I will however give this much info:

Some of the questions are about what test cases would you write to test some made up app
Some of the questions deal with when it is best to use different styles of testing
One of the questions is a trick question - it asks about how you can tell when you've found all the bugs in an app. No one has gotten this one right yet.

The reason I am thinking about this now is because we are trying to find the right tester to fill a new position at my company, AdaptiveBlue. If you might be the one, or know someone who might be, please let me know.

To Find the Unfindable Bug

2008-04-09T09:31:00.005-04:00

This morning I have been tasked with reproducing an unreproducible bug.

Since the last update of our software, we've heard from a very small number of new users who were unable o create an account. This is a big deal for this small percentage of people. It is possible that the number of people that are reporting this problem are a fraction of the people who are experiencing it, since many people will just drop-off if they can't create an account, but even so, we also know that most people don't see this problem. We've also not seen this problem during extensive testing.

Except that we have seen this problem in testing. A few times in dozens and dozens of attempts. Each time, the exact steps were recreated, and it worked fine the next time. Its what we in the testing business call "a fluke".

But, back to the business at hand - how can I solidly recreate this most elusive of bugs? The first thing to do is to find out all we can from the people that experience the bug. Using our best detective skills, we should try to determine how the people who see the bug are different from those who don't. It could be something about about how they are interacting with our product or something about their environment (physical resources, other software, even their physical location). If you can isolate what the difference is, it should be easy to reproduce.

If you can't isolate the difference, then it is time to bring in your testing skills. Two approaches you can try:

"The Crazy Uncle" - Test unusual user flows. Since most people don't see the bug, perhaps we need to try something weird and unexpected to make it occur. Never underestimate a user's ability to use your software in a way that is totally unimaginable.
"The Big Snafu" - Make what can go wrong, go wrong. This means limiting or disabling your systems resources while going through the use case. This could include such things as unplugging from the network half way through, removing the install CD at the wrong time, or devoting but a sliver of your system's RAM to some application other than what you are testing.
"Brute Force" - Test the usual flow many many many times. It could be what is happening is some sort of race condition that will only occur 1 out of 100 times. This approach is extremely time consuming and boring (and therefore a good candidate for automation), but if you can make it fail when everything else is normal, that is important information that the developer can use to fix it.

If after all this you still can't reproduce the bug, then your choices are to keep trying, live with it, or hope the developers can find it by scrutinizing and carefully reviewing the code. None of this is especially appealing, but no one promised you a pleasure cruise.

Does anyone else have suggestions on ways to find the unfindable bug? Please add them to the comments if you do.

One of Those Days

2008-03-26T22:56:00.003-04:00

Some days you get in a zone where the bugs come out of hiding to follow you like you were the Pied Piper of Defects. For me, that describes exactly how this day wasn't.

One thing I was testing involved passing strings of text. Of course all the basic, simple text worked fine. I also tested such odd ball characters like brackets and quotes and other punctuation. I even tested non-English characters including € and the like, and even some Japanese characters. Everything I tried worked perfectly, so I declared it good to go.

My boss then decided to give it a try. He happened to choose some text including a $. Of course, it failed miserably.

Later I was testing a new widget. I tested in on FF on Windows, and found some issues. These were fixed. I tested it on IE7, found some issues, and these were fixed. I tested it on FF3 Beta on Mac, and it worked there. It was good to go. Or would have been, had it not been on Safari. Guess who tried on Safari? That's right - my boss.

Sigh.

Some days you get the bugs. Some days the bugs get you.

Pay Back Bad Developers

2007-12-21T17:10:00.000-05:00

Here we are, in the thick of the holiday season when we give gifts to the important people in our lives. When doing your Christmas shopping, you should be careful not to overlook the founders of the feast - the Bad Developers.

I say this not because developers write the code that gets sold to your users, thus paying the bills. I say this because without Bad Developers, and the wonderful bugs they tuck away in the product like lumps of coal in a stocking, there would be no need for testers.

That is why I recommend that testers buy a gift for the worst Developers on the team. I say the worst, because it is these venerable engineers that make necessary most of the testing we do. You don't need to reward the good developers - if they are not going to send some wonderful bugs your way, you don't need to get them a thing. So Karen, Jeff, Dominick, and Alex - no gifts for you because you make me work too hard to find those precious few bugs.

But what is the best gift to show how much you appreciate the Bad Developers and their lack of best practices, unit testing, common sense, and skill than a fruit-cake. Nothing says "Thanks for the lousy code! Keep up the bad work!" like a cake made of fruit. This classic no-prize is just what every Bad Developer needs and wants.

Sharpen the Flaw

2007-12-19T16:36:00.000-05:00

The Seven Habits of Highly Effective Testers
Part 7 of 7

This is finally the seventh post in a series many thought would never be finished. It is based ever-so loosely on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.

Like college freshmen, bugs rarely live alone. A group of bugs living together is called a pod, or maybe that is dolphins. Swarm? Colony? The best name would have to be the same as crows - a murder of bugs. Yes, I realize I am getting off track. Sorry.

The point is, when you find one bug, you can be sure that there are other bugs nearby. As a tester, you should exploit this by banging on the features in which you find the bug. Like pulling on a loose thread, testing heavily around a new bug can lead to the whole program unravelling.

You should also test similar functions in other features. Developers are at least as lazy as normal people, so when they can reuse a bit of nifty code, they will, and thus make another instance of the bug. The right way for developers to do this is to abstract the function and then call it from everywhere that needs it, but as we know, the right way to do things is not always the way things are done. And thank goodness for that.

Previous Posts:
1 - Be Protractive
2 - Begin With the Vend in Mind
3 - Put Worst Things First
4 - Think When/When
5- Seek First to Understand, Then to Exploit
6 - Sinner Guise

Sinner Guise

2007-12-04T15:32:00.002-05:00

The Seven Habits of Highly Effective Testers
Part 6 of 7

This is the sixth post in a recently neglected series which is based ever-so loosely on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.

If Moses was alive today he would be a technical writer. Just like the ten commandments, the purpose of software documentation is to tell people what they should do and should not do. Also like the ten commandments, much software documentation is ignored.

Software is designed to be used in a certain way. Careful use cases and flows are thought out and implemented by designers and developers. Testers test these happy flows and make sure everything works the way it should. All is well until a user gets the software and tries to do something with it that no one ever imagined. What happens next is crucial.

A sign of quality software is that it either handles the abuse, or at worst, fails gracefully. It is our job as testers to make sure that this happens. To do this, part of your testing should include ignoring everything you know about what a user Shalt Do or Shalt Not Do, and go crazy:

Put negative numbers into text fields
Put ridiculously long strings anywhere you can,
Paste inappropriate stuff - like binary files into text boxes
Open 13 instances of your app at once
Unplug the network or the power in the middle of an important flow
Change the system date and time
Install on a system way below the minimum specs
Use weird characters like ì È Ö
Run 10 other apps while you are testing
Delete crucial files
Don't close other applications before proceeding

In other words - be great and terrible in the way you treat the software. Put on the guise of a sinner and break all the commandments. The developers may not forgive you for doing this, but your users won't forgive if you don't.

Previous Posts:
1 - Be Protractive
2 - Begin With the Vend in Mind
3 - Put Worst Things First
4 - Think When/When
5- Seek First to Understand, Then to Exploit

Next Up:
7- Sharpen the Flaw

Firefox Beta Goes To 11 (Thousand)

2007-11-21T09:48:00.000-05:00

When I got the heads-up from Josh and Duncan that the beta of Firefox 3 was now available, what put a twinkle in my eye was that Mozilla has fixed 11,000 issues with this release. I could not believe it, but here is the info directly from the source:

Firefox 3 Beta 1 is based on the new Gecko 1.9 Web rendering platform, which has been under development for the past 27 months and includes nearly 2 million lines of code changes, fixing more than 11,000 issues.

My first reaction was "How great and wonderful Mozilla is for giving us such a vastly improved product!" This was followed 2 seconds later by "Yikes! My most used application and the product my company has hitched our wagon to has at least 11,000 known issues!"

My next thought was "How great and wonderful the Mozilla test team must be for finding that many bugs and driving them to resolution." I doubt I've found anywhere near 11,000 bugs in my career, and I've been a software tester since long it was cool and hip. Even if these issues have been accumulating over the last 27 months, that is more than 10 bugs a day - and that is just the ones they fixed.

I then came to the conclusion that Mozilla actually has hundreds of millions of the best kind of testers - free ones - aka users. That is the only way they could find, track, and fix 11,000 bugs - with a strong and active community. So, my final thought was "How great and wonderful the Mozilla community is for finding and reporting all these bugs!" Mozilla should buy them 11,000 beers.

A Touch of Quality

2007-11-15T22:22:00.000-05:00

I am not, by any means, an Apple fanboy. For most of my life I've been quite the opposite. A couple of years ago, I got a first generation iPod Nano, and thought it was fairly decent. Last year I got a MacBook and it has slowly become my laptop of choice - especially when I travel. When the iPhone came out, I was intrigued, even more so after getting to play with one. Since I am boycotting Cingular however, the iPhone would have to remain something I would admire from a distance.

So, you can imagine my interest when Mr. Steve announced the iPod Touch which had many of best parts of the iPhone but with no phone, and therefore no Cingular (now AT&T) contract. I resisted temptation for what seemed like a long time, but a few weeks ago I gave in and bought a Touch. I am heavy into gadgetry, so it really means something when I say that the iPod Touch is the

Best Gadget Ever.

This is a blog about quality, and I am here to tell you that the Touch is chock full of quality, both in the hardware and in the software. Lets start with the hardware. The thing is crazy thin. Even when I was 6'3" and 140 pounds (a long time ago) I was nowhere near this thin. Even so it feels solid and sturdy. The screen is plenty big and bright, and easy to read. The wi-fi is strong. Even though there are only 2 buttons, this is actually just enough, and I have yet to find myself wishing for more.

As good as the hardware is, the software is even better. Whether you are making your way through music, videos, pictures, or real web pages the navigation is thoughtful and intuitive. Even entering text is not that bad. The software is also very solid - and this from a guy who makes his living breaking software.

That is not to say the iPod Touch could not be improved. Some changes I'd like to see:

A speaker - This thing begs to shown off and passed around, and so it would be real nice to have a speaker to play the music and video soundtracks so your friends can enjoy it without getting goo on your ear buds.

Some games - These will be coming after Apple opens up the platform next year, but it would nice if they included some simple time killers from the get-go

Attached mode - This is a bit unusual, but stay with me. Most of my iTunes listening occurs on my computer - the screen for which is taken up with stuff other than iTunes. Wouldn't it be cool if I could plug in my Touch, then use it as dedicated interface to control iTunes on my computer?

Good work Apple. The Touch is so good that I may actually break my Cingular boycott and get myself an iPhone one of these days - though I am going to *try* to wait until the next generation which I *hope* will be announced in January.

Seek First to Understand, Then to Exploit

2007-11-14T17:32:00.000-05:00

The Seven Habits of Highly Effective Testers
Part 5 of 7

This is the fifth post in a series which is based ever-so loosely on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.

Customers don't have inside knowledge of how an app is designed, architected, or coded. That is why I've always been a big fan of black-box testing - it best approaches the way a customer will use the app. Still, there is something to be said for glass-box testing as well. Looking inside the box can give you insight into where the good bugs are.

Sitting in on developer meetings is one good way to gain some understanding about how the code is built. When there is disagreement about how something should be designed or coded, that means that the features being discussed will likely be target rich environments once they are ready for test. The more heated the argument about a feature, the more you should plan to test it. As an added bonus - after an hour in a developer meeting, you'll be even more glad that you are not a developer.

Unit test tools can also provide you some insight. If your developers are running these tools themselves, ask if you can take a look at the reports and logs. If they are not running any unit tests (which is another issue), look into running these tools yourself. Even if you don't totally understand the results, it is a pretty safe bet that areas with a higher density of findings will be more likely to have bugs, and therefore, more worthy of your testing time.

One other important thing you should know about the application you're testing is which developers were responsible for which parts. Like with any profession, there are good developers and not-so-good developers. By knowing what the weaker developers coded, you know where to focus your test energy. Likewise, you don't have to spend as much time testing stuff written by the more competent programmers.

By taking a little peek under the hood and getting and understanding of how the app is written, you can better exploit weak areas and find more bugs.

Previous Posts:
1 - Be Protractive
2 - Begin With the Vend in Mind
3 - Put Worst Things First
4 - Think When/When

Next Up:
6- Sinner Guise

Think When/When

2007-11-13T18:56:00.000-05:00

The Seven Habits of Highly Effective Testers
Part 4 of 7

This is the fourth post in a series which is based ever-so loosely on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.

This post is about defect triage. I first learned abut triage from watching M*A*S*H. When the choppers and ambulances would bring in the wounded, the doctors and nurses would assess the men and decide who needed to get surgical attention first. These decisions would be based on the severity of the wounds as well as what part of the body the was injured. So a massive leg wound might not get precedence over a moderate heart wound.

When deciding when defects should be fixed it should work the same way, but it often does not. That is because of the confusion between severity and priority. Generally severity is measured on a scale from Sev1 - Crash the system to SevX - No harm to foul.

The problem comes when severity is the only factor that is considered. In such a situation, a bug that crashes the system but only occurs through a very unlikely usage scenario will get fixed before a more moderate problem that happens in a common user flow. Also, important bugs that are not especially severe will be deferred.

This is just wrong.

A better way that more advanced teams use is to assign a priority to each defects. To determine this priority, many factors should be considered. First and foremost, is severity, which can be assessed by the person reporting the defect. Also to be included is the likeliness of the defect being encountered by users. This can be determined by the support team with perhaps some input from the marketing folks. Another factor is the current workload of the specific development resources needed to fix the problem. This system therefore requires stakeholders from across the project to be most effective.

When these other factors besides just severity are considered, you will come up with a much better answer to the question of "When should this defect be fixed?"

Previous Posts:
1 - Be Protractive
2 - Begin With the Vend in Mind
3 - Put Worst Things First

Next Up:
5- Seek First to Understand, Then to Exploit

Put Worst Things First

2007-11-12T18:05:00.000-05:00

The Seven Habits of Highly Effective Testers
Part 3 of 7

This is the third post in a series which is based ever-so loosely on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.

Lets face it, you usually don't have enough time to test everything that needs testing as thoroughly as it should be tested. That means that you will need to prioritize your testing to be effective. But what is the best way to do this?

A simple way to approach test prioritization is to use the one fifth method - where the test team consumes a fifth of whisky before test planning. Just kidding (as far as you know). The one fifth method involves asking yourself what you would do if the software needed to ship in one fifth the time you really have. For example, if you are supposed to have five weeks to test, what would you test if you only had one week? If you are supposed to have a week, what would you test if you only had a day?

Approaching the problem of prioritization in this manner will quickly lead you to the target-rich areas where you are likely to quickly find many bugs. By testing these first, you give the developers something to keep them busy, and also help ensure you'll have time to retest fixes.

Previous Posts:
1 - Be Protractive
2 - Begin With the Vend in Mind

Next Up:
4- Think When/When

Begin With the Vend in Mind

2007-10-08T22:53:00.000-04:00

The Seven Habits of Highly Effective Testers
Part 2 of 7

This is the second post in a series based loosely on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.

A long time ago in another life I was listening to an audio tape of motivational marketing guru Zig Zeigler. In this tape, Zig said "Nothing happens until somebody sells something." Though I don't totally agree with Zig on this, he does have a point. He also has a really cool name, but I digress.

If you are going to get paid to test software, somewhere along the line, someone has to pony up some money. Traditionally, this has been the customer. For the customer to make the decision that your software is worth their money, there are at least two things that she needs to believe:

1) The software should work
2) The software should solve some problem for her or fulfil some need

As testers, we tend to focus on #1. We get specs (when we're lucky) then we design and execute tests to determine if the software does what the specs say it should. The origins of these specs will be different for different companies, but usually they are written by some combination of marketing, sales, and development. But whos to say if the specs are right and if the resulting software will be what the customer really needs?

You are.

Before your first test case is even a twinkle in your eye, you should be reviewing the specs, and making some noise if you don't think they are right. Sales and marketing might know the customer best, and development might know the inner workings of the code the best, but it is the test team that is the expert at the intersection of the two. Testers know how to use the software better than anyone, so we have insights into the the design, the features, and the UI that are crucial.

Your company can produce solid software with nary a bug, but if it is not what the customer needs, it will not be sold, and as Zig reminds us, that is when everything starts to fall apart. This is a BAD THING that you can help prevent with carefully and critically reviewing any and all specs, and always asking yourself "Is this something the customer will want to buy?"

Previous Posts:
1 - Be protractive

Next Up:
3- Put Worst Things First

Be Protractive

2007-10-03T16:50:00.000-04:00

The Seven Habits of Highly Effective Testers
Part 1 of 7

This is the first post in a series based on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.

One of the interesting challenges of software testing is caused by the fact that testing usually happens at the very end of the release cycle. This is a problem when all the milestones before testing are pushed back while the milestones after testing are unchanged. For example, the test-handoff is three weeks late but the GA date remains the same. That's called the squeeze, and it means that testers rarely have enough time to test as thoroughly as we should.

What can be done? You could try to push on the developers so that the software is delivered to test on the scheduled date. You could also try push on the executive's for a large corner office and a company Ferrari, with about as much luck. For many reasons, legitimate and otherwise, development is going to take longer than planned - so learn to live with it.

The other option is to be to be protractive and push back the GA dates as necessary to give your test team the time it needs. This will not make you popular because nobody likes to miss the GA date. If you want to be popular, I'll suggest to you that Software Test might not be best profession to pursue. Instead, you need to be firm. As bad as it is to be late, it is much worse to be on-time and buggy

One company I worked for had a policy that there would be a day-to-day slip of the GA date for every day the software was late into test. This policy showed a commitment to quality and a strong belief in the importance of test. That is not to say that as testers we can't or shouldn't work extra hard to make up some of the slip. If we can dothat, everyone wins because the software is less late, less buggy, and the testers are the heroes.

So, be efficient but thorough and make sure you take the time you need to test the software as much as you know it needs to be tested.

Next Up - Habit 2: Begin With the Vend in Mind

That is not to say tha

Celebrity Bug Watch

2007-09-27T13:43:00.000-04:00

If you are a bug and you manage to escape the erstwhile testers, it must be a dream come true to become famous. What usually happens is that these bugs with dreams of becoming celebrities wind up becoming waiters or waitresses, or at best part-time models, while waiting for their "big break" that never comes. Every once in a while however, a bug gets the attention of the critics and the press and become famous.

That is happening now with the Excel 100000 Bug, or E100k as it is called among friends. This bug's talent is that if you multiply 77.1*850, Excel will display 100,000 instead of 65,535 as it should. I don't know about you, but it happens to me all the time. Joel Spolsky offers an explanation of why this bug occurs, in case you care.

I'm glad I am not the tester that let this bug get by. Given the press that this thing is getting, I'm sure the Microsoft execs are getting all in a dither, and are in turn getting the upper management stirred up, who are taking it out on middle management, who must be going after the 1st line manager of the Excel Multiplication QA Team. That means this manager will ask this poor tester "Why did you not find this bug?"

I hate that question because there is no good answer to it. There are lots of good reasons why bugs get missed, but when you have upper-management involved, every valid reason sounds like an excuse. What can this tester say ?

"I tested 77*850 and 77.2*850, and since they both worked, I thought it was good to go."

"I didn't know there would be any math involved"

"You mean 77.1*850 is not 100,000?"

"I assumed no one would want to calculate that product"

"Clippy told me it was OK"

So whatever the reasons for this bug getting out, and the consequences, let me just say Congratulations to E100K on making the big time.

Testing Outside The Black Box

2007-09-21T08:14:00.000-04:00

Much of what we focus on as quality engineers is determining the quality of a given application. Software testers test software, but software is not all that software companies produce. Companies also produce web sites, demo applications, marketing material, and more. Who is responsible for the quality of this stuff?

Hopefully the teams that create these things have systems in place to ensure their quality. Realistically, you might as we that all bugs will fix themselves, the company stock will triple every year, and there will be pizza and chocolate cake every day. It could happen, but don't count on it.

As an example, my company just launched a book widget gallery. The purpose of this blog is to showcase our SmartLink widgets for books by a set of authors. The hope is that these authors and their fans will see these widgets, love them, and put them on their own sites. It uses our technology, but it is not one of our products. It would be an easy thing for test team (aka me) to say, "Well, that is a marketing thing, so we don't need to worry about it." The problem is, it is the testers that have the mad bug finding skills, so who better to do testing, even of stuff that is outside our "responsibilities". As such, I went ahead and did some testing on the gallery, and continue to keep my eyes on it.

So, ask yourself, where else in your company can things go wrong? For everywhere you identify, then ask yourself what you can do about it. This may mean venturing into areas that testers have never been, but any wise person in your company will appreciate your insights into how to make things better. Think of yourself as a missionary of quality.

When Quality Goes...

2007-09-18T10:39:00.000-04:00

My current all-time favorite hobby is photography. Being a Web 2.0 guy, a large portion of the time I spend on this hobby and the enjoyment I get from it comes from sharing my pictures online as part of a community. For the past year or so, my photo sharing service of choice has been Zooomr. Last spring, Zooomr released Mark III, and said it would change the world. It was a very bumpy launch, the site was down for many days, and even now, many of the features we had in Mark II have yet to resurface.

Yet, I still stay with Zooomr, and so do many others. For every person however, there is a line past which they can't take it anymore, and will leave. Raoul Pop, a great photographer, longtime Zooomr advocate, and one of my "friends" on Zooomr has had nearly enough of the missing features and serious bugs that get promises instead of fixes, and that is why he is taking a break from Zooomr. If you read the comments of his post, you'll see that many other users are like minded, and unless something changes, Zooomr could be in trouble.

This blog is about finding bugs, but as Zooomr shows us, just knowing about the bugs is not enough. As testers, we can't just find a nice stack of bugs, then put our feet up on the desk, smoke a big cigar, and congratulate ourselves on a job well done. Instead, we have to make sure that these bugs get fixed.

What if there are no dedicated testers? In the case of Zooomr, the job of testing mostly falls to the user, since they have no QA staff, as far as I know. When quality is important, the tester serves as the first customer. When the customer serves as the first tester, there is a problem. A loyal user might tolerate bugs and missing features, report them, and even fight for the fixes. But most users aren't that loyal, and will instead just leave quitely.

When quality goes, so go the users. When the users go, so goes the company.

The point here is not to single out Zooomr as a company in trouble. Unlike Raoul, I'm not ready to take a break from Zooomr. At least not yet. Intead, this post should be read as a reminder, not just for Zooomr, but for all software companies, that the cost of not fixing bugs is usually much higher than the cost of fixing them.

Cache Test Dummy

2007-08-23T10:36:00.000-04:00

Money may be the root of all evil, but it is cache that causes the biggest headaches when you are testing an app with any web based components.

Life in the fast lane of web development usually includes rapid cycles of development, deployment, and test. Cache is the monkey-wrench to this fine tuned machine because of the mysterious way that the browser locally stores files that would otherwise be served fresh off the server

I don't know how many times I've found a really good bug and excitedly reported it to the erstwhile developer only to have them say "Hmm, that's odd. Doesn't happen for me. Try clearing your cache." More often than not, this makes the bug go away, and makes me want to shoot myself.

The real problem is, testers are not the only ones that have cache. Users do too. And users don't tend to feel the joy of bugs likes testers do. What can be done? For one, you can ask your users to clear their cache. We just did that on our blog to help ensure our users and ourselves that our product is not to blame for issues some people are seeing in Firefox.

That is not the most elegant of solutions. It is better to find and fix these issues before a user ever sees them. So, when you find a bug that is caused by something being cached or not cached as expected, clearing your cache to make the bug go away is not enough. You need to go the next step of figuring out in what scenarios caching is causing trouble, then working with development to make them understand the situation, and improving the code to deal with it.

Because of the nebulous nature of how browsers handle cache, tackling these problems can be like driving into a wall at 65 MPH, but such is our role as testers. Better us than the users.

The Seen and Unseen

2007-08-10T09:50:00.000-04:00

I found the kind of bug this morning that I really enjoy. Well, it would be more accurate to say I missed the kind of bug I really enjoy, but found it eventually.

Yesterday we updated some of the images in our product. These images reside on our server, and are dynamically loaded when a user opens the options page in their browser. When the new images were put on the server, I made sure they were right in both the released version (since those users will see them too) and in the upcoming version that I am currently testing. Good job testing by me, right?

Wrong.

Where I did not test them was on a Mac. This was a simple change and I did not think it was necessary, especially given all the other stuff that I was testing so we could finish this release on time. My mistake. I got a midnight e-mail from my boss that there was a small white space on the bottom of two of the buttons on the Mac:

OK, not a huge deal, but something we wanted to fix right away since all our current users (who use Macs) could see this. This is where the story gets good. New graphics were created and uploaded to the server this morning. Both my boss and I checked them on Windows and Mac, and the white space was definitely gone. I gave a big thumbs-up to the change. Good job testing by me, right?

Wrong.

About ten minutes later, in the course of testing something else, I realized that in fixing the whitespace problem, somehow the names of the images got switched for the first two buttons. In other words, the first button looked like the second button should, and vice versa, and so neither one did what it said it would. This was a much worse problem, and not just because it effected Windows users too.

What are the lesson to learn from this? I'll save the testing on Mac vs PC lessons for another post (or ten). I'm also going to save the lesson about how fixing minor bugs often creates major bugs. Today's lesson is about how a software test professional such as myself (not to mention my boss) could miss the blatant fact that the buttons were totally wrong when we were looking right at them. It was because of selective attention - we were focused on whether or not there was a little white space under the buttons. In software testing, focus is a fickle mistress. You can just as easily have too much as you can not enough. That is why I like exploratory testing. By not looking for just one specific thing, you tend to see a lot more. But that is also the topic for another post.

God Speed, NASA Testers

2007-07-26T22:59:00.000-04:00

In everyday life, computers fail all the time in all kinds of wonderful and exciting ways. Thank goodness for this, because it keeps us testers employed. Like I always say, within these failures lies joy, at least to the destructive mind of the test professional.

That is, unless you work somewhere like NASA. When lives are on the line, I'd think that testers would feel relief at finding some nasty bugs, along with some fear about "What if we had missed this one?" or worse, "What if there is something else we missed?" That is why I have such respect for NASA testers.

That is also why I have to scoff at the person who sabotaged a computer that was about to travel on the space shuttle Endeavor to be installed in the space station. He did this by cutting a bunch of wires. What was this guy thinking? Did he really think this not-so subtle "bug" would get past the testers of NASA? We've all missed a bug or two in our days, but I mean, how poor a tester would you have to be to let this one get by you?

The only way this could happen would be if the testers went out drinking with the astronauts before running their tests. Two things about that. For one, if I was strapped into a space craft attached to giant tanks of highly explosive rocket fuel, all built by the lowest bidder, I might want to have a drink or three as well. Second, sometimes I find that having a drink before testing actually helps (six-pack test factor).

So, lets not worry too much about the astronauts knocking down a few the night before the launch. And lets really not worry at all about disgruntled contractors sabotaging the space shuttle, because we can be confident that the testers of NASA will find out. What NASA should really worry about is that their contractors are hiring people so stupid that this was the best idea they could come up with for sabotage.

What's Behind Your Product?

2007-07-23T22:37:00.000-04:00

There is a lesson in photography that teaches you when composing a shot, to not just pay attention to your subject, but also check the rest of the frame to make sure there is nothing lurking in the background that should not be. This will help you keep trees from growing out of people's heads, and other distractions that would make your photography less compelling.

One of the main products that I currently work on is BlueOrganizer, which is a Firefox add-on for smart browsing. Given that we run in Firefox, we are very much dependent on having a solid Firefox behind us as a platform. Last week Firefox had a minor update, 2.0.0.5. Well, it was minor for them, but it totally broke some important functionality in our product. Luckily we happened to realize this quickly, and were able to get a ~~hack~~ patch out to work-around the problem.

This reminded me of that photography lesson. It does not matter if your product runs in a browser, native to an OS, or on a device. As testers we tend to focus on testing our product, and ignore the foundation it runs on. We focus on our own stuff and don't think about the potential problems behind it, often until it is too late.

For us it was lucky we found the problem on the same day Firefox started to roll-out their update. It would have been better had we found it the day or even the week before. This would have saved our users some frustration and saved us from yet another fire-drill. What I should have done, and will now do in the future is to keep track of what the folks at Mozilla are up to, and work out some way to try their new releases before they are released.

What does your product rely on, and how can you keep it on the radar?

Harry Potter - Software Tester

2007-07-17T09:48:00.000-04:00

With the release of Harry Potter and the Deathly Hallows, the seventh and final book in the series, still a few days off, we still don't know what will become of Harry. Will he defeat Voldemort? Will he live or die? Will he graduate from Hogwarts and become a software tester?

OK, so maybe we can make a good guess about that last question and say that Harry Potter will probably not become a software tester. But what if he did? How would his life experiences prepare him for this noble profession.

Dealing With Bullies - During all the time he lived with the Dursleys, Harry had a lot of experience with being bullied. He eventually learned to stand up for himself. This is a key trait of an effective software tester. When the deadline looms and the pressure mounts, a software tester needs to stand up to anyone who tries to ship before the software is ready.

Exploration - Whether it is sneaking through secret passages to Hogsmeade, or playing with mysterious devices in Dumbledor's office, Harry has always been one to let his curiosity lead him to try new things. If he used this part of his nature to do exploratory testing, he would likely find a good deal of bugs that lie off the beaten path.

Training - Sure, Harry has lots of magical training, but I don't recall him ever taking a class on quality engineering, test automation, or dealing with developers (the last one might have been covered in Defense Against the Dark Arts). So from a training stand-point, Harry might not be the best software tester. But, didn't most of us come into testing with little or no formal training or education? If you have the knack, it is possible to learn this job on the job, so Harry might be OK. Of course, he might be able to use some spell that could give him an advantage over Muggle testers. One well placed "Repairo!" could fix any piece of shoddy software.

Hedwig - Harry has shown that he is a good communicator, which is crucial for a software tester since we deal in information. He might need to adjust his ways a bit and use e-mail or IM instead of his owl, but we have every reason to believe he will get the message through.

So, assuming Harry lives and decides to become a software tester, I'd say he has a good chance at being quite successful.

Don't Let The Bad Bugs Bite

2007-07-16T15:34:00.000-04:00

Software testers are in the unique position of enjoying bugs, and are actively hoping to find them. That is why this blog is called Blue Screen Joy - to celebrate the rush you get when you find that truly nasty bug. I'm talking about those wonderful Severity 1, crash your system, lose your data, smoke coming out of the box kind of bugs.

Those are good bugs. I love those bugs . At least I do when I find them in something I am testing. Not so much when they happen to me outside of work.

For example, last week, when I smelled that smell that electronics make when they are burning, it did not make me excited. Unsuccessful at finding the source, I wrote the smell off as a fluke. I'll tend to do the same thing when I am testing and see a really good bug that I can't reproduce. That is what I consider a bad bug. I don't love those.

Just like with software testing, writing off non-reproducible bugs as flukes can come back to bite you. Last night, the upstairs of my house was a tropic 87 degrees Fahrenheit, despite the fact that the air-conditioner was running. The downstairs was only slightly cooler. I'm no Bob Vila, but after some investigation I discovered:

- The furnace filter was long past due for being changed
- A good portion of the air-conditioning system was frozen

Not only had I thought I had found the bug, but I also thought I had an easy (and free) solution. All I needed to do was change the filter and let the air conditioner thaw, and everything would be fine, right? Wrong. Remember that burning smell that I could not track down? The friendly neighborhood ARS man found out that the source of this smell was some high voltage wiring that led to a capacitor in the blower assembly, which is what makes air conditioning possible.

With the wires charred and the capacitor destroyed not only would my house get really hot, but I would also be unable to travel through time. Good news is that for a sum of money that made me question my career choice, the ARS man was able to fix everything and make me cool again.

The lesson here is that whether it is a bad bug you can't reproduce or a nasty smell you can't track down, you need to be careful when you write something off as a fluke. Instead, look at other seemingly unrelated problems you are seeing and try to imagine any possible connection. Failing that, come up with a Plan B. Cold beer works nicely in both situations.

QA vs. QE

2007-07-13T13:08:00.000-04:00

Depending on where you go, software testers might be involved in Quality Assurance (QA) or Quality Engineering (QE). Each of these terms have been defined countless times, often in contradictory ways. They also tend to be used interchangeably to mean the same things.

But are they the same? If we look at the words that make up the terms, they would seem to indicate two different activities.

Quality Assurance to me implies the job of assuring that the software has some level of quality. This works great when the software is good - you can just say, "Yup, the software is good. I assure you." But what about when the software is not so good. How can you assure quality when there is none? What do you say then?
"This software stinks"?
"This software is buggier than a stagnant pond in July"?

True as these statements may be, it is not assuring anyone that it is quality software, which is what the term Quality Assurance is all about. That is why I don't like the term. To steal from the overused truism of why you should not assume, you should not assure, because it makes an ass out of you and R.E.

Quality Engineering on the other had seems to be more about a process for improving quality. Engineering is about designing and building things, so Quality Engineering should be about designing and building quality. The input could be of any level of quality, but after it goes through the QE process it should be of higher quality than when it came in. That means that in QE you not only find bugs in the product, but also in the process.

That all sounds great, especially if you are giving an talk at a software testing conference, but what does it really mean to people in the trenches. Whatever QE process your company uses, it could probably be improved. Unfortunately, we're usually too busy finding bugs, writing defect reports, and verifying fixes to have much time left for improving the process. You could move the world if you had a place to stand, right? But, how do you get to that place? My answer to that question is the same as my answer to "How do you eat an elephant?" One bite at a time. Every thing you can do to improve the process, no matter how small, should mean less bugs finding their way into test, which means more time for the Quality Engineer to make more improvements to the process.

In the end, I don't think it really pays to get all worked up about if you are doing QA, QE, or Testing. These are all just words. What matters is that you know what you are trying to accomplish and how to get there. Let people who spend more time with PowerPoint slides than bugs worry about the terminology.

Blue Screen Joy is On

2007-07-11T10:00:00.000-04:00

The first step to making something better is to point out what is wrong with it.

With that in mind, I welcome you to Blue Screen Joy - a blog about software testing, quality engineering, the pursuit of bugs, and the ongoing quest for the holy grail - the Blue Screen of Death.

In this blog I will be sharing what I've learned about software testing during the last 10+ years as a professional software quality engineer. During that time I've done all kinds of testing in companies ranging from 100,000+ employees, to start-ups that were at times as small as 2 people. Speaking of people, I've also been known to test the patience of some folks along the way, especially developers. It is all for the cause, though.

I'll also be offering reviews and rants about software and tech products for which I am just a user. Nothing drives me as crazy as when I pay money for something and it does not work well - especially when the flaws could have and should have easily been discovered during test.

So, I hope you enjoy the blog and find it useful. Now, lets go kick the tires and light the fires!