Test This Blog - Eric Jacobson's Software Testing Blog

Sometimes I Feel Stupid as a Tester

2009-07-08T12:57:00.004-04:00

As a tester, while striving for the impossible goal of perfect software, I sometimes feel stupid. How valuable am I to the team? Do I really have any hard skills different than the next guy? Am I a testing failure?

I feel stupid when…

production bugs have to be patched (the kind I should have caught).
devs talk about code or architecture I don’t understand.
non-testers log bugs.
I have to execute brainless tests that the guy on the street could execute.
I can’t remember if I tested a certain scenario and my executed test documentation is incomplete.
the team celebrates individual dev accomplishments for feature sets and QA is not recognized.
my bug is rejected by dev for a legitimate reason.
I read a software testing blog post about some tester with 95% of her tests automated.

As a fellow tester, maybe you have felt stupid at times too. Feeling stupid is not fun and eventually will lead to disliking your job. I guess there are two solutions; 1.) find a new job or 2.) try not to feel stupid.

I talk my way out of feeling stupid as a tester the same way I do outside of work during conversations with doctors, physicists, CEOs or other potentially intimidating experts of some field. I remember that everyone is an expert at something…just something different. In the examination room, the doctor may be the expert at prescribing the treatment, but put the doctor and me at the bottom of a 300-foot-deep pit in a wet cave, and suddenly the doctor is asking me for help (I’m a caver).

When it comes to testing, we don’t know the same things the developers or BAs know but we shouldn’t feel stupid about it. It doesn’t mean we should stop learning, we just need to put things in perspective instead of feeling inadequate. Faking your knowledge is way worse than saying “I don’t know”.

Don’t second guess your skills as a tester.

In a future post, I'll tell you when I feel awseome as a tester.

Ask For Bug Resolution Comments

2009-06-29T13:18:00.004-04:00

“Added a validation in the service codes to make sure either "DSP" or "Already in GE" is true for Accession.”

Do your devs write bug resolution comments? If not, ask for them. All bug tracking systems have a spot for devs to write notes about what they did to fix the bug. It’s tough to get devs to write their resolution but it’s well worth the pain.

When you verify fixed bugs you should be asking yourself what regression testing is necessary; what did this bug fix break? If you don’t know how the dev fixed it, you don’t really know what could have broken as a result. Sometimes, devs are cool enough to provide tips of where other bugs may be lurking…

“This problem is bigger than the shoot value hiding. In fact, the entire preCat is not reloading after a save which is hiding other bugs (UI refresh and service layer issues.)”

Some of my devs are terrible at writing resolution comments. Sometimes I have no idea what they did to fix it. My emotions tell me to Reopen the bug. But after listening to my brain, and asking the dev what they did, I usually discover an unexpected fix that still solves the problem. Dev comments would have been nice.

You may also discover your devs share some of your angst when dealing with scatterbrain stakeholders, as evident in comments I’ve recently seen such as the following:

“UI now allows multiple preCats and DSP check box, even though earlier requirements conflict with both of these.”

“Yet another requirements change. The need dsp check box has always been there. It was expressed that it should only be visible when there IS and accession, but I guess we are changing that.”

Poor dev guy. I feel your pain. I miss the good old pre-agile days too.

Please Don’t Make Me Test Cosmetics

2009-06-17T14:04:00.001-04:00

One of my new AUTs has stakeholders who are obsessed with cosmetics. Despite having an AUT full of business processes gaps and showstopper bugs, during stakeholder meetings their first priority is to rattle off a list of all the cosmetic things they want changed. For example:

titles should left align
read-only fields should be borderless
certain fields should be bigger/smaller
less white space
no scroll bars
don’t like text color or font
buttons should be same width regardless of their names

Theoretically, Agile is supposed to address this kind of perpetual scope creep. But I hate it because even after listening to the stakeholders, it still becomes awkward for the dev to code and the tester to verify.

Something truly lacking in custom in-house (not shrink-wrap) apps is the ability for users to customize UIs until they’ve bored themselves to death. I’ve never been one to bother to “change skins” on my apps or even to change my desktop background. But cosmetics is a major concern for some users. Forcing me to test someone’s notion of what they think looks good on a UI is not interesting to me, as a tester. Let’s write software that lets users make their own cosmetic changes on their own time. I’ll test its engine. That sounds interesting.

Good Old Unreliable QuickTest Pro

2009-06-10T13:11:00.009-04:00

JART found a bug this morning. But it wasn't in my AUT.

JART had been happily smoke testing our pre-production environment this morning for an hour. I was eagerly awaiting the results for a group of anxious managers. After seeing QTP’s auto-generated test results consistently for 144 previous test runs, QTP suddenly decided to give me this instead of the results:

I didn’t change any versions of any software on this box, of course. After waiting another hour while JART repeated all the tests, the next results file was fine. …Annoying.

Why Perfect Regression Testing is Impossible

2009-06-04T13:30:00.002-04:00

We’ve been spinning our wheels investigating a prod bug that corrupted some data yesterday. Once we cracked it, we realized the bug had been found and fixed more than a year ago. …Depressing. My first thought? Why didn’t I catch this when it broke?

Perfecting regression testing is a seemingly impossible task. Some of you are thinking, “just use test automation...that's what they said in that Agile webinar I just attended”. If my team had the bandwidth to automate every test case and bug we conceived of, the automation stack would require an even larger team to maintain. And it would need its own dedicated test team to ensure it properly executed all tests.

It’s even more frustrating if you remove the option of automated regression testing. Each test cycle would need to increase by the same amount of time it took to test the new features in the last build, right? So if iteration 4 is a two week iteration, and I spend a week testing new features. That means, iteration 5 needs to be a three week iteration; I’ll need that extra week so I can run all the iteration 4 tests again. They’ll give me eight weeks to test iteration 10, right?

Wrong? You mean I have the same amount of test time each iteration, even though the amount of tests I have to execute are significantly increasing? This is a reality that somehow we all deal with.

Obviously, none of us have "perfect" regression testing. The goal is probably "good enough" but the notion of improving it is probably driving you crazy, as it is me. This topic is glossed over so much, I wonder how many testers have an effective strategy.

What is your regression test strategy?

Am I The Only Tester With No Time To Test?

2009-05-27T13:00:00.004-04:00

During a recent phone call with Adam White, he said something I can’t stop thinking about. Adam recently took his test team through an exercise to track how much of their day was actually spent testing. The results were scary. Then Adam said it, “If you’re not operating the product, you’re not testing”…I can’t get that out of my head.

Each day I find myself falling behind on tests I wanted to execute. Then I typically fulfill one of the following obligations:

Requirement walkthrough meetings
System design meetings
Write test cases
Test case review meetings
Creating test data and preparing for a test
Troubleshooting build issues
Writing detailed bug reports
Bug review meetings
Meetings with devs b/c tester doesn’t understand implementation
Meetings with devs b/c developer doesn’t understand bug
Meetings with business b/c requirement gaps are discovered
Collecting and report quality metrics
Managing official tickets to push bits between various environments and satisfy SOX compliancy
Update status and other values of tested requirements, test case, and bug entities
Attempt to capture executed exploratory tests
Responding to important emails (arriving multiple per minute)

Nope, I don’t see "testing" anywhere in that list. Testing is what I attempt to squeeze in everyday between this other stuff. I want to change this. Any suggestions? Can anyone relate?

JART – Automating Verifications For Report AUTs

2009-05-20T12:15:00.003-04:00

If your test automation doesn’t verify anything useful, it is essentially worthless. First, there are some basic tests I decided to programmatically verify with JART. These are tests that often fail during manual testing.

Can I access the report app for a given environment?
Does a working link to each report exist?
Does each report’s filter page display?
Does each report’s filter page display the expected filter controls I care about?

The above can be verified without even executing any reports. Piece of cake!

Next, I need to verify each report executes with some flavor of expected results. Now I’m bumping it up a notch. There are an unlimited amount of results I can expect for each report and these all require knowledge or control of complex reportable business data. This also means I have to examine the report results, right? My AUT uses MS ActiveReports and displays results in an object not recognized by QuickTest Pro. According to the good folks at SQA Forums, the standard way to extract info from the results is to use the AcrobatReaderSDK, which I don’t have. The workaround, which I use, is to install a free app that converts pdf files to text files. I wrote a little procedure to save my report results as pdf files, then convert them to text files, which I can examine programmatically via QuickTest Pro. So far, it works great. The only disadvantage is the extra 5 seconds per report conversion.

So what am I examining in the report results for my verifications? So far, I am just looking at each report’s cover page, which displays the specified filter criteria returned, along with its filter name (e.g., “Start Date = 3/20/2006”). If it returns as expected, I have verified the AUT’s UI is passing the correct filter parameters to the report services. This has been a significant failure point in the past, which is no surprise because the UI devs and service devs are poor communicators with each other.

Currently, JART verifys 59 Reports and up to 9 filters on each. It takes about 1 hour to complete. JART is ready to perform my next sanity test when we go live. So far I have put in about 24 hours of JART development.

I’ll discuss the simple error handling JART uses in a future post.

Note: The failures from the test run result summary above were the results of QuickTest Pro not finding the text file containing the converted report results. I couldn’t repro this JART error but now I may have to invest time researching the fluke and determining how to handle it. This is time not spent testing my AUT.

JART’s High-level Framework

2009-05-07T12:31:00.003-04:00

An early decision I had to make was whether I should programmatically determine which reports were available to test and programmatically determine which of their parameters were required, etc… or if I should track my own expectations for the reports I expected and the parameters owned by each. I went with the later because I don’t trust the devs to keep the former stable. JART needs the ability to determine when the wrong reports get released; a common dev mistake.

Since I have about 150 distinct reports, each with their own combinations of shared filter controls and possible filter values, I made a matrix in MS Excel. The matrix rows represent each report, the columns represent each filter control, and the intersections are the filter values I use to pass into JART for each report’s filter criteria controls. This single spreadsheet controls all the tests JART will execute.

Another advantage, for me, to controlling the tests via an Excel spreadsheet is that my BL already maintains an Excel spreadsheet that specifies which of the 150 reports should be available in each build. The BL’s list can control which reports JART tests, just like the BL's list controlled which reports I tested.

JART simply loops through each report in said matrix and provides standard verifications for each. Verifications are important, and tricky for report AUTs, so I’ll save those for the next post.

Jacobson’s Automated Report Testing

2009-04-29T13:11:00.002-04:00

It’s true what they say; writing automated tests is waaaay more fun than manual testing. Unfortunately, fun does not always translate into value for your team.

After attempting to automate an AUT for several years, I eventually came to the conclusion that it was not the best use of my time. My test team resources, skills, AUT design and complexity, available tools, and UI-heavy WinForm AUT were a poor mix for automated testing. In the end, I had developed a decent framework, but it consisted of only 28 tests that never found bugs and broke every other week.

Recent problems with one of my new AUT’s have motivated me to write a custom automated test framework and give the whole automated test thing another whirl.

This new AUT has about 50 reports, each with various filters. I’m seeing a trend where the devs break various reports with every release. Regression testing is as tedious as it gets (completely brainless; perfect to automate) and the devs are gearing up to release another 70 additional reports! …Gulp.

In this case, several aspects are pointing towards automated test potential.

The UI is web-based (easier to hook into)
The basic executed test is ripe for a data-driven automation framework; crawl through 120 reports and perform nearly the same actions and verifications on each.
Most broken report errors (I’m targeting) are objectively easy to identify; a big old nasty error displays.

I wrote the proof of concept framework last week and am trying to nail down some key decisions (e.g., passing in report parameters vs. programmatically determining them). My team needs me to keep testing, so I can only work on automation during my own time…so it’s slow going.

This is my kick-off post. I’ll explain more details in future posts. More importantly, I’ll tell you if it actually adds enough value to justify the time and maintenance it will take. And I promise not to sugar coat my answer, unlike some test automation folks do, IMO.

Oh, I’m calling it JART (Jacobson’s Automated Report Tester). Apparently JART is also an acronym for "Just a Real Tragedy. We’ll see.

Wear Your Tester Value on a Shirt

2009-04-22T12:22:00.008-04:00

During last fall’s STPCon, I attended a session about showing your team the value of testing. It was presented by a guy from Keen Consultants. He showed us countless graphs and charts we could use to communicate the value of testing to the rest of our team. Boring…zzzzzzzz.

In the spirit of my previous post, Can You Judge a Tester by Their Bug List Size?, here is a more creative approach, that is way simpler and IMO more effective, at communicating your value as a tester….wear it!

(I blurred out my AUT name)

You could change it up with the number of tests you executed, if that sounds more impressive to you. Be sure to wear your shirt on a day the users are learning your AUT. That way, you can pop into the training room and introduce yourself to your users. Most of them didn’t even know you existed. They will love you!

Now I just need to come up with an easy way to increase the bug count on my shirts (e.g., velcro numbers). Because, like all good testers know, the shirt is out-dated within an hour or so.

Test This #3 – Change Your Mind

2009-04-16T13:01:00.004-04:00

Users change their minds. They save new items in your app, then want to delete those saved items and try again. Does your AUT support this behavior? Did you test for it?

My devs recently added a dropdown control with two values (i.e., DVS or ESP). Soon after being able to save or change the values, I stopped testing. Later, a user pointed out there is no way to remove the values from that (optional) field if you change your mind. Now, many of our dropdowns look like this:

While testing, I often look for data saving triggers (e.g., a button that says “Save”). Then I ask myself, "Okay, I saved it by mistake, now what?".

Devs and BAs are good at designing the positive paths through your AUTs. But they often overlook the paths needed to support users who change their minds or make mistakes. Your AUT should allow users to correct their mistakes. If not, your team will get stuck writing DB scripts to correct production data for scenarios users could not fix on their own. It’s your job to show your team where these weaknesses are.

Can You Judge a Tester by Their Bug List Size?

2009-04-08T12:19:00.002-04:00

In one of James Whittaker’s recent webinars , he mentioned his disappointment when tester folks brag about bug quantities. It has been popular, lately, to not judge tester skills based on bug count. I disagree.

Last Monday night I had a rare sense of tester accomplishment. After putting in a 14 hour day, I had logged 32, mostly showstopper, bugs; a personal record. I felt great! I couldn’t wait to hear the team’s reaction the next day. Am I allowed to feel awesome? Did I really accomplish anything? Damn right I did. I found many of those bugs by refining my techniques throughout the day, as I become familiar with that dev’s mistakes. I earned my pride and felt like I made a difference.

But is it fair to compare the logged bug list of two testers to determine which tester is better? I think it is...over time. Poor testers can hide on a team because there are so few metrics to determine their effectiveness. Bug counts are the simplest metric and I think it’s okay to use them sometimes.

I work with testers with varying skills and I see a direct correlation. When a tester completes an entire day of work without having logged a single bug, I see a problem. The fact is, one logged bug proves at least some testing took place. No logged bugs could mean the AUT is rock solid. But it could also mean the tester was more interested in their Facebook account that day.

“If it ain’t broke, you’re not trying hard enough.”

This silly little cliché actually has some truth. When I test something new, I start out gentle, running the happy tests and following the scripted paths…the scenarios everybody discussed. If the AUT holds up, I bump it up a notch. And so on, until the bugs start to shake loose. That’s when testing gets fun. Logging all the stupid brainless happy path bugs is just busy work to get to the fun stuff. (Sorry, a little off subject there)

Anyway, from one tester to another, don’t be afraid to celebrate your bug counts and flaunt them in front of your fellow testers…especially if it makes you feel good.

BTW - Does anyone else keep a record of most bugs logged in a day? Can you beat mine? Oh, and none of my 32 got rejected. :)

Do Bug Titles Matter?

2009-04-02T12:43:00.002-04:00

Most bug readers will agree, a simple “Expected Results” vs. “Actual Results” statement in the bug description will remove all ambiguity. But what about the bug title? Is a bug title supposed to include the expected results, actual results, or both? Every time I log a bug, I pause to consider the various title possibilities. I want a unique but concise title that will summarize the bug, making the description as unnecessary as possible. My head swims with choices…

If user does X after midnight, user gets Y.
If user does X after midnight, user should not get Y.
If user does X after midnight, user should get Z.
If user does X after midnight, user gets Y instead of ZUser should get Z when they do X after midnight.
etc…

Here is what I think.

There is no “best” bug title. Any bug title is good enough as long as it:

is unique within my bug tracking system
describes the problem to some extent
includes key words (these words will be used someday to find this bug; searching for bugs where title contains some text)

So unless someone convinces me otherwise, with a comment on this post, I have decided to just use the first distinct bug title popping into my mind and stop worrying about bug title perfection.

Dealing With Smarty Pants Tester

2009-03-25T12:25:00.005-04:00

Roshni Prince asks,

Can you suggest some tips to deal with testers that attempt to dig into code and fix their own bugs?

I like the question, so I’ll tell you what I think.

If we had unlimited test time, SmartyPantsTester would rock! However, if we have to provide as much information as possible, about the current AUT quality, in a limited time, SmartyPantsTester gets in our way.

So how do we deal with SmartyPantsTester?

The Carrot Approach:

Convince SmartyPantsTester the real hero is the tester who can tell us something meaningful about the AUT quality. Can anyone help us…Please? We need someone smart enough to find the weak points in our AUT? We need someone familiar enough with the business to tell us if FeatureA will solve the user’s problems. Is anyone creative enough to determine how to test the feature the devs said was impossible to test? Is anyone methodical enough to determine the repro steps to this intermittent problem? We need someone brave enough to QA Certify this AUT for production. Get it? Appeal to the ego.

The Stick Approach:

Ask SmartyPantsTester to work extra hours until she can answer questions like the following:

Cool, you found an incorrect join statement, how does the rest of the AUT look?
Do the new features work properly?
How much of the AUT have you tested?
How many tests have you executed?
How many showstopper bugs have you logged?
In your opinion, is the AUT ready to ship?

And once again, I find myself with the same conclusion; there is simply too much to test in the available time. Testers reduce their chances of success by trying to do the devs’ job too.

We Don’t Need No Stinkin Expected Results

2009-03-18T13:13:00.002-04:00

Should you log a bug without knowing the expected results?

Someone chuckled at my “?” for the expected results, during a bug review meeting. The bug (like many of my bugs) looked like this:

Repro Steps:
1. do something
2. do something else
3. do something else

Expected Results: ?
Actual Results: an unhandled error is thrown

Good testers determine scenarios that nobody thought of. That is one skill that makes us good testers. Some time ago, I didn’t let myself log bugs until I tracked down the proper oracles and determined the expected results; a practice that sounds good…until you try it. Unfortunately, the oracles are never quick to respond. Often they pose questions to the users, which open additional discussions, meetings, etc…until the tester forgets the details of their wonderful test.

So these days, if I encounter a bug and I don’t know the expected results, I log the bug immediately and let someone else worry about expected results…if they even care. It’s not because I’m too lazy to seek out my oracles. It’s because my time is better spent logging more bugs! When in doubt, remember, the tester’s primary job is to provide information about the AUT. It’s not the responsibility of the tester to determine expected results. If the tester identifies a scenario that will crash the AUT, they should log the bug now.

How Do You Screen Capture?

2009-03-10T12:28:00.003-04:00

I’m surprised by how many people still send around bmp files of their entire desktop when they are only interested in showing some small error message displaying in a little window. They are using the [Print Screen] key. Some, at least know they can use [Alt]/[Print Screen] to capture only the active window.

Others prefer to capture only the area the audience needs to understand; they may use a screen capture app. I’ve been using Wisdom-soft’s free ScreenHunter . I’ve got it customized to capture the area within a rectangle I draw, after pressing F6. After drawing my rectangle, its contents capture as an auto-named gif file and a clipboard item.

Screen-capture-type-stuff I think about :

I try to avoid screen capturing error messages, opting instead to capture the error message in text format, from an error log. That way the dev can see the whole message and copy the text if they want to search the code or something. If the devs don't log the error, they're stuck with a screen capture.
If the screen capture needs other context (e.g., which programs are running in my tray, what time is it) I still capture the entire desktop.
Occasionally I mark up the screen capture (in Paint.NET) to circle something or add other annotations.
If capturing action is better, I capture video.
Sometimes I save time by using a screen capture to support repro steps. Example: Capture a filter page for a report and write a repro step that says "specify all filter criteria as depicted in the screen capture".

What (free) screen capture program do you use? What screen capture tips did I miss?

Test This #2 – Abuse Buttons

2009-02-25T12:56:00.001-05:00

Chances are, your AUT probably has buttons on the UI somewhere. If so, those buttons trigger actions. A common oversight is to not handle multiple actions being triggered at nearly the same time.

Testers and devs are familiar with standard UI controls. We know buttons don’t generally require double-clicks. However, many users don’t have this instinct. These users double-click everything, even buttons. Become one of them!

My AUT had a bug that actually allowed users to rapid-fire-click a generate invoice button, and get duplicate invoices. Ah…yikes.

Here is the bread and butter test:
Get your mouse over a button that triggers an action. Get your finger ready. Click that button multiple times as quickly as you can. Now go look in the DB, error logs, or wherever you need to look to determine if multiple actions where triggered inappropriately. No bug? Try it a few more times. Or try getting the focus on the button and using a rapid-fire [Enter] key.

Got any variations on this?

What We Can Learn From Dumb Testers

2009-02-18T13:01:00.000-05:00

The more one learns about the inner workings of an AUT, the more one may get distracted from logging important bugs.

I saw this happen last week with a fellow teammate. She discovered a search criteria control that didn’t allow the user to search by certain values (e.g., they could search by A, B, D, but not C). Instead of logging a bug, she explained to me why it didn’t work. She was thrilled with her knowledge of countless dev discussions trying to fix the data structure beneath said search control. It was more exciting for her to explain the complex data than to worry about the little users who would not be able to search per their expectations. It was like she was suddenly on the side of the developers…and the users would never understand the complex challenges of making software work. “It’s not a bug, nobody can figure out how to deal with the complex data”.

Huh?

Dumb testers don’t have this problem. If it doesn’t follow the spec, they log it. And that’s the good thing about dumb testers. Be careful with your knowledge.

The Unwanted Bug Fix – Conclusion

2009-02-10T13:11:00.006-05:00

Alex and Joe,

I agree with both your comments to my last post.

Joe blames the dev…

“- No bug report- [dev] corrected it on their own”

Obviously devs catch defects in their code (logic errors, missed specs, inefficiencies). If devs find and resolve defects in previously released code, should a bug be logged? On one hand, if the dev corrects it before anyone else catches it, why should the dev have to tell anyone? If a tree falls in the forest and nobody is around, does it make a sound? On the other hand, we can’t assume an unnoticed defect in the field, means said defect’s resolution will also be unnoticed. Thus, IMO it’s best for devs to let the team know about all changes. Typical ways to do this:

Dev logs bug

Dev tells tester to log bug

Dev logs a work item representing work they did (e.g., refactored code for FeatureA). The work item should be visible to the team. The dev gets credit for their wonderful coding, and the tester gets the tap on the shoulder to attempt some testing.

Alex blames the tester…

“That test should have failed a long time ago.”

In this case, the feature did not match its spec. You're damn right, the tester should have caught this! The tester should thank their dev for doing the testers job. (Unfortunately, if the tester relied on the spec, we would be right back where we started.)

So a third member must also take the blame; the spec author (e.g., Business Analyst), who did not capture the true business need of the feature. However, we can’t put all the blame on them. “The requirements sucked!” …this is a tired argument, one that I decided never to waste my time with. Anyone who disagrees probably has not attempted to write requirements.

The next time you feel compelled to point the finger at a teammate, don’t. Responsibilities blur across the software development trades and the whole team can share success or failure.

(sorry, this post is a little dull. I’ll try for something juicier next week)

The Unwanted Bug Fix

2009-02-03T18:25:00.008-05:00

My team just released a new build to prod that included a freebie from a dev. I say "freebie" because the dev noticed code not complying to a requirement and corrected it on their own. There was no bug logged.

It turns out, our users preferred the feature prior to the dev's fix. We are now in the process of rushing through a “showstopper” production patch to insert the previous code back into prod for business critical reasons.

What went wrong?

Application Egos Are Bad

2009-01-28T12:16:00.002-05:00

Testers are weird. When they find a bug in their AUT they feel good. But when it comes to integration testing, testers feel a sense of defeat when a bug is found to be in the AUT they are responsible for, rather than the AUT the other tester is responsible for. Can you relate? Stay with me.

When discussing integration testing observations, devs, testers, and business people say things like…

“we do this”
“then we do that”

…to describing the application they associate themselves with. And if the discussion includes an external application, folks start saying things like:

“when they send us this, we send them that”
“they create the XML file and we import it”

I hate when people use subjective personal pronouns instead of proper names. For one thing, it would be easier to understand a sentence like “Application_A sends the file to Application_B”, than a sentence like “They send us the file”. Afterall, they are called subjective personal pronouns. But the other reason I hate this way of communicating is that it reinforces an unhealthy sense of pride. People connect themselves, too intimately, with the application they refer to as “we”. People are biased towards the group they belong to. They start to build a bubble around their app; “It’s not our bug, it’s their bug”.

My little language tip may seem trivial, but think about it the next time you discuss system integration. If you resist the urge to use subjective personal pronouns, I think better communication will occur and your ego will be less likely to distract from effective teamwork.

There IS Such a Thing as Testing Too Early

2009-01-21T12:57:00.002-05:00

“Test early” has been banged into my head so much, it has ruined me.

I just started testing a new app with business language/processes that I am unfamiliar with. While waiting on a UI, I began testing the services. I also selected an automation framework and tried to write some functions to leverage later via an automation library. At first, I was proud of myself for testing early. I did not have to make decisions about what to test because there was so little to test, I could just test it all! How nice. Things would soon change however.

As I sat through the domain walkthroughs, I realized I was learning very little about the complex functionality that was coming. I didn’t know which questions to ask because each business process was an enigma. The more I hid my confusion, the less valuable I felt, and the less I knew which tests to execute.

Finally, I broke out of my bubble and set up a meeting with the primary business oracle. Knowing close to nothing about the business side of the app, I asked the oracle one simple question:

“Can you walk me through the most typical workflow?”

She did. And even if only 10% of what she explained made sense, it became my knowledge base. Later, I could ask a question related to the 10% I understood. If I understood the answer, now I understood 11% of the app. And so on. Knowledge leads to confidence. Confidence leads to testing the right stuff.

So don’t get wrapped up in all the fancy "test early" stuff that makes for impressive hallway discussions. Start with the simple, low-tech approach of learning what your AUT is supposed to do.

Being technical <> being valuable.

Test This – Inactive Items

2009-01-14T15:28:00.001-05:00

Chances are, your AUT has some items that can be deactivated somewhere; probably in an admin screen. This is a great place to catch some serious bugs before they go to prod. Here are a few tests you should execute.

Start with the easy ones:

1. Make ItemA “in use” (something in your AUT depends on itemA).
2. Attempt to deactivate ItemA.
Expected Results: ItemA cannot be deactivated. User communication indicates ItemA is in use.

1. Deactivate an unused item (call it ItemA).
2. Attempt to use ItemA somewhere (e.g., does ItemA display in a dropdown menu?).
Expected Results: ItemA cannot be used because it is unavailable.

Then try something more aggressive:

1. UserA opens a UI control that displays ItemA as a potential selection.
2. UserB deactivates ItemA (e.g., from an admin screen).
3. UserA selects ItemA from the UI control.
Expected Results: ItemA cannot be used by UserA because it is unavailable. Communication to UserA explains ItemA is inactive.

Got any good variations?

Stressed Testing

2009-01-08T13:24:00.001-05:00

Just before the holidays, we went live with another relatively huge chunk of users who require slightly different features than our previous users. The bug DB is quickly filling up with bugs discovered in production. These bugs are logged by the business/support arm of our team because testers can’t keep up. Many of the bugs don’t have repro steps and appear to be related to multiple users, performance, deadlocking, or misunderstood features. Other bugs are straight forward; oversights uncovered after users take the app through new paths for the first time.

My team is struggling to patch critical production issues to keep the users working through their deadlines. I want to investigate every new bug to determine the repro steps and prepare for verifying their fixes. Instead, I’m jumping from one patch to the next, attempting to certify the patches for production. Keeping up is difficult. New emails arrive every few seconds.

This is an awkward phase, but the team is reacting well, maintaining a good reputation for quick fixes. Nevertheless, I’m stressed.

Am I doing something wrong?
Do I suck for letting these bugs get to prod in the first place?
Should I be working late every night to clean up the bug DB?
Am I the bottleneck, too slow at getting patches to users?
Should I certify patches that are only partially fixed?
Should I be writing new tests to verify these bugs and prevent them from returning?
Should I be out in the trenches, watching the user behavior?

Can you relate?

Testing is Like Raking Leaves

2008-12-15T08:44:00.005-05:00

My manager recently said she hates raking leaves because as soon as she rakes her yard, she turns around and there are more leaves to rake.

I immediately thought…weird, that’s exactly what testing feels like.

We get a build. It’s full of bugs. We work hard all day logging bugs. By the time we have a chance to turn around and admire our working AUT, we’ve gotten a new build and the bugs are back. So we get out the rake and start all over.

If I’m short on time, sometimes I just rake the important parts of the yard. If I’m not sure where to start, I usually look under the large Oak trees; I’ve noticed fewer leaves under the Loblolly Pines. If I’m expecting a windy day, I usually wait until the next day to rake, allowing the fallen leaves to accumulate. Sometimes, I find an obscure leaf…I have to ask my wife if I should rake it or leave it there. If I get done early, I might rake out some of those leaves from last season, from the garden bed on the side of the house.

It’s exhausting, really. But somebody’s got to keep the yard clean.