SoftWare TestinG

Open source QA tool for automated Web application testing

2009-03-31T10:21:00.001-07:00

Q- Could you recommend a quality assurance (QA) tool for automated regression/functional testing (open source or free tools are preferred) for testing a Web application that contains a lot of JavaScript for opening pop-up windows, redirects, etc.? I'm using HTTPUnit and Selenium but these tools are not handling pop-up windows and redirects well. Thanks

A- My preferred automation solution for Web applications is a combination of *Unit and Wati* -- for instance, JUnit plus Watij. I like Watij over Selenium RC simply because it seems a little more object-oriented than Selenium. But this is totally personal, I've used both tools successfully.

Handling pop-ups and other interactive display changes can be a challenge, regardless of the tool. You might consider a couple of approaches. First, be active in the tool's user group, seeking solutions. There are several groups available on the Internet -- just pick one or two. Be polite: post your question once, rather than blasting across multiple groups. If you post to the Selenium user's group, you will definitely encounter other testers who have faced similar challenges in the past -- they'll probably have tried-and-true solutions for you. Secondly, pay close attention to your implementation. If your Web app uses a lot of rich Internet applications (RIAs), your might get away with the use of divs rather than handling each window (in RIA, developers can "pop up" windows which are, in fact, just hidden divs being exposed). Experiment with different programming solutions and see which is more reliable. There is generally more than one way to accomplish what you're trying to do. You're looking for the way which is 1) feasible, 2) most reliable and 3) most performant (in that order).

The third approach may be the most challenging and, in the short term, costly. However, if you're working on an ongoing, long-term project it will have payoff. If the current implementation is not very testable, propose alternative implementations to the development team. For instance, if the current project creates multiple pop-up windows (rather than using Ajax to expose elements), ask that the team take time to change this, implementing a solution which you can test more reliably and in a shorter amount of time. You need to be very, very detailed in your reasoning -- you will need to include schedule and cost savings. Point to gains down the road when your automated regression and functional tests run more reliably. By implementing testability, you will be addressing what Agile teams call "technical debt." You take a short-term hit on schedule, with the outcome being a long-term improvement in effectiveness and efficiency.

Will penetration testing be replaced by preventative tools?

by michaeldkelly

I recently read the article "Penetration Testing: Dead in 2009" by Bill Brenner. In the article Mr. Brenner follows a small debate around the idea that over time penetration testing will be largely replaced by preventative checks.

The debate opens with some quotes from Brian Chess from Fortify Software. Fortify creates code analysis tools that scan for security concerns and adherence to good secure coding practices. That potential bias aside, I suspect that Mr. Chess' statement — that "Customers are clamoring more for preventative tools than tools that simply find the weaknesses that already exist [...]. They want to prevent holes from opening in the first place" — is absolutely true. I know I clamor for those tools, and I'm just a lowly test manager.

I'm a big fan of the work companies like Fortify, IBM and HP are doing in this space. If my project team can find a potential issue before we deploy the code, I'm all for it. It can save us time and helps us focus on different and potentially higher-value risks. However, I've yet to see a tool that can deal with the complexity of a deployment environment (setup, configuration, code, etc…) and while I'm a big believer in doing everything you can up front (design, review, runtime-analysis, etc.), I believe there will always be a roll for a skilled manual investigation of what gets deployed.

Testing (penetration or other) is about applying skill and judgment to uncover quality-related information about the product. That's not just code — it's more than that. Your typical penetration tester today covers more than today's automated tools can cover. While there are different tools to test various components (some that focus on code, some that focus on the network, etc.), and they should absolutely be used, those tools will never be able to uncover all the potential issues with a system. And, what's sometimes worse, is they can lead to a false sense of security.

Two-minute guide to determining software testing coverage

2009-03-31T10:19:00.001-07:00

By Michael Kelly

Deciding what to test really involves two different questions. The first is a question of scope: "Out of everything that I could possibly test, which features are the right ones to test?" There will always be more to test than you will have time to test. The second is a question of technique and coverage: "For each feature I am testing, how do I want to test that feature?" Different quality criteria will lead to covering different product elements and different testing techniques.

In this two-minute crash course, I'll provide some details on how I answer those questions and how I structure my test execution to ensure I'm testing for the right risks at the right time.

2:00: Figure out the scope of your testing
For the question about scope -- what features should we test -- I like using Scott Barber's FIBLOTS mnemonic (which he presents in his Performance Testing Software Systems class). Each letter of the mnemonic helps us think about a different aspect of risk. Here's a summary of how I apply FIBLOTS when thinking about scope:

Frequent: What features are most frequently used (e.g., features the user interacts with, background processes, etc.)?
Intensive: What features are the most intensive (searches, features operating with large sets of data, features with intensive GUI interactions)?
Business-critical: What features support processes that need to work (month-end processing, creation of new accounts)?
Legal: What features support processes that are required to work by contract?
Obvious: What features support processes that will earn us bad press if they don't work?
Technically risky: What features are supported by or interact with technically risky aspects of the system (new or old technologies, places where we've seen failures before, etc.)?
Stakeholder-mandated: What have we been asked/told to make sure we test?

1:33: Understand the details of each feature you're testing
Once I understand what it is I want to test, I move on to understanding what aspects of each feature I'd like to cover. For that, I pull out the Satisfice Heuristic Test Strategy Model. I use the product elements list in that document to determine what aspects of the feature I need to focus on. At a high level, I think of coverage in terms of:

Structure: This is everything that comprises the physical product or the specific feature I'm looking at (code, hardware, etc.).
Functions: Everything that the product or feature does (user interface, calculations, error handling, etc.).
Data: Everything that the product or feature processes (input, output, lifecycle).
Platform: Everything on which the product or feature depends (and that is outside your project).
Operations: How the product or feature will be used (common use, disfavored use, extreme use, etc.).
Time: Any relationship between the product and time (concurrency, race conditions, etc.).

1:03: Structure your work in a way that makes sense to you
I typically start by structuring my work in lists or spreadsheets. Then, once I know what I'm going to test, I start to think of how I'm going to test it. It's not real to me until I can visualize the testing taking place. Do I need specialized software to help (like runtime analysis tools)? Will I need to write code or coordinate some activity (like a network failure)? Even visualizing something as simple as the data that I'll need can sometimes trigger a new idea or obstacle I'll need to tackle. As I think about each test, I'll start to group my tests into charters.

Once I have my charters figured out, I'll start to tackle whatever obstacles or setup tasks need to be done to allow me to run them. Some charters won't have any, and others might require a joint effort across teams. Generally, I'm ready to start testing once two conditions are satisfied:

There is software somewhere that's ready for some level of testing.
I have at least one charter that's ready to be executed (setup is completed or wasn't required).

0:35: Get your hands on the software you're testing
You'll notice I don't have a lot of entry criteria for my testing. That's because I'm always interested in seeing the software as soon as possible. I don't care how buggy it might be, once I see what I'm going to be testing, often my test ideas change. So the sooner I see it, the sooner I can provide feedback to the developer and start refactoring my tests.

While this philosophy won't work for all of my testing (in general I need something that's functionally sound before I can really start performance testing), it's reflective of a value I have to be an asset to the rest of the team. While I of course always want the most bug-free code I can find (well-designed, unit-tested, peer-reviewed), I'm a realist. Sometimes my feedback is more valuable to the team if I can get eyes on the product sooner rather than later.

0:21: Start with the components and build your way out from there
That said, I do have some general timing heuristics I use when thinking about when to test what. In general, I won't start doing any sort of end-to-end testing (following data through multiple parts of a system or subsystems) until I'm fairly confident each piece of the system is working to some degree (basic functionality has been confirmed, it's relatively stable and so on).

I typically don't try to do much automation or performance testing until I get at least one "stable" interface. The interface could be a Web service, a user interface, or even a method call, but I want it to be through at least one or two rounds of preliminary testing and I want to have some indication from the programming team that they don't plan to make major changes to the interface any time soon. I'm not looking for a promise it won't change, things change all the time -- I just want us to agree that right now we don't expect it to change.

0:05: Don't forget to regression test
Finally, I typically won't start regression testing until I've completed my first round of chartered test execution. Schedule constraints can of course override that, but I like the idea of regression testing being the last thing I do. It makes me more comfortable with the changes made as a result of my testing, and it gives me one last (often more relaxed) look at the product.

7 Tips to be More Innovative in the Age of Agile Testing to Survive an Economic Crisis

2009-03-31T10:18:00.001-07:00

What is Agile Testing?
"Agile testing involves testing from the customer perspective as early as possible, testing early and often as code becomes available and stable enough from module/unit level testing." - A wikipedia definition.

Why Need of Innovations in the Age of Agile Testing?

Global Recession/Economic downtime effect
Current Events are not Current Trends –

When global downturns hit, there is certain inevitability to their impact on information technology and Finance Sectors. Customers become more reluctant in giving software business. Some customers are withdrawing their long term projects and some customers using the opportunities in quoting low price. Many projects that dragged much longer than expected and cost more than planned. So, Companies started to explore how "Agile with different flavors" can help their Enterprises more reliably deliver software quickly and iteratively. The roles and responsibilities of Test Managers/Test Architects become more important in implementing Agile Projects. Innovations are increasingly being fueled by the needs of the testing society at large.

The Challenges in Agile Testing

Agile Testers face lot of challenges when they are working with Agile development team. A tester should be able to apply Root-Cause Analysis when finding severe bugs so that they unlikely to reoccur. While Agile has different flavors, Scrum is one process for implementing Agile. Some of the challenging scrum rules to be followed by every individual are

Obtain Number of Hours Commitment Up Front
Gather Requirements / Estimates Up Front
Entering the actual hours and estimated hours daily.
Daily Builds
Keep the Daily Scrum meetings short
Code Inspections are Paramount

So, in order to meet the above challenges, an agile tester needs to be innovative with the tools that they have. A great idea happens when what you have (tangible and intangible) meets the world's deepest hunger

How Testers Can be More Innovative in the Age of Agile Testing?

Here are Important Keys to Innovation:

1. Creative

A good Agile Tester needs to be extremely creative when trying to cope up with speed of development/release. For a tester, being creative is more important than being critical.

2. Talented

He must be highly talented and strives for more learning and innovating new ideas. Talented Testers are never satisfied with what they have achieved and always strives to find unimaginable bugs of high value and priority.

3. Fearless

An Agile Tester should not be afraid to look at a developer's code and if need be, hopefully in extreme cases, go in and correct it.

4. Visionary

He must have a comprehensive vision, which includes client's expectations and delivery of the good product.

5. Empowered

He must be empowered to work in Pairs. He will be involving in Pair Programming to bring shorter scripts, better designs and finding more bugs.

6. Passionate

Passionate Testers always have something unique to contribute that may be in terms of their innovative ideas, the way they carry day-to-day work, their outputs and improve things around them tirelessly.

7. Multiple Disciplines

Agile Tester must have multiple skills like, Manual, Functional, Performance testing skills and soft skills like Leadership skills, Communication skills, EI, etc. so that agile testing will become a cake walk.

What's the difference between priority and severity of bugs in Software Testing?

2009-03-31T10:16:00.001-07:00

Source: one stop software testing

Priority" is associated with scheduling, and "severity" is associated with standards.

"Priority" means something is afforded or deserves prior attention; a precedence
established by order of importance (or urgency).

"Severity" is the state or quality of being severe; severe implies adherence to rigorous standards or high principles and often suggests harshness; severe is marked by or requires strict adherence to rigorous standards or high principles, e.g. a severe code of behavior.

The words priority and severity do come up in bug tracking. A variety of commercial, problem tracking/management software tools are available. These tools, with the detailed input of software test engineers, give the team complete information so developers can understand the bug, get an idea of its 'severity', reproduce it and fix it.

The fixes of bugs are based on project 'priorities' and 'severity' of bugs. The 'severity' of a problem is defined in accordance to the customer's risk assessment and recorded in their selected tracking tool. A buggy software can 'severely' affect schedules, which, in turn can lead to a reassessment and renegotiation of 'priorities'

How to write effective bug report?

"The purpose of a bug report is to let the developers see their faults and failures of the application under test. The bug report explains the gap between the actual result and expected result, and the details of that how to reproduce the bug."

Many times it happens that if the bug report is not effective or incomplete then programmers face many problems while fixing the bugs.

Due to the Bad bug report:
1. bug is not reproducible by developers
2. bug is fixed but with incorrect functionality.
3. delay in bug fixes
and many more….

Sample of bad bug report:

Bug Title: Error message

When running the application, I get an "Internal Server Error" that says "See the .log file for more details".

Steps to Recreate:
Happens when "Document.create = null". It is not happening when changed to " Document.create".

Expected results:
this error message should not appear when status is "Document.create = null"

Observed results:
See above.

So now how to write effective bug reports? Below I am giving some Bug report best practices:

1. Once the bug is found
Check the bug repository and search that if the bug is already exist. If exists, then check whether the status of bug is CLOSED OR OPEN. If the Status of bug is closed then REOPEN it.
Now, if the bug is not there in the repository and it is a new bug, then you need to report the bug.

2. If the bug is Reproduce able, then report it, Otherwise avoid reporting of non-reproducible bugs (best practices).

3. Report a new bug: "Bug description" also known as "Short description" or "Bug Summary": It should be a small statement, which briefly points towards the exact problem. Writing a one line description is an ART. Bug Summary helps everyone quickly review outstanding problems. It is the most important part of the bug. It should describe only the problem, not the replication steps.
If it is not clear then managers might defer the bug by mistake and also it affects the individual performance of a tester.

4. The Language of the bug: Language should be as simple as possible and as straight as possible. Don't point any developer through your words. Remember – the nasty is the bug, not the programmer.

The language should be such that is the bug report should be easily understandable by developers, fellow testers, managers, or in some cases, even the customers

5. Steps to Reproduce:
- The steps should be in a logical flow. Don't break the flow or skip any step.
- Mention the Pre-requisites clearly.
- Use attachments and screenshots of errors, and annotate the screenshots.
- The details must be elaborated like which buttons were pressed and in what order.
Note – Please don't write an essay on it. Be clear and precise. People do not like to read long paragraphs

6. Give Examples: either with actual data or the dummy scenario. It will be easy for developers to recreate the bug.

7. Provide the Test Case ID, requirement ID, and Specs Reference.

8. Define the proper Severity and Priority.
The impact of the defect should be thoroughly analyzed before setting the severity of the bug report. If you think that your bug should be fixed with a high priority, justify it in the bug report.

This justification should go in the Description section of the bug report.
If the bug is the result of regression from the previous builds/versions, raise the alarm. The severity of such a bug may be low but the priority should be typically high.

8. Read what you wrote. Read the report back to yourself, and see if you think it's clear. If you have listed a sequence of actions which should produce the failure, try following them yourself, to see if you missed a step.

9. Mention the correct environment, application link, build number, and login/password details (if any).

10. Common issues: Many times it happens that the bug is not reproducible (even though the bug report is good) by developers, the don't worry, arrange go to meeting/walkthrough with them and help them in order to recreate the bug. And sometimes it happens like first day the bug is appearing then on next day the same bug is not appearing. In this case, the bug can be assigned back to you. Now you need to accept it and close the bug with appropriate comments like
"It is working fine now, but previously this problem was appearing. So, will close this bug after verifying in next build."
Of course, you need to close the bug after verifying in the next release/build/patch because it is an inconsistent bug.
Thus a good tester needs to be patient & always build a defense mechanism in the form of preserving test data & screenshots etc. to justify his statements.

11. Don't Assume the Expected results. Write the expectations which are mentioned in the test case, requirements documents, FDD or in Specification documents.

That's all. Practices make us perfect.

Code Coverage " A White Box Testing Technique

What is code coverage – An analysis method that determines which parts of the software have been executed (covered) by the test case suite and which parts have not been executed and therefore may require additional attention.

As per wiki – "Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing."

Code coverage measurement simply determines those statements in a body of code have been executed through a test run and those which have not. In general, a code coverage system collects information about the running program and then combines that with source information to generate a report on test suite's code coverage.
Code coverage is part of a feedback loop in the development process. As tests are developed, code coverage highlights aspects of the code which may not be adequately tested and which require additional testing. This loop will continue until coverage meets some specified target.

The main ideas behind coverage:
- Systematically create a list of tasks (the testing requirements)
- Check that each task is covered during the testing

Code coverage is defined in six types as listed below:

• Segment coverage – Each segment of code b/w control structure is executed at least once.
• Branch Coverage or Node Testing – Each branch in the code is taken in each possible direction at least once. Branch Coverage Gives a measure of how many assembler branch instructions are associated with each line. In addition, a measure of the number of branches taken/not taken is given.
• Compound Condition Coverage – When there are multiple conditions, you must test not only each direction but also each possible combinations of conditions, which is usually done by using a 'Truth Table'
• Basis Path Testing – Each independent path through the code is taken in a pre-determined order. This point will further be discussed in other section.

Basis path testing is a white box testing technique first proposed by Tom McCabe. The Basis path method enables to derive a logical complexity measure of a procedural design and use this measure as a guide for defining a basis set of execution paths. Test Cases derived to exercise the basis set are guaranteed to execute every statement in the program at least one time during testing.

• Data Flow Testing (DFT) – In this approach you track the specific variables through each possible calculation, thus defining the set of intermediate paths through the code i.e., those based on each piece of code chosen to be tracked. Even though the paths are considered independent, dependencies across multiple paths are not really tested for by this approach. DFT tends to reflect dependencies but it is mainly through sequences of data manipulation. This approach tends to uncover bugs like variables used but not initialize, or declared but not used, and so on.
• Path Testing – Path testing is where all possible paths through the code are defined and covered. This testing is extremely laborious and time consuming.

Path coverage
- Goal is to ensure that all paths through program are taken
- Too many paths
- Restrict to paths in a subroutine
- or to two consecutive branches

• Loop Testing – In addition to above measures, there are testing strategies based on loop testing. These strategies relate to testing single loops, concatenated loops, and nested loops. Loops are fairly simple to test unless dependencies exist among the loop or b/w a loop and the code it contains.
This white box technique focuses exclusively on the validity of loop constructs. Four different classes of loops can be defined:

1. simple loops,
2. nested loops,
3. concatenated loops, and
4. unstructured loops.

Simple Loops:
The following tests should be applied to simple loops where n is the maximum number of allowable passes through the loop:

1. skip the loop entirely,
2. only pass once through the loop,
3. m passes through the loop where m < n,
4. n - 1, n, n + 1 passes through the loop.

Nested Loops:
The testing of nested loops cannot simply extend the technique of simple loops since this would result in a geometrically increasing number of test cases. One approach for nested loops:

1. Start at the innermost loop. Set all other loops to minimum values.
2. Conduct simple loop tests for the innermost loop while holding the outer loops at their minimums. Add tests for out-of-range or excluded values.
3. Work outward, conducting tests for the next loop while keeping all other outer loops at minimums and other nested loops to typical values.
4. Continue until all loops have been tested.

Concatenated Loops:
Concatenated loops can be tested as simple loops if each loop is independent of the others. If they are not independent (e.g. the loop counter for one is the loop counter for the other), then the nested approach can be used.

Unstructured Loops:
This type of loop should be redesigned not tested!!!

The role of a software test manager

2009-03-31T10:15:00.001-07:00

By David W. Johnson

The role of the software test manager or test lead is to effectively lead the testing team. To fulfill this role, the lead must understand the discipline of testing and how to effectively implement a testing process while fulfilling the traditional leadership roles of a manager. What does that mean? The manager must manage and implement or maintain an effective testing process. That involves creating a test infrastructure that supports robust communication and a cost-effective testing framework.

What the test manager is responsible for:

Defining and implementing the role testing plays within the organization.
Defining the scope of testing within the context of each release/delivery.
Deploying and managing the appropriate testing framework to meet the testing mandate.
Implementing and evolving appropriate measurements and metrics.

To be applied against the product under test.
To be applied against the testing team.

Planning, deploying and managing the testing effort for any given engagement/release.
Managing and growing testing assets required for meeting the testing mandate:

Team members
Testing tools
Testing processes

Retaining skilled testing personnel.

The test manager or lead must understand how testing fits into the organizational structure. In other words, he must clearly define its role within the organization. This is often accomplished by crafting a mission statement or a defined testing mandate. Example: "To prevent, detect, record and manage defects within the context of a defined release."

Now it becomes the test lead's job to communicate and implement effective managerial and testing techniques to support this "simple" mandate. Your team, peers' (development lead, deployment lead and other leads) and superior's expectations need to be set appropriately given the timeframe of the release and the maturity of the development team and testing team. These expectations are usually defined in terms of functional areas deemed to be in scope or out of scope. Examples of those in scope include creating a new customer profile and updating a customer profile. Examples of those out of scope may include security and backup and recovery.

The definition of scope will change as you move through the various stages of testing. The key thing is to make sure your testing team and the organization as a whole clearly understands what is being tested and what is not being tested for the current release.

The test lead/manager must employ the appropriate testing framework or test architecture to meet the organization's testing needs. Although the testing framework requirements for any given organization are difficult to define, there are several questions the test lead/manager must ask. The answers to the questions and others will define the short- and long-term goals of the testing framework.

What is the relationship between product maturity and testing?
In the chart below, the first arrow leads to the product being ready for deployment. The second arrow leads to the product being ready to be tested as an integrated or whole system. The third arrow indicates functional testing can be performed against delivered components. The fourth arrow indicates the developer can test the code as an un-integrated unit. And the fifth arrow leads to the product concept being captured and reviewed.

How can the testing organization help prevent defects?
There are really two sides to testing verification and validation. Unfortunately the meaning of those terms has been defined differently by several governing/regulatory bodies. To put it more succinctly, there are tests that can be performed before the product is constructed or built, and there are tests that can be performed after the product has been constructed.

To prevent defects from occurring, you must test before the product is constructed. There are several methods for doing that. The most powerful and cost-effective method is reviews. Reviews can be either formal, technical reviews or peer reviews. Formal product development life cycles will provide the testing team with useful materials/deliverables for the review process. When properly implemented, any effective development paradigm should supply those deliverables. Example of development models and at what point during those models you can get information for the review process:

Cascade or waterfall

Requirements
Functional specifications

Agile or Extreme Programming

High-level requirements
Storyboards

Testing needs to be included in this review process, and any defects found need to be recorded and managed.

How and when can the testing organization detect software defects?
The testing organization can detect software defects after the product or some operational segment of it has been delivered. The type of testing to be performed depends on the maturity of the product at the time. The classic hierarchy or sequence of testing is as follows:

Design review
Unit testing
Functional testing
System testing
User acceptance testing

The testing team should be involved in at least three of those phases: design review, function testing and system testing.

Functional testing involves the design, implementation and execution of test cases against the functional specification and/or functional requirements for the product. This is where the testing team measures the functional implementation against the product intent using well-formulated test cases and notes any discrepancies as defects (faults). One example is testing to ensure the Web page allows the entry of a new forum member. In that case, you are testing to ensure the Web page functions as an interface.

System testing follows much the same course (design, implement, execute and defect), but the intent or focus is very different. While functional testing focuses on discrete functional requirements, system testing focuses on the flow through the system and the connectivity between related systems. An example of that is testing to ensure the application allows the entry, activation and recovery of a new forum member. In that case, you are testing to ensure the system supports the business. There are several types of system tests; what is required for any given release should be determined by the scope:

Security
Performance
Integration

What are the minimum set of measurements and metrics?
The single most important deliverable the testing team maintains is defects. Defects are arguably the only product the testing team produces that are seen and understood by the project as a whole. This is where the faults against the system are recorded and tracked. At a minimum each defect should contain the following:

Defect name/title
Defect description: What requirement is not being met?
Detailed instructions on how to replicate the defect.
Defect severity.
Impacted functional area.
Defect author.
Status (open, in progress, fixed, closed)

This will then provide the data for a minimal set of metrics:

Number of defects raised
Distribution of defects in terms of severity
Distribution of defects in terms of functional area

From this baseline the measurements and metrics a testing organization maintains are dependent on its maturity and mission statement. The Software Engineering Institute (SEI) Process Maturity Levels apply to testing as much as they do to any software engineering discipline:

Initial: (Anarchy) Unpredictable and poorly controlled.
Repeatable: (Folklore) Repeat previously mastered tasks.
Defined: (Standards) Process characterized, fairly well understood.
Managed: (Measurement) Process measured and controlled.
Optimizing: (Optimization) Focus on process improvement.

How disciplined the testing organization needs to become and what measurements and metrics are required depend on a cost/benefit analysis conducted by the test lead/manager. What makes sense in terms of the stated goals and previous performance of the testing organization?

How to grow and maintain a testing organization?
Managing or leading a testing team is probably one of the most challenging positions in IT. The team is usually understaffed and lacks appropriate tooling and financing. Deadlines don't move, but the testing phase is continually being pressured by product delays. Motivation and retention of key testing personnel under these conditions is critical. How do you accomplish this seemly impossible task? I can only go by my personal experience both as a lead and a team member:

If the timelines are impacted, modify the test plan appropriately in terms of scope.
Clearly communicate the situation to the testing team and project management.
Keep clear lines of communication with development and project management.
Whenever possible sell, sell, sell the importance and contributions of the testing team.
Ensure the testing organization has clearly defined roles for each member of the team and a well-defined career path.
Measure and communicate the testing team's return on investment. If the detected defect would have reached the field, what would have been the cost?
Explain testing expenditures in terms of investment (ROI) not cost.
Finally, never lose your cool

How to evaluate Testing Tool

2009-03-31T10:12:00.001-07:00

Recently started testing of an enterprise grade web application developed in JAVA, that brought me round to that ever important question: How to evaluate testing tools?

After shooting this question to many senior candidates during interviews, I found out that the crowd out there has not yet given a good thought on that one. Here are the criteria for evaluating a tool/ language before you start developing automation frameworks.

1. NEVER choose a tool/ language for developing you automation framework simply because that's the only thing you know or have worked on in the past. Like I say "Don't try to use a Hammer on a screw."

2. Begin by determining what kind of user interfaces are there to test and what kind of software components your framework would have to interface with. This could range from GUI's like plain HTML,flex,win32 apps to components like Java/C API's, Databases, 3rd party services like ssh support etc etc. Ascertain whether your target tool/language can "talk" to these interfaces comfortably.

3. Determine the amount of support available for development using these languages/ tools in the domain areas required. Official/ unofficial forums, extent of search engine indexing, active developer community out there being a few of them.

4. Determine the ease of developing frameworks using these as against other choices. This is where a couple of POC's can come in handy.

5. How maintainable and scalable is your framework developed on this. Since testing frameworks are highly dynamic pieces of code that work with changing product code bases.

6. How lightweight is the tool, since you do NOT want a tool that is so heavy that impacts the performance of your test runs.

7. What is the amount of support it has for reporting and collaboration. You might want to generate reports in various formats if you want it to range from being "mailable" to your stakeholders to interfacing with your defect management and test case management software.

How to learn white box testing

2009-03-31T10:11:00.001-07:00

Q- If I have been a QA tester, and don't have white box testing, where do you learn the skill?

A- White box testing, also known as glass box or clear box testing, is testing that takes place where the testing had working knowledge of the code. In the AST BBST Foundations course, glass box testing concerns are illustrated with the question, "Does this code do what the programmer expects?" This is in contrast to the black box concern of, "Does this product fail to do what users -- human and software -- expect?"

From a learning perspective, this can mean a number of things. Testers working at this level are often comfortable with programming, hardware, networks, databases and application servers. Depending on the software, specialized knowledge of certain programming techniques or specific technologies may be required. Examples could include custom protocols, effective use of connection polling, language-specific test frameworks, among others.

Here's what I normally recommend for people getting started down this path:

Learn the basics of computer science: You can do this through schooling or through some well selected books with a good self-study ethic. I recommend looking for entry-level courses and books on computer organization, networking, databases and file systems, data structures, assemblers/compilers/interpreters, algorithms, and discrete mathematics.
Get comfortable working in a language: Today, I favor Ruby. But I taught myself Pascal as a teenager, learned C/C++ in college, and used Java in my first job as a programmer. It doesn't matter what language you learn -- just get fluent. You need to be comfortable enough that you can turn out simple programs rather easily and you need to be able understand and follow more complicated code -- even if you couldn't write it yourself.
Practice writing unit tests, stubs, and harnesses: As you learn a language, you'll want to be sure you learn how to do unit testing in that language as you progress. The reason isn't so you can write better code yourself -- that's a happy side effect -- but so you get comfortable looking at and editing those types of tests. Similarly, as you start to practice testing, you're more than likely going to encounter situations where you'll want to mock out part of your test. This is a common pastime for white box testers, as they often use custom stubs and harness to mock out parts of the component their testing or to add more visibility to the results of their testing.
Download and play with tools: This type of testing usually involves some tooling help if you plan on getting anywhere fast. Anything from your common runtime analysis tools (memory profilers, performance monitors, code coverage tools, etc.) to simple static analysis tools can help you learn the lingo and can get you thinking about common problems in white box testing. There are several good open source sites where you can find tools in this area; I currently favor opensourcetesting.org.
Learn about security testing: Get to know your buddies at the Open Web Application Security Project (OWASP). You can think of security testing as a capstone project for white box testers. Not only is it a practical application of the skills and tactics that make white box testers successful, but it pulls together everything else you're learning. If you work through the OWASP materials, you see that you'll need to understand a lot of the computer science materials. You'll need to be able to read code. You'll need to use tools -- lots of tools -- and it will be helpful to be able to write your own simple scripts. I find that OWASP makes all of their material accessible to the beginner or near beginner). They also have local groups -- perhaps in your area -- where you can get some peer support for your learning.

Testers: Time to gear up for mobile software testing

2009-03-31T10:10:00.001-07:00

The economy has many businesses retrenching or in a holding pattern -- but mobile applications designed to be accessed via smartphones or personal digital assistants (PDAs) are poised to be one of the next big things, according to many experts. If so, what impact will that have on enterprise quality assurance (QA) and testing organizations?

"Good testing practices apply regardless of the platform," said author and testing consultant Judy McKay, but she and other testing pros say mobile apps will also pose some unique challenges.

For starters, "the mobile phone is a frontier-based mentality," said William Coleman, vice president of business development at LogiGear Corp., a software testing services company in San Mateo, Calif. "There are four or five operating systems all competing for supremacy," he said, and "very loose standards … that many phone manufacturers and app developers circumvent."

But despite the lack of standards -- and the down economy -- it appears mobile app development is forging ahead. In a study released in January, Evans Data Corp. found that 94% of corporate developers expect the development of wireless enterprise applications to either increase (47.6%) or stay the same (46.4%) this year, with the Asia-Pacific region leading the growth.

For the business user, "we're seeing enterprise apps being developed on enterprise standard OSes like WinMo [Windows Mobile] and RIM, but we don't see a rush to the others just yet. Enterprises trust Microsoft and RIM," said Coleman. According to the Evans Data study, 40% more developers plan to target Windows Mobile than Apple iPhone, and 46% more plan to target .NET (compact framework) than Google's Android platform.

Not only are there multiple operating systems to take into account, but testers will also have to deal with multiple versions of an operating system, various hardware devices and form factors, and the strength of a carrier's network connections and services.

"The amount of permutations does create a significant problem for testing and time to market," said Doron Reuveni, CEO and co-founder of uTest, a Software as a Service (SaaS) marketplace for software application testing based in Southborough, Mass. "The issues in mobile apps are quite significant compared to testing Web and desktop apps."

Andrew Reshefsky is a uTest tester who has worked on mobile email encryption programs for corporate clients. A big challenge, he said, is that users are on different networks, and "you have to figure out whether the issue is because of the network or the software."

Another issue is the device itself. "Every phone has an OS; some phones can update the OS, some can't. If the problem is with the phone you have to figure that out, because software developers don't like to fix problems that don't exist [with the software]," he said.

Testers also face the challenge of shorter development cycles. "The time to market is extremely short, so usability testing is a big thing," McKay said. According to Evans Data, 40% of wireless development projects take three to six months to complete, and 60% are completed in less than six months.

McKay added that there will be "more emphasis on performance, connectivity and upgradeability."

"When you have a captive user, they're stuck with the performance you give them, but with mobile devices they expect very fast performance, but you don't have a guaranteed level of connectivity," she said. If users are unhappy with performance, "there's a higher risk that the app will be completely rejected."

uTest's Reuveni said their customers are looking for two levels of testing. First, "they want some level of certification that this works on this variety of carriers, networks, phones, etc., with this version of the software, so there's a degree of comfort when they release it," he said.

Second, he said, is to look for flaws. "In mobile there's a lot of ad hoc exploratory testing, partly due to the apps being developed so quickly, but also the customer wants real user behavior, so they're looking for a combination of flaws and feedback from real users that understand and have seen a lot of mobile apps."

Test automation will be key

To meet the challenges of testing mobile apps, automation will be key, McKay said. "Any time you do performance load you've got to use tools," she said. LogiGear's Coleman added, "As complexity [increases], the requirements for testing get more demanding. The only way to capture that is through automation."

LogiGear specializes in test automation, and Coleman said they are working in the mobile area. "The big hole in this space is automation for mobile phones," he said.

It's a nascent area, said Manish Mathuria, founder and chief technology officer of InfoStretch Corp., an outsourcer of quality assurance, test automation, software development and mobile testing services. "One reason is the form factors of the different devices in use vary so widely; for an automation tool to cover all of them reliably is not an easy task," he said.

In terms of adopting automation, Mathuria said, QA teams have a long way to go as well: "Enterprise QA teams are sometimes barely equipped to manage automation well on the desktop side."

Mathuria said that today the great majority of mobile apps his company sees are consumer-facing, with games being the biggest area, followed by utilities, but everyone interviewed for this article agreed that the enterprise is likely to follow. "The smartphone is the new computer abstraction," Coleman said. "Ten years from now we won't be using laptops."

Are testers prepared for the mobile revolution? "Testers tend to be behind the development curve," McKay said. "Developers are out experimenting with new things, but testers may be deeply involved in the last project."

Her advice? "Keep an eye on what developers are doing and thinking, and the ops people. [Testers] can no longer pretend they don't need to know about performance of load and connectivity. There is a lot of expertise that will have to be developed, and this will be the time to get better."

Cloud computing creates software testing challenges

2009-02-05T05:17:00.000-08:00

The "cloud" promises to create new opportunities for enterprise developers as well as for suppliers offering services and tools for this new paradigm. For testing organizations, there will be both new challenges and new tools for answering what Soasta CEO Tom Lounibos calls the one key question: Can I go live?

The impact on testing is that the end-user experience is being influenced by the cloud provider and all other parties involved.
Vik Chaudhary
VP of product management and corporate development, Keynote Systems Inc.

"Testing all the layers — from your application to the cloud service provider — is something testers will have to become efficient in," said Vik Chaudhary, vice president of product management and corporate development at Keynote Systems Inc. in San Mateo, Calif.
According to market research firm IDC, spending on IT cloud services is expected to grow nearly threefold, to $42 billion by 2012. Cloud computing will also account for 25% of IT spending growth in 2012 and nearly a third of the IT spending growth in 2013, IDC projected.
IDC makes a distinction between "cloud services" and "cloud computing." Cloud services, according to the market research firm, are "consumer and business products, services, and solutions that are delivered and consumed in real-time over the Internet." In contrast, cloud computing as defined by IDC is the infrastructure or "stack" for development and deployment that enables the "real-time delivery of products, services, and solutions over the Internet."
Chaudhary explains the shift: "Enterprises like Schwab, Travelocity, etc. have been deploying their own data centers for years. The challenge was to manage highly scalable applications and how to ensure the best experience. Legions of people were employed by these companies to monitor/test/add servers, etc."
What's happening more recently with new cloud infrastructure like Google App Engine, he said, is that organizations can run their applications on Google's infrastructure.
"That means the bar to deploy applications in the cloud is so much lower. You don't have to have data centers or ops teams; you can focus on building the application and the functionality. It's a paradigm shift in application development," he said.
It's a shift for the tester, too. For example, Chaudhary said, "If you build an application and you use the BlackBerry to access a manufacturing application hosted by a cloud company like Salesforce, Salesforce does a certain amount of testing, to ensure the server is available, etc. But when it comes to the application itself, does it run on two phones or 50 phones? Do you have a long page to load?"
In addition, the cloud hosting company may use a third-party service to speed performance. "The impact on testing is that the end-user experience is being influenced by my company, by the cloud provider, and all other parties involved," he said.
Reducing testing costs
While Lounibos said Mountain View, Calif.-based Soasta Inc. has a growing group of customers that don't own servers and do everything in the cloud, "the majority are still more traditional; they use managed service providers and are dabbling in the cloud." However, he said, cloud-based testing is a way for organizations to learn the cloud and reduce the costs of testing at the same time.
"Traditional customers see testing as a money pit. They're looking for ways to reduce costs. The [main] argument for cloud computing for the enterprise is, is it reliable enough," he said. "This is not so for testing. Testing [in the cloud] just replicates the real world; it doesn't have the issues associated with production, but it has the benefits of cost reduction."
With cloud computing, Lounibos said, testers "have access, availability, and affordability to enormous amounts of computing power, which is what's needed in testing. The idea of being able to provision 125 servers in 5 to 8 minutes and only pay for the hours you test is so compelling. You no longer have to have huge test labs for Web applications."
Soasta's CloudTest, for example, is available as an on-demand virtual test lab in the cloud or as an appliance. It supports load, performance, functional, and Web UI/Ajax testing. According to Lounibos, "We were built on top of the cloud for the cloud."
For its part, Keynote offers KITE (Keynote Internet Testing Environment) for testing and analyzing the performance of Web applications across the Internet cloud. KITE offers instant testing from the desktop as well as from a variety of geographic locations.
For Internet applications in particular, Chaudhary said performance testing needs to move to the cloud.
"When it comes to performance, you're not depending just on the application but on all the providers [involved]. And do you [the user] have DSL or a dialup line, or a mobile device? Performance testing by nature is environmental," he said.
For mobile applications, Chaudhary said both performance and functional testing should move to the cloud.
"For mobile applications, the functional testing also depends on the providers. Say you've got a screen for login. The size of the page and the screen on phone and the provider can all affect if the application works," he said.
By testing in the cloud, Chaudhary added, organizations can more easily and cost-effectively test for hundreds of devices.
With applications that run on the cloud, "you need to test network performance, server performance, database performance, software performance on the application, and how it's cached in the client," said Dennis Drogseth, a vice president at market research company Enterprise Management Associates Inc., based in Boulder, Colo. "If you have a single application that runs in one place, you can test it geographically in one place. What you have with an Amazon or Facebook, for example, is all kinds of pieces coming in from different geographies, and you can't know ahead of time where they'll be. It's definitely more complicated than running a test script on a single server-based application."
The challenge is to run tests across all the diverse components and geographies to identify problems, he said, and organizations that develop an application "typically don't have access to those types of environments. So [a company like] Keynote is giving those testers a working environment where they leverage the Internet cloud and all the vagaries and look at real networks and desktops."
New testing tools needed
Drogseth said new types of testing tools will be needed. "You can't do cloud computing with application development testing tools for a LAN or a single server. You need tools to allow you to understand the network and desktop implications and all the pieces. You need to bring the network to the developer."
"I suspect over the next five years every testing vendor will try to come to the cloud. I think we will have a new generation of testing companies," said Lounibos. "This [cloud computing] is a big market coming down road; it's just the way we'll consume services."

Software Testing Advice for Novice Testers

2009-02-05T05:15:00.000-08:00

Novice testers have many questions about software testing and the actual work that they are going to perform. As novice testers, you should be aware of certain facts in the software testing profession. The tips below will certainly help to advance you in your software-testing career. These ‘testing truths’ are applicable to and helpful for experienced testing professionals as well. Apply each and every testing truth mentioned below in your career and you will never regret what you do.
Know Your Application
Don’t start testing without understanding the requirements. If you test without knowledge of the requirements, you will not be able to determine if a program is functioning as designed and you will not be able to tell if required functionality is missing. Clear knowledge of requirements, before starting testing, is a must for any tester.
Know Your Domain
As I have said many times, you should acquire a thorough knowledge of the domain on which you are working. Knowing the domain will help you suggest good bug solutions. Your test manager will appreciate your suggestions, if you have valid points to make. Don’t stop by only logging the bug. Provide solutions as well. Good domain knowledge will also help you to design better test cases with maximum test coverage. For more guidance on acquiring domain knowledge.
No Assumptions In Testing
Don’t start testing with the assumption that there will be no errors. As a tester, you should always be looking for errors.
Learn New Technologies
No doubt, old testing techniques still play a vital role in day-to-day testing, but try to introduce new testing procedures that work for you. Don’t rely on book knowledge. Be practical. Your new testing ideas may work amazingly for you.
You Can’t Guarantee a Bug Free Application
No matter how much testing you perform, you can’t guarantee a 100% bug free application. There are some constraints that may force your team to advance a product to the next level, knowing some common or low priority issues remain. Try to explore as many bugs as you can, but prioritize your efforts on basic and crucial functions. Put your best efforts doing good work.
Think Like An End User
This is my top piece of advice. Don’t think only like a technical guy. Think like customers or end users. Also, always think beyond your end users. Test your application as an end user. Think how an end user will be using your application. Technical plus end user thinking will assure that your application is user friendly and will pass acceptance tests easily. This was the first advice to me from my test manager when I was a novice tester.
100% Test Coverage Is Not Possible
Don’t obsess about 100% test coverage. There are millions of inputs and test combinations that are simply impossible to cover. Use techniques like boundary value analysis and equivalence partitioning testing to limit your test cases to manageable sizes.
Build Good Relations With Developers
As a tester, you communicate with many other team members, especially developers. There are many situations where tester and developer may not agree on certain points. It will take your skill to handle such situations without harming a good relationship with the developer. If you are wrong, admit it. If you are right, be diplomatic. Don’t take it personally. After all, it is a profession, and you both want a good product.
Learn From Mistakes
As a novice, you will make mistakes. If you don’t make mistakes, you are not testing hard enough! You will learn things as you get experience. Use these mistakes as your learning experience. Try not to repeat the same mistakes. It hurts when the client files any bug in an application tested by you. It is definitely an embracing situation for you and cannot be avoided. However, don’t beat yourself up. Find the root cause of the failure. Try to find out why you didn’t find that bug, and avoid the same mistake in the future. If required, change some testing procedures you are following.
Don’t Underestimate Yourself if Some of Your bugs Are Not Fixed
Some testers have assumptions that all bugs logged by them should get fixed. It is a good point to a certain level but you must be flexible according to the situation. All bugs may or may not be fixed. Management can defer bugs to fix later as some bugs have low priority, low severity or no time to fix. Over time you will also learn which bugs can be deferred until the next release.

SQL Injection – How to Test Web Applications against SQL Injection Attacks

2009-02-05T05:14:00.000-08:00

Security testing of web applications against SQL Injection, explained with simple examples - By Inder P Singh.

Many applications use some type of a database. An application under test might have a user interface that accepts user input that is used to perform the following tasks:

1. Show the relevant stored data to the user e.g. the application checks the credentials of the user using the log in information entered by the user and exposes only the relevant functionality and data to the user

2. Save the data entered by the user to the database e.g. once the user fills up a form and submits it, the application proceeds to save the data to the database; this data is then made available to the user in the same session as well as in subsequent sessions

Some of the user inputs might be used in framing SQL statements that are then executed by the application on the database. It is possible for an application NOT to handle the inputs given by the user properly. If this is the case, a malicious user could provide unexpected inputs to the application that are then used to frame and execute SQL statements on the database. This is called SQL injection. The consequences of such an action could be alarming.

The following things might result from SQL injection:

1. The user could log in to the application as another user, even as an administrator.

2. The user could view private information belonging to other users e.g. details of other users’ profiles, their transaction details etc.

3. The user could change application configuration information and the data of the other users.

4. The user could modify the structure of the database; even delete tables in the application database.

5. The user could take control of the database server and execute commands on it at will.

Since the consequences of allowing the SQL injection technique could be severe, it follows that SQL injection should be tested during the security testing of an application. Now with an overview of the SQL injection technique, let us understand a few practical examples of SQL injection.

Important: The SQL injection problem should be tested only in the test environment.

If the application has a log in page, it is possible that the application uses a dynamic SQL such as statement below. This statement is expected to return at least a single row with the user details from the Users table as the result set when there is a row with the user name and password entered in the SQL statement.

SELECT * FROM Users WHERE User_Name = ‘” & strUserName & “‘ AND Password = ‘” & strPassword & “’;”

If the tester would enter John as the strUserName (in the textbox for user name) and Smith as strPassword (in the textbox for password), the above SQL statement would become:

SELECT * FROM Users WHERE User_Name = ‘John’ AND Password = ‘Smith’;

If the tester would enter John’– as strUserName and no strPassword, the SQL statement would become:

SELECT * FROM Users WHERE User_Name = ‘John’– AND Password = ‘Smith’;

Note that the part of the SQL statement after John is turned into a comment. If there were any user with the user name of John in the Users table, the application could allow the tester to log in as the user John. The tester could now view the private information of the user John.

What if the tester does not know the name of any existing user of the application? In such a case, the tester could try common user names like admin, administrator and sysadmin. If none of these users exist in the database, the tester could enter John’ or ‘x’=’x as strUserName and Smith’ or ‘x’=’x as strPassword. This would cause the SQL statement to become like the one below.

SELECT * FROM Users WHERE User_Name = ‘John’ or ‘x’='x’ AND Password = ‘Smith’ or ‘x’=’x’;

Since ‘x’=’x’ condition is always true, the result set would consist of all the rows in the Users table. The application could allow the tester to log in as the first user in the Users table.

Important: The tester should request the database administrator or the developer to copy the table in question before attempting the following SQL injection.

If the tester would enter John’; DROP table users_details;’—as strUserName and anything as strPassword, the SQL statement would become like the one below.

SELECT * FROM Users WHERE User_Name = ‘John’; DROP table users_details;’ –‘ AND Password = ‘Smith’;

This statement could cause the table “users_details” to be permanently deleted from the database.

Though the above examples deal with using the SQL injection technique only the log in page, the tester should test this technique on all the pages of the application that accept user input in textual format e.g. search pages, feedback pages etc.

SQL injection might be possible in applications that use SSL. Even a firewall might not be able to protect the application against the SQL injection technique.

I have tried to explain the SQL injection technique in a simple form. I would like to re-iterate that SQL injection should be tested only in a test environment and not in the development environment, production environment or any other environment. Instead of manually testing whether the application is vulnerable to SQL injection or not, one could use a web vulnerability scanner that checks for SQL injection.

Improving software testing skills and manual vs. automated testing

2009-01-30T04:29:00.001-08:00

Q- I am working as software tester in India. I want to become more efficient in this field. Please kindly tell me what to do for that? Could you also give me some tips for manual testing and tell me what you think is best, manual or automated testing?

There are many ways to learn about software testing, including reading books and articles, attending training classes, listening to webcasts and hands-on work in the field. But there is no one answer book or one path to learn software testing. Consider, too, that there are many different types of software and what works well for one type of software may not work at all for another type. I'm using the word "type" to identify financial applications, e-commerce Web sites, medical devices, embedded software, manufacturing applications and so forth.

Tools and techniques that work in one environment might not work at all with a different type of application. I point this out not to confuse you or to be obscure, but as a reminder that as you learn what makes sense for what you are testing now, it would be best not to become too attached or rigid in your thinking about testing software. Even if you remain testing one type of software, technology changes and oftentimes our tools and techniques have to change as well.

These warnings aside, there are two books in the software testing field I often recommend:

Testing Computer Software, 2nd Edition
By Cem Kaner, Jack Falk, Hung Q. Nguyen
ISBN-13: 978-0471358466
Lessons Learned in Software Testing
By Cem Kaner, James Bach, Bret Pettichord
ISBN-13: 978-047108112

Keep reading and keep learning. As James Bach suggested in his tutorial on Self-Education for Testers, build your own curriculum. Find the books, articles, Web sites and blogs that help you learn. Build your own educational plan. James does a great job of pointing out how to survey a book. I interpreted his idea this way -- sometimes we want to learn in detail and sometimes we just need to know a little bit about a topic. For instance, if you decide you want to learn more about relational databases, investigate books at your local bookstore or library. You can also go online for continued research. Surveying multiple topics will help you choose what topics to you want to learn in-depth and what topics you want to gain an awareness of.

Additionally, I recommend finding opportunities to work with the technology you want to know more about. You can read, research via the internet, attend training classes and read tech magazines, but nothing compares to live experience. Seek work opportunities that help you grow.

Look at your current position and determine if there are ways to increase your knowledge with the opportunities that you already have access to. I recommend a mix of learning -- learn technical skills about current technology and learn about testing practices. If you vary your reading, your own education will become more full-bodied and varied. I wouldn't recommend trying to learn one area -- say, Java -- in detail without learning more about requirements building or managing a defect process. So many topics are important in the field of software testing that it's nearly impossible to create one list and say, "Here, go and read these books." But I do believe it's essential to keep reading and keep learning. I generally plan several study times for myself each week.

Automated vs. manual testing
Regarding your question about which form of testing is best, manual or automated. There is no one right answer. Here are a few considerations to help you make the assessment whether a task is worth automating.

Will the functionality need to be tested multiple times due to multiple internal builds before production release?
Will the functionality need to be tested on different operating systems and/or on different browsers?
Will the functionality need to be tested with many different types of data?
If automation is built, will it be used for one product release or for many releases in the future?
Does your company have sufficient staff to bring a tool in-house?
Do you have or can you gain sufficient knowledge of the tool and automation techniques in general to put a tool to good use?

--
Thanks and Regards,
Venu Naik Bhukya.

Guidelines and Checklist for Website Testing

2009-01-30T04:13:00.000-08:00

Worldwide web is browsed by customers with different knowledge levels, while testing websites (static/Dynamic) QA department should concentrate on various aspects to make effective presentation of website in www.

Aspects to cover

Functionality

Usability

User Interface

Serverside Interface

Compatibility

Security

Performance

Description

Functionality

1.1 1.1 Links

Objective is to check for all the links in the website

1.1.1 1.1.1All Hyperlinks

1.1.2 1.1.2All Internal links

1.1.3 1.1.3 All External links

1.1.4 1.1.4 All Mail links

1.1.5 1.1.5Check for orphan pages

1.1.6 1.1.6 Check for Broken links

1.2Forms

Check for the integrity of submission of all forms

1.1.7 1.2.1All Field level Checks

1.1.8 1.2.2 All Field level Validations

1.1.9 1.2.3Functionality of create,modify,delete and view

1.1.1 1.2.4Handling of wrong inputs

1.1.1 1.2.5Default values if any (standard)

1.1.1 1.2.6Optional Versus Mandatory fields

1.3 Cookies

1.3.1Check for the cookies that has to be enabled and how it has to be expired

1.4 Web Indexing

Depending on how the site is designed using metatags, frames, HTMLsyntax, dynamically created pages, passwords or different languages, our site will be searchable in different ways

1.4.1 1.4.1Meta Tags

1.4.2 1.4.2Frames

1.4.3 1.4.3HTML syntax

1.5 Database

Two types of errors that may occur in web application

a) 1.5.1 Data Integrity : Missing or wrong data in table

b) 1.5.2Output Errors : Errors In writing ,editing or reading operation in the table

Usability

How simple customer can browse the website

2.1 Navigation

Navigation describes the way user navigate with in a webpage, between different user interface controls (buttons, text boxes, combo boxes, dropdown lists ...etc)

2.1.1 Application navigation is proper through tab (key board)

2.1.2 Application navigation through mouse

2.1.3 Main features accessible from the main/home page (mother window)

2.1.4 Any hotkeys, control keys to access menus

2.2 Content

Correctness is whether the information is truthful or contains misinformation. The accuracy of the information is whether it is without grammatical or spelling errors. Remove irrelevant information from your site this may otherwise cause misunderstanding or confusion

2.2.1 Spelling and grammar

2.2.2 Updated information (contact details mail IDs help reports)

2.3 General appearance

2.3.1 Page appearance

2.3.2 Colour, font size

2.3.3 Frames

2.3.4 Consistent Designs

2.3.5 Symbols and logos (localization)

User Interface

All about how exactly website looks like (view)

Basic guidelines for web User interface (Microsoft)

3.1 Verify whether screen resolution been taken in to account while using browser, will the UI resize itself as you maximize/minimize this must be tested with various screen resolutions

3.2 verify whether the number of controls per page have been checked, generally 10 is a good number

3.3 Where there are options to choose from, user should be forced to choose from a set of radio buttons and the default should be pointing at default message (none, select one..etc)

3.4 Every dropdown box should have the first choice as NONE (it could be any other meaningful sentence such as Choose one or select

3.5 Ensure persistency of all values, when you enter values in a form, and move on the next page and then return to change one of the previous screen .The users must not be forced to re-enter all the values once again, this must be checked and ensured

3.6 Horizontal scrolling is not preferable in general .Avoid using horizontal scroll bar Ensure that the use of vertical scroll bar is judicious

3.7 Consider the use of pagination where appropriate.

3.8 Ensure that shift click control click work in list boxes, clarify these features work in both browsers

3.9 The location of buttons (OK and Cancel) should be at the right hand bottom side of a screen and consistent.

3.10 Clarify whether encryption of the password occurs from your login page all the way to the backend (the login page must not transmit clear text password)

3.11 Illegal operations should give popup messages (message should be simple and clear) 3.12 Verify if there is a requirement to use image maps in the application (does net scape support this well

3.13 positive popup messages should be displayed (submitted, deleted, updated, done and cleared)

3.14 Ensure that you have multiple check boxes when multiple selections is to be performed

3.15 Avoid long scrolling drop down list make it short

3.16 website URL should be small and simple

Serverside Interface

4.1 Serverside Interface

4.1.1 Verify that communication is done correctly, web server-application server, application server-database sever and vice versa

4.1.2 Compatibility of server software, hardware, network connections

4.1.3 Database compatibility

4.1.4 External interface if any

Client side Compatibility

5.1 platforms

Check for the website compatibility with

5.1.1 Windows (95, 98, 2000, NT)

5.1.2 Unix

5.1.3 Linux

5.1.4 Macintosh (if applicable)

5.1.5 Solaris (if applicable)

5.2 Browsers

5.2.1 Internet Explorer (3.x, 4.x, 5.x)

5.2.2 Netscape Navigator (3.x, 4.x, 6.x)

5.2.3 AOL

5.2.4 Browser settings (security settings graphics, java etc..)

5.2.5 Frames and cascade style sheets

5.2.6 HTML specification

5.3 Graphics

Loading of images graphics, etc.

Security

6.1 Valid and invalid login

6.2 limits defined for the number of tries

6.3 Can it be bypassed by typing URL to a page inside directly in the browser?

6.4 verify log files are maintained to store the information for traceability

6.5 verify encryption is done correctly if SSL is used (if applicable)

6.6 No access to edit scripts on the server without authorization

Performance

7.1 Connection speed

7.1.1 Try with different connection speeds (14.4, 28.8, 56.6, ISDN, cable DSL, T1, T3)

7.2 Load

7.1.1 Perform load test as per the SLA (Service Level Agreement)

7.1.2 What is the estimated number of users per time period and how will it be divided over the period

7.1.3 Will there be peak loads and how systems react

7.1.4 Can your site handle a large amount of users requesting a certain page?

7.1.5 Is large amount of data transferred from user?

7.3 stress

7.3.1 Stress testing is done in order to actually break a site or certain features to determine how the system reacts

7.3.2 Stress tests are designed to push and test system limitations and determine whether the system recovers gracefully from crashes

Note: Hackers often stress systems by providing loads of wrong in-data until it crash and then gain access to it during start-up

7.3.3 System abnormal conditions (stress testing)

Less bandwidth of cable

Low disk memory

Low processor speed

7.4 Soke test

7.4.1 Is the application or certain features going to be used only during certain periods of time or will it be used continuously 24X7

7.4.2 Will downtime be allowed or is that out of the question

7.4.2 Verify that the application is able to meet the requirements and does not run out of memory or disk space

--
Thanks and Regards,
Venu Naik Bhukya.

Tools, methods to test software more efficiently

2009-01-30T04:00:00.001-08:00

Several classes of testing tools are available today that make the testing process easier, more effective and more productive. When properly implemented, these tools can provide a test organization with substantial gains in testing efficiency.

However, test tools need to fit into the overall testing architecture and should be viewed as process enablers -- not as the "answer." Test organizations will often look to tools such as reviews, test management, test design, test automation and defect tracking to facilitate the process. It is quite common for a testing tool or family of tools to address one or more of those needs, but for convenience they will be addressed from a functional perspective not a "package" perspective.

It is important to note that, as with any tool, improper implementation or ad-hoc implementation of a testing tool can negatively impact the testing organization. Ensure a rigorous selection process is adhered to when selecting any tool and do such things needs analysis, on-site evaluation, and an assessment of return on investment (ROI).

Reviews
Reviews and technical inspections are the most cost-effective to way detect and eliminate defects in any project. This is also one of the most underutilized testing techniques. Consequently there are very few tools available to meet this need. Any test organization that is beginning to realize the benefits of reviews and inspections but is encountering scheduling issues between participants should look to an online collaboration tool. There are several tools available for the review and update of documents, but only one that I'm aware of that actually addresses the science and discipline of reviews -- ReviewPro by Software Development Technologies. I normally do not "plug" a particular tool, but when the landscape is sparse I believe some recognition is in order.

Test management
Test management encompasses a broad range of activities and deliverables. The test management aid or set of aids selected by an organization should integrate smoothly with any communication (email, network, etc.) and automation tools that will be applied during the testing effort. Generic management tools will often address the management of resources, schedules and testing milestones but not the activities and deliverables specific to the testing effort -- test requirements, test cases, test results and analysis.

Test requirements: Requirements or test requirements often become the responsibility of the testing organization. For any set of requirements to be useful to the testing organization, it must be maintainable, testable, consistent and traceable to the appropriate test cases. The requirements management tool must be able to fulfill those needs within the context of the testing team's operational environment.

Test cases: The testing organization is responsible for authoring and maintaining test cases. A test case authoring and management tool should enable the test organization to catalogue, author, maintain, manually execute and auto-execute automated tests. These test cases need to be traceable to the appropriate requirements and the results recorded in such a manner as to support coverage analysis. Key integration requirements when looking to test case management tools: Does it integrate with the test requirement aide? Does it integrate with the test automation tools being used? Will it support coverage analysis?

Test results and analysis: A test management suite of tools and aids needs to be able to report on several aspects of the testing effort. There is an immediate requirement for test case results -- which test case steps passed and which failed? There will be periodic status reports that will address several aspects of the testing effort: test cases executed/not executed, test cases passed/failed, requirements tested/not tested, requirements verified/not verified, and coverage analysis. The reporting mechanism should also support the creation of custom or ad-hoc reports that are required by the test organization.

Test automation
Several test automation frameworks have been implemented over the years by commercial vendors and testing: record & playback, extended record and playback, and load/performance.

Record and playback: Record and playback frameworks were the first commercially successful testing solutions. The tool simply records a series of steps or actions against the application and allows a playback of the recording to verify that the behavior of the application has not changed.

Extended record and playback: It became quickly apparent that a simple record and playback paradigm was not very effective and did not make test automation available to non-technical users. Vendors extended the record and playback framework to make the solution more robust and transparent. These extensions included data-driven, keyword and component-based solutions.

Load/performance: Load/performance test frameworks provide a mechanism to simulate transactions against the application being tested and to measure the behavior of the application while it is under this simulated load. The load/performance tool enables the tester to load, measure and control the application.

Defect tracking
The primary purpose of testing is to detect defects in the application before it is released into production; furthermore, defects are arguably the only product the testing team produces that is seen by the project team. The defect management tools must enable the test organization to author, track, maintain, trace to test cases, and trace to test requirements any defects found during the testing effort. The defect management tool also needs to support both scheduled and ad-hoc analysis and reporting on defects.

Web application testing: The difference between black, gray and white box testing

2009-01-30T03:06:00.001-08:00

Web application testing: The difference between black, gray and white box testing

Black, white and gray box tests provide different approaches for assessing the security of Web applications. Each approach has specific advantages and disadvantages, and selecting a testing approach needs to be done based on the time and resources available, as well as the overall goals of the test being performed.

You can assume most real-world attackers will approach systems from a black-box perspective. But to better account for the advantage attackers have with regard to time and resources, and to avoid relying on security through obscurity, gray and white box tests can be appropriate approaches as well. Maximizing the security value of testing approaches when you have limited time and resources requires careful test planning and a thorough understanding of how testing constraints affect the completeness of testing results.

Let's take a look at the differences between the three tests.

Black box testing
Black box testing refers to testing a system without having specific knowledge to the internal workings of the system, no access to the source code, and no knowledge of the architecture.

In essence, this approach most closely mimics how an attacker typically approaches applications. However, due to the lack of internal application knowledge, the uncovering of bugs and/or vulnerabilities can take significantly longer. Black box tests must be attempted against running instances of applications, so black box testing is typically limited to dynamic analysis such as running automated scanning tools and manual penetration testing.

White box testing
White box testing, which is also known as clear box testing, refers to testing a system with full knowledge and access to all source code and architecture documents. Having full access to this information can reveal bugs and vulnerabilities more quickly than the "trial and error" method of black box testing. Additionally, you can be sure to get more complete testing coverage by knowing exactly what you have to test.

However, because of the sheer complexity of architectures and volume of source code, white box testing introduces challenges regarding how to best focus the testing and analysis efforts. Also, specialized knowledge and tools are typically required to assist with white box testing, such as debuggers and source code analyzers.

In addition, if white box testing is performed using only static analysis techniques using the application source code and without access to a running system, it can be impossible for security analysts to identify flaws in applications that are based on system misconfigurations or other issues that exist only in a deployment environment of the application in question.

Gray box testing
When we talk about gray box testing, we're talking about testing a system while having at least some knowledge of the internals of a system. This knowledge is usually constrained to detailed design documents and architecture diagrams. It is a combination of both black and white box testing, and combines aspects of each.

Gray box testing allows security analysts to run automated and manual penetration tests against a target application. And it allows those analysts to focus and prioritize their efforts based on superior knowledge of the target system. This increased knowledge can result in more significant vulnerabilities being identified with a significantly lower degree of effort and can be a sensible way for analysts to better approximate certain advantages attackers have versus security professionals when assessing applications.

Selecting a testing methodology
The testing approach you use should depend on a number of factors, including time allocated to the assessment, access to internal application resources and goals of the test.

Tests intended to best approximate short-term efforts of attackers with limited resources can be conducted using black box methodologies.

If the test is intended to reflect longer-term efforts by attackers who have more significant resources, gray box tests can help to reflect knowledge that attackers might learn about application internals without requiring the assessment team to expend the full amount of resources that would be available to attackers.

Teams that need to make the most insightful and far-reaching recommendations about applications within a limited amount of time should use white or clear box testing.

Security testers should be flexible and be able to plan a test approach for any of these scenarios given the time and access to resources available for an application. By meshing the availability of assessment time and testing resources with the overall goals of the testing, analysts can select a testing methodology that will maximize the security benefits of the findings within the given constraints. Given an understanding that time and testing resources favor attackers in the wild, assessments teams should optimize their activities accordingly.

Designing test cases using Cause-Effect Graphing Technique

2009-01-09T10:15:00.001-08:00

Q- Can you explain how you can design test cases using the Cause-Effect Graphing Technique?

A- The Cause-Effect Testing Technique is another of several efforts for mapping input to output/response. In the Cause-Effect Graphing Technique, input and output are modeled as simple text, such as this:

Cause --> intermediate mode --> effect

See the Wikipedia article "Cause-effect graph" for additional information.

Much more information about the Cause-Effect Graphing Technique can be found in the "Cause-Effect Graphing User Guide", which is an entire PDF for the use of Bender RBT testing tool. (Note that this manual was written in 2006, so it might be somewhat out of date.)

The goal of cause-effect graphing is to reduce the number of test cases run. Many input/modifiers/effect combinations have the same output and exercise the same code in the system under test. By reducing the combinations, fewer tests can be run while still achieving the same confidence in quality.

A somewhat related modeling concept is PICT -- Pairwise Interdependent Combinatorial Testing. This technique goes a step beyond cause-effect and actually applies combinatorial theory to reduce the number of combinations needed to be tested.

Each theory has its application. In a highly related system where cause and intermediate effect are tightly coupled, it might be a challenge to model the system in PICT (although some PICT tools, including Microsoft's free PICT modeling tool, do allow you to weight various factors as well as create combinations).

My concern about cause-effect is simple: How could you model cause-effect on a high-availability Web service with 30 methods, each containing 40 or 50 parameters? By manually identifying each cause and effect, you'll spend so much time detailing the test that I'm not convinced you could actually test the application. This is why tools like Bender RBT exist -- to help automate this process.

My biggest complaint about a methodology like cause-effect is its lack of real-world application. I doubt your manager would tolerate your spending five or six days modeling your application into a cause-effect graph. Your manager is more likely to expect you to apply good testing sense and develop cases quickly, but keeping an eye on critical combinations. Still, further research is warranted, and if you can efficiently apply this model to your efforts, you'll be rewarded with higher quality at a reduced input cost.

How to test Web services

2009-01-05T10:05:00.001-08:00

In many ways testing a Web service is no different than testing anything else. You still do all the same steps of the test method, but some of them can take on a bit of a different flavor. Here are a couple of tidbits I've picked up over the last couple of years testing Web services.

Determining coverage and oracles
When testing Web services, I find that scheme coverage becomes important. You still probably care about all the same coverage you cared about with non-Web service testing (scenario coverage, requirements coverage, code coverage, and application data coverage), but now you get to add schema coverage to the list.

For me, schema means not only performing a schema validation against all the response XMLs that come back from the service you're testing, it also includes testing for the right number of maximum and minimum repeating elements, correct reference identification tag values, and other subtle nuances for XML. This can sometimes create an oracle problem. I find that I'm often comparing against a schema along with a mapping document of some sort. A mapping document tells me what data should be stored in what element and when it should be there.

Determining the test procedures
One of the initial problems to be solved when performing Web service testing is management of all the data files (typically XML, but not always). There are tools that do this file management for you (I'll cover some of them briefly below), but you should have a game plan for files that need to be managed manually. When I'm testing I typically track three files: a request file, an expected response file, and an actual response file.

The naming convention I've picked up along the way is to put a "-rq.xml", "-rs.xml", or "-result.xml" on the end of all my files. That way for each file name you have a set of XMLs that paint the entire picture for that test case. Once you get a naming convention worked out, get the files into source control if you don't have a tool that keeps them all straight for you.

Once you have all these XMLs (sometimes hundreds, sometimes thousands), now you get the joy of keeping them up to date. I've found that schema changes can happen quite often on a project, as can mapping changes. Whenever one of those changes occur, you get the pleasure of updating all those XML files. And remember, if you have 100 test cases, you have 200 files -- assuming you don't bother to update your actual result file since you'll be re-running the test case.

The way I handle most schema changes is to have a Ruby script handy that I can use to make the updates for me programmatically. If you do it enough times, you eventually build a library of scripts for just about every type of change that comes your way. There are some exceptions to that where you may need to do some changes manually, but I find those changes don't normally affect all the test cases, just a subset.

Operating the test system
There are a lot of great tools available for testing web services. MindReef SoapScope is a fine commercial option, and SoapUI is a fine open source option. I've found them both to have all the basic features I've needed. SoapScope is a bit better about data trending and analysis. SoapUI has a few more technical features for test execution. I've also spent some time using IBM Rational for SOA Quality, which is an Eclipse-based tool focused on Web service testing. It's a latecomer, but has a nice feature set.

More often then not, I find that I use homegrown tools written in Ruby and Java to perform Web service testing. It gives you more control over the interfaces, features, and a working knowledge of what's actually going on in the test tool you're using. There are some drawbacks like support and documentation. Even the open source SoapUI team does a fantastic job in support -- they turned around a defect for me in 24 hours once. That's better then you'll see from MindReef or IBM I'm sure.

Try a couple of tools before you settle. I've found that I switch between the different tools based on the team I'm working with and what we're tying to do. Once you've compared a couple of them you'll figure out which features you really need and which are nice but optional.

Evaluating the test results
Evaluating the results for Web service tests can sometimes be really easy, and sometimes painful. There are couple of things you'll want to practice getting good at:

writing XSLTs to transform your actual response to mask out values you don't care about (server dates for example);
writing Xpath queries to check for specific values in an XML document;
learning all the command line options on your favorite diff-tool;
and ensuring you have at least one person on the team that knows the schema inside and out and can see the entire mapping document in their head when they look at the response files.

The better you are at the first three, the less important the last one is. I've also found that custom logging (both in your test execution tool and in the Web service under test) can also help add visibility to the results. Depending on what you're testing, sometimes you can cut out the need for file comparison entirely.

Once you get up to speed with the basics of manually testing the Web service, performance testing is normally trivial. Some of the tools have the ability to generate load built in. If you're using a homegrown option, all you need to do is thread your requests and you have the same ability. Getting the usage model right can sometimes be a challenge, but it's doable.

Other concerns that come into play can be authentication/authorization and encryption. I've not personally had to do a lot there, but I know that it's a problem for some. I imagine that if you're testing a payment gateway like your questions states, you'll need to do some investigation around what that means for your data and the tools you can use.

How to evaluate testing software and tools

2009-01-05T10:02:00.001-08:00

Once a testing organization reaches a certain size, level of maturity or workload, the requirement to purchase/build testing software or aides becomes apparent. There are several classes of testing tools available today that make the testing process easier, more effective and more productive. Choosing the appropriate tool to meet the testing organization's long-term and short-term goals can be a challenging and frustrating process. But following a few simple guidelines and applying a common-sense approach to software acquisition and implementation will lead to a successful implementation of the appropriate tool and a real return on investment (ROI).

One of the simplest questions to ask when looking at testing software is "What is ROI?" The simplest answer is "Anything that reduces the resource (people) hours required to accomplish any given task." Testing tools should be brought into an organization to improve the efficiency of a proven testing process. The value of the actual process has already been established within the organization or within the industry.

Example: Test management
The organization has meticulously tracked test requirements and test cases using spreadsheets, but it finds this to be a cumbersome process as the test organization grows. It has been shown that this process has reduced the number of defects reaching the field, but the cost of maintaining the approach now impacts its effectiveness. Solution: Invest in a test management tool or suite of tools.

Example: Test automation
The organization has created a suite of manual test cases using a text editor, but it finds it difficult to maintain, use and execute these test cases efficiently as the test organization's role grows. The test cases have proven effective in detecting defects before they reach production, but the time required to manage and execute these test cases now impacts the ROI. Solution: Invest in a test automation tool or suite of tools.

Example: Defect Management
The test organization has implemented a defect tracking process using email and a relational database, but it now finds that defects are being duplicated and mishandled as the volume of defects grows. Solution: Upgrade the current in-house solution or invest in a defect management tool.

Needs analysis

The first thing an organization must accomplish is catalogue what needs or requirements the testing software is expected satisfy. For an organization that is new to the acquisition process, this can be an intimidating exercise. There are three categories or "points of view" that must be addressed: management/organization, test architecture and end user.

Needs analysis: Management/organization
Management or the test organization needs to clearly state what the objective is for purchasing testing software. It must state the mission or goal that will be met by acquiring the test software and the expected ROI in terms of person-hours once the tool has been fully implemented. That can be accomplished by creating a simple mission statement and a minimum acceptable ROI.

It should be noted that any ROI fewer than three hours saved for every hour invested should be considered insufficient because of the impact of introducing a new business process into the testing organization. This should be a concise statement on the overall goal (one to three sentences) not a dissertation or catalogue of the products capabilities.

Example: Test management
The selected Test Management system shall enable end users to author and maintain requirements and test cases in a Web-enabled, shareable environment. Furthermore, the test management tool shall support test management "best practices" as defined by the Test Organization. The minimum acceptable ROI is four hours saved for every hour currently invested.

Example: Test automation
The selected Test Automaton tool shall enable end users to author, maintain and execute automated test cases in a Web-enabled, shareable environment. Furthermore, the test automation tool shall support test case design, automation and execution "best practices" as defined by the Test Organization. The minimum acceptable ROI is five hours saved for every hour currently invested.

Example: Defect management
The selected Defect Management tool shall enable end users to author, maintain and track/search defects in a Web- and email-enabled, shareable environment. Furthermore, the defect management tool shall support authoring, reporting and tracking "best practices" as defined by the Test Organization. The minimum acceptable ROI is four hours saved for every hour currently invested.

Needs analysis: Test architecture
Management has defined the immediate organizational goal, but the long-term architectural necessities must be defined by the testing organization. When first approaching the acquisition of testing software, test organizations usually have not invested much effort in defining an overall test architecture. Lack of an overall test architecture can lead to product choices that may be effective in the short term but lead to additional long-term costs or even replacement of a previously selected toolset.

If an architectural framework has been defined, then the architectural needs should already be clearly understood and documented. If not, then a general set of architectural guidelines can be applied. Example:

The selected testing software and tool vendor shall do the following:

Have a record of integrating successfully with all Tier 1 testing software vendors.
Have a history of operational success in the appropriate environments.
Have an established end-user community that is accessible to any end user.
Support enterprisewide collaboration.
Support customization.
Support several (1 to n) simultaneous engagements/projects.
Provide a well-designed, friendly and intuitive user interface.
Provide a smooth migration/upgrade path from one version of the product to the next.
Provide a rich online help facility and effective training mechanisms (tutorials, courseware, etc.).

The general architectural requirements for any tool will include more objectives than the nine listed above, but it is important to note that any objective should be applied across the entire toolset.

Needs analysis: End user
The end user needs analysis should be a detailed dissertation or catalogue of the envisioned product capabilities as they apply to the testing process. (It would probably be a page or more of requirements itemized or tabulated in such a way as to expedite the selection process.) This is where the specific and perhaps unique product capabilities are listed. The most effective approach is to start from a set of general requirements and then extend into a catalogue of more specific/related requirements.

Example: Test management
The Test Management solution shall do the following:

Support the authoring of test requirements.
Support the maintenance of test requirements.
Support enterprise wide controlled access to test requirements (Web-enabled preferred).
Support discrete grouping or partitioning of test requirements.
Support traceability of requirements to test cases and defects.
Support "canned" and "user-defined" queries against test requirements.
Support "canned" and "user-defined" reports against test requirements.
Support coverage analysis of test requirements against test cases.
Support the integration of other toolsets via a published API or equivalent capacity.
And so on…

The key here is to itemize the requirements to a sufficient level of detail and then apply these requirements against each candidate.

Example: Test automation
The Test Automation solution shall do the following:

Support the creation, implementation and execution of automated test cases.
Support enterprise wide, controlled access to test automation (Web-enabled preferred).
Support data-driven automated test cases.
Support keyword-enabled test automation.
Integrate with all Tier 1 and 2 test management tools that support integration.
Integrate with all Tier 1 and 2 defect management tools that support integration.
Enable test case design within a non-technical framework.
Enable test automation and verification of Web, GUI, .NET, and Java applications.
Support the integration of other toolsets via a published API or equivalent capacity.
And so on…

Once again, the key is to itemize the requirements to a sufficient level of detail. It is not necessary for all the requirements to be "realistic" in terms of what is available. Looking to the future can often lead to choosing a tool that eventually does provide the desired ability.

Example: Defect management
The Defect Management solution shall do the following:

Support the creation of defects.
Support the maintenance of defects.
Support the tracking of defects.
Support enterprise wide controlled access to defects (Web-enabled preferred).
Support integration with all Tier 1 and 2 test management tools that support integration.
Enable structured and ad-hoc searches for existing defects.
Enable the categorization of defects.
Enable customization of defect content.
Support "canned" and customized reports.
And so on…

In all cases, understanding of the basic needs will change as you proceed through the process of defining and selecting appropriate testing software.

In addition, for each case you must make sure that a particular vendor does not redefine the initial goal. Becoming an educated consumer in any given product space will lead to a redefinition of the basic requirements that should be recognized and documented.

Identify candidates

Identifying a list of potential software candidates can be accomplished by investigating several obvious sources: generic Web search, online quality assurance and testing forums, QA and testing e-magazines and co-workers. Once you create a list of potential software candidates, an assessment of currently available reviews can be done. Remember to keep an eye open for obvious marketing ploys.

It is also important to note which products command the largest portion of the existing market and which product has the fastest growth rate. This relates to the availability of skilled end users and end-user communities. Review the gathered materials against the needs analysis and create a short list of candidates (three to five) for assessment.

Assess candidates

If you have been very careful and lucky, your first encounter with the vendor's sales force will occur at this time. This can be a frustrating experience if you are purchasing a relatively small number of licenses, or it can be an intimidating one if you are going to be placing an order for a large number of licenses. Being vague as to the eventual number of licenses can put you in the comfortable middle ground.

Assessments of any testing software should be accomplished onsite with a full demonstration version of the software. When installing any new testing software, install it on a typical end-user system, check for "dll" file conflicts, check for registry entry issues, check for file conflicts, and ensure the software is operational. Record any issues discovered during the initial installation and seek clarification and resolution from the vendor.

Once the testing software has been installed, assess the software against the previous needs analysis -- first performing any available online tutorials and then applying the software to your real-world situation. Record any issues discovered during the assessment process and seek clarification and resolution from the vendor. Any additional needs discovered during an assessment should be recorded and applied to all candidates.

The assessment process will lead to the assessment team gaining skills in the product space. It is always wise to do one final pass of all candidates once the initial assessment is completed. Each software candidate can now be graded against the needs/requirements and a final selection can be made.

Implementation

Implementation obviously is not part of the selection process, but it is a common point of failure. Test organizations will often invest in testing software but not in the wherewithal to successfully use it. Investing hundreds of thousands of dollars in software but not investing capital in onsite training and consulting expertise is a recipe for disaster.

The software vendor should supply a minimum level of training for any large purchase and be able to supply or recommend onsite consultants/trainers who will ensure the test organization can take full advantage of the purchased software as quickly as possible. By bringing in the right mix of training, consulting and vendor expertise, the test organization can avoid much of the disruption any change in process brings and quickly gain the benefits that software can provide.

SAP R/3 Testing

2009-01-05T00:45:00.000-08:00

Summary: Are you in the midst of an SAP R/3 upgrade or implementing SAP R/3 from scratch? Are you prepared to perform all the activities and tasks that are concomitant with the various SAP testing phases? The author identifies testing risks and shares lessons learned from testing SAP R/3. The author provides insights and recommendations for avoiding these testing risks that could halt SAP testing efforts and consequently delay the expected deployment of SAP into a live production environment. The insights range from documenting test cases to working with testing artifacts from the ASAP methodology.

Title: Lessons Learned from Testing SAP R/3

SAP R/3 is the market leader in ERP installations and ERP sales. SAP has thousands of tables, multiple industry specific solutions, thousands of transactions, and connectivity to an unlimited number of legacy systems. Furthermore SAP can be configured differently from one company to another which creates a myriad of permutations for executing an SAP transaction. Installing and customizing SAP is a daunting challenge. Testing SAP R/3 is in and of itself another intractable challenge.

Many projects fail to test SAP correctly and consequently suffer staggering financial losses after deploying SAP into a live environment. The key to maximizing the value and ROI of SAP is to install and customize SAP correctly based on the documented requirements and to test it extensively based on the documented test cases and end-to-end business scenarios. Below some lessons learned are offered and identified to help organizations test SAP.

1. Not following methodology or No methodology at all
Some companies implementing SAP adhere to the ASAP methodology. Other companies have ad-hoc or ASAP-like methodologies for implementing SAP.

Even companies that are supposedly implementing SAP based on the ASAP methodology are not very strict and stringent in adhering to all the activities, deliverables, and tasks associated with ASAP. Consequently, but not surprisingly these companies have much confusion, obfuscation, befuddlement when they attempt to implement SAP. Compounding this problem is the fact that many large companies implementing SAP hire two or more implementation partners and multiple subcontractors that have incompatible approaches, methodologies, and lessons learned for implementing SAP.

The project manager and the steering committee should specify within the project charter how SAP will be implemented and what deliverables will be produced based on either ASAP or some other proprietary methodology. The objective is to have defined, proven, and repeatable processes for implementing SAP and that the project members have the knowledge or know-how for adhering to the methodology. The creation of an audit team or standards team would be helpful in enforcing compliance with the chosen methodology.
2. Inadequate test tools

SAP R/3 comes with an internal recording tool known as CATT (eCATT). One of the advantages of CATT (eCATT) is that since it is part of the standard SAP system it’s free of charge. However CATT does have some limitations which impel many companies to procure other test tools.

There are many vendors offering commercial automated test tools and test management tools for testing SAP. Companies purchasing automated test tools expect and erroneously believe that the test tools will be the panacea to their entire SAP recording and testing needs. Unfortunately, this is not the case, since no two SAP implementations are exactly the same across two or more companies or at times even within different divisions of the same company. Consequently, a company implementing SAP might need to procure test tools from more than one vendor in addition to the CATT (eCATT) tool.

An SAP implementation could be implementing SAP add-ons such as BW (Business Warehouse), APO (Advanced Planning Optimization), SEM (Strategic Enterprise Management) or even modules such as PS (Project Systems) that generate graphs and charts that a recording tool does not recognize. Furthermore, a company may move its SAP GUI from the desktop (fat client) to running SAP as web-based (thin client) or through an emulated Citrix session which could render the existing test tools useless.

Companies that wish to move to an automated testing strategy should articulate and document what SAP modules and SAP add-ons they are installing in addition to any legacy applications integrating with SAP. This information should be provided to the vendors of automated test tools in order to determine what can actually be recorded and tested with the test tools. The company should further investigate with the vendor what additional benefits over CATT (eCATT) the automated test tools provide. The objective is to get the test tools that will maximize SAP recording.
3. Decentralized test teams

Some projects that I have consulted for, have a decentralized testing approach where each individual business process team, technical team (ABAP/4), and Basis (security) team conduct their own testing in the absence of a QA/testing team.

The decentralized method suffers from various drawbacks including redundancy, inconsistency, lack of standards, missing testing metrics, unreported test results, limited managerial visibility, limited test coverage, chaotic defect resolution, no independent verification of test results, etc.

Projects with a decentralized test approach perceive control as a significant advantage of decentralized testing. With decentralized testing each team has the independence and control as to how their particular area will be tested based on their own defined test procedures.

The perceived benefits from decentralized testing do not offset its drawbacks. A more robust approach for testing an ERP system is centralized testing where there is a QA team for defining the standards and procedures for the various testing cycles and documenting the test plan, lessons learned, UAT plan and the test strategy. The test procedures and test strategy include what test case templates will be used, how and what metrics will be reported, how peer reviews will be conducted, ensuring that all test requirements have been met, etc. In addition to the QA team under the centralized model the testing team provides expertise for executing test cases whether manually or with automated test tools. The testing team also provides independent verification for the test results since the testers were not associated with the configuration of the transactions, workflow development, creation of user roles, or the development of the RICE (Reports, Interfaces, Conversions, and Enhancements) objects, etc.
4. Working in Silos: Limited or no access to SMEs, Configuration Team

Since SAP is integrated and a change in one of its components can have a cascading effect on other aspects of the software it is often the case where the various SAP teams need to collaborate with one another.

This concept seems mundane and at first hand easy to grasp. But due to internal politics, deadlines, budget constraints, etc this concept is ignored and overlooked. When collaboration falls through the cracks the various SAP teams (configuration, RICE, Basis, testing, infrastructure, data, etc) start working independently in isolation or in a silo.

I have been in projects where the testing team members who arrived during the middle of the realization phase have been unable to get support from the SAP configuration team members and SMEs for drafting test cases, peer reviews, and sign offs.. In other projects the QA and testing team are almost considered a “bother”, “nuisance” to the other SAP teams.

The test manager and PM need to foster an environment of cooperation among the testing team and the various other SAP teams for the successful completion of testing artifacts and testing cycles.
5. Missing test bed

Within the SAP client landscape a dedicated QA environment should be allocated to the testing team for the various testing phases. The environment should be production sized, and contain production like data, have a stable baseline, and frozen configuration by the time the testing cycles commence.

What I have seen at many projects is a so-called “dedicated testing environment” that many SAP teams have access to for training, last minute configuration, etc during the middle of a testing cycle. This creates conflict among the teams and causes confusion over the ultimate ownership of the testing environment.

The test manager should insist on a test environment where the testing team has complete control (for instance control over the accounting periods, running payroll, etc) and where the environment is not shared with other entities in particular during the test execution phase.
6. No controls for promoting/transporting objects into production

In many SAP projects that I have consulted for, there are very little controls for promoting objects into a live production environment. Other projects are very lax with their rules for promoting objects and have no means for auditing the approval process for transporting objects or do not include all the necessary stakeholders in transporting objects.

ERP systems like SAP R/3 offer benefits such as tight integrations across its modules and sub-modules and access to real-time data, but these benefits can backfire or even become otiose when a poorly tested or not even tested object is transported into a production environment. A faulty transported object such as a configuration change or code for an interface can have a deleterious cascading effect on other SAP functionality in the production environment. For instance a change to the HR module within payroll can affect functionality in the FI-CO module.

To mitigate the risk of transporting objects it is necessary to include approvals and set up a system of workflow where stakeholders from the testing team, Basis team, and configuration leads have input into objects that will be transported into production. A project should set priorities for transporting objects based on their criticality, document and approve objects that get transported after having been tested, and establish the frequency in which objects will be transported (i.e. every Friday at 4:00 pm) to avoid impacting end users as much as possible.

The company Mercury Interactive (www.mercuryinteractive.com) offers the tool Kintana which can help companies to move objects into various SAP clients and instances while maintaining a history log for auditing purposes of the objects that were transported and the entities approving the transports. Alternatively, I was a project where the organization created a home grown repository within Lotus Notes that provided managerial visibility for moving and transporting SAP objects which included notifications and approvals.

Whether a commercial tool is purchased or a home grown application is developed the main objective is to place stringent controls for moving objects into production with the necessary approvals while maintaining a history log.
7. Flaws with BPPs

For those projects adhering to the ASAP SAP methodology BPPs (Business Process Procedures) are artifacts or outputs from the realization phase. BPPs are documented based on stand alone or for single SAP transactions. The BPP provides detailed information in the form of instructions with screen shot print outs for how to execute a given SAP transaction (i.e. SAP transaction MM01 – Create Material).

Admittedly, BPPs can assist a tester or end user on how to execute a given SAP transaction based on the project’s specific customizations. However, and this is often the case the SAP configuration team will document BPPs in a given environment such as the configuration environment with configuration data and then the SAP configuration team fails to update the BPPs to reflect the changes of the QA environment. The failure to update the BPPs diminishes their value for the testing team.

When BPPs are not updated they contain obsolete data, and may lack the necessary information to execute an SAP transaction for an SAP environment that has had customizations since the time the BPPs was initially created. Furthermore, many organizations do not perform any form of version control on BPPs which makes it difficult to discern whether a BPP is finalized, completed, peer-reviewed or still undergoing changes.

Poorly documented BPPs or outdated BPPs have a propagating effect on the creation of test cases and test scripts. Since BPPs are documented per stand alone SAP transaction, the testing team will need to link multiple BPPs for end to end SAP test cases (i.e. Hire–to–Fire test case) that involve multiple SAP transactions with pre and post conditions. A single poorly written BPP or outdated BPP can inhibit the creation of a test case containing several SAP transactions.
8. Missing peer reviews

Peer reviews help refine work-products and deliverables. Peer reviews also provide independent verification and give the end customers or SMEs an opportunity to provide feedback during the early stages of the SAP implementation. Peer reviews can occur at many junctures during the implementation of SAP for tasks such as filling out CI (Customer Input) templates, drafting test cases, creating functional design specs, development of business process flow diagrams, documenting BPPs, code walk-through, etc.

Inexplicably many SAP projects do not engage in the practice of peer reviews or have any templates or forms for documenting peer review feedback. The end result is that often times the end users have complaints about the quality of test cases during the UAT (User’s Acceptance Test) and about how a particular process was customized in SAP versus how the process was previously executed in the end user’s legacy systems.

Test managers should solicit the feedback, input and even the sign-off from the SMEs for the various testing artifacts as soon as possible or as the testing artifacts are produced. It is in the best interests of the test team to identify problems with the testing artifacts as soon as possible as opposed to weeks before the SAP cut-over or SAP go-live dates.
9. Problems obtaining valid test data

Arguably, the most prevalent risk to conducting an SAP test whether it is integration, functional, string, volume, smoke, or security test is obtaining valid SAP test data.

When dealing with SAP data invariable one or more of the questions below will arise before a testing cycle ensues:

Where will the test data come from? How will one produce new data for transactions that require unique data? How will the data be loaded into a new SAP client and instance? What happens when an interface or conversion needs to send data into core SAP R/3 from a legacy system and the interface or conversion is not executing properly or has yet to be developed? What happens when the transactional data has not been created?

Assuming that the testing team has a dedicated test bed or QA box the test manager will need to ensure that all the necessary data (master, transactional, test, etc) is properly loaded as part of the assessment for the test readiness review. Ideally an SAP test will be conducted within a test bed containing data that closely mirrors production data and where the testing environment is production-sized. The test manager should prioritize the test cases that will be executed and determine and document any work-around for test cases that cannot be executed due to missing test data.
10. Missing flow processes, diagrams

Diagramming or modeling how a business scenario will flow within SAP provides invaluable insights to the testing team and the configuration team. Diagrams and flow processes can illustrate the lifecycle, stages, sequences, activities, states and events associated with a particular business process. Unfortunately, many projects undergoing an SAP implementation or SAP upgrade fail to diagram their business processes leaving testers or newly hired resources in a bind to comprehend how legacy processes, or new customizations derived from gap analysis will be executed within SAP. SAP testers and end users participating in the user’s acceptance test should have access to diagrams depicting how business processes flow within SAP. The absence of such diagrams puts undue burden on the testing team and places the end users in a position of disadvantage.

The SAP business analysts in conjunction with the SAP configuration team, and possibly the SMEs (Subject Matter Experts) can develop diagrams in tools such as Visio, or the ARIS modeling tool that integrates and links with SAP’s Q&adb (Question and answer database) for those SAP implementations following the ASAP methodology. Alternatively, with Rational’s tools one can develop UML (Unified Modeling Language) diagrams to illustrate processes with interaction diagrams (sequence and activity diagrams).
11. No integration testing with external components

SAP integrates with many legacy systems via interfaces, IDOCs, conversions, BAPIs, connectors, or even middle ware such as Mercator. SAP can even be integrated with other commercial ERP systems, forecasting/planning systems, and CRM applications.

Although, the integration of SAP with other systems is critical to the successful implementation and deployment of SAP, many companies merely test during the integration test the interaction of SAP among its various modules and add-ons. Although the aforementioned type of integration testing is necessary it is also critically important to test the integration of SAP with non-SAP systems.

Some examples of integration testing that I have conducted between SAP and non-SAP systems are: SAP integrating via IDOCs to bar coding software, SAP properly sending outbound data to legacy system and the legacy system processing the data correctly, end to end financial reconciliations between SAP and data warehouses, integration between SAP and planning systems for MRP runs, etc.
12. Limited security access

I have seen SAP testing efforts come to a halt because the testers did not have the necessary privileges and permissions to execute certain SAP transactions or the necessary roles to submit electronic approvals. This problem is exacerbated when testers are executing test cases at odd hours or during weekend hours and there is no Basis support to augment the tester’s security role.

Successful execution of certain test cases in SAP whether manually or with automated test tools will require the test team members to have super user access. This is of utmost importance when conducting positive and negative testing for security roles. The testing team should coordinate and plan with the Basis team the necessary roles and permissions for developing test cases and for executing test cases in the designated QA environment or test bed.

Tips to software practitioners

2009-01-04T22:27:00.000-08:00

1Software Testing Objectives
The Major Objectives of Software Testing:
- Uncover as many as errors (or bugs) as possible in a given timeline.
- Demonstrate a given software product matching its requirement specifications.
- Validate the quality of a software testing using the minimum cost and efforts.
- Generate high quality test cases, perform effective tests, and issue correct and helpful problem reports.

Major goals of Software Testing:
uncover the errors (defects) in the software, including errors in:
- requirements from requirement analysis
- design documented in design specifications
- coding (implementation)
- system resources and system environment
- hardware problems and their interfaces to software
2Software Quality Factors
Functionality (exterior quality)

- Correctness, reliability, usability, and integrity

Engineering (interior quality)

- Efficiency, testability, documentation, structure

Adaptability (future qualities)

- Flexibility, reusability, maintainability
3Software Testing Myths
- We can test a program completely. In other words, we test a program exhaustively.

- We can find all program errors as long as test engineers do a good job.

- We can test a program by trying all possible inputs and states of a program.

- A good test suite must include a great number of test cases.

- Good test cases always are complicated ones.

- Software test automation can replace test engineers to perform good software testing.

- Software testing is simple and easy. Anyone can do it. No training is needed.
4Software Testing Limits
- Due to the testing time limit, it is impossible to achieve total confidence.

- We can never be sure the specifications are 100% correct.

- We can never be certain that a testing system (or tool) is correct.

- No testing tools can copy with every software program.

- Tester engineers never be sure that they completely understand a software product.

- We never have enough resources to perform software testing.

- We can never be certain that we achieve 100% adequate software testing.
5Software Testing Principles
Principle #1: Complete testing is impossible.

Principle #2: Software testing is not simple.
Reasons:
Quality testing requires testers to understand a system/product completely
Quality testing needs adequate test set, and efficient testing methods
A very tight schedule and lack of test tools.

Principle #3: Testing is risk-based.

Principle #4: Testing must be planned.

Principle #5: Testing requires independence.

Principle #6: Quality software testing depends on:
Good understanding of software products and related domain application
Cost-effective testing methodology, coverage, test methods, and tools.
Good engineers with creativity, and solid software testing experience
6Software Testing Activities
Test Planning
Define a software test plan by specifying:
- a test schedule for a test process and its activities, as well as assignments
- test requirements and items
- test strategy and supporting tools

Test Design and Specification
- Conduct software design based well-defined test generation methods.
- Specify test cases to achieve a targeted test coverage.

Test Set up
- Testing Tools and Environment Set-up
- Test Suite Set-up

Test Operation and Execution
- Run test cases manually or automatically

Test Result Analysis and Reporting
- Report software testing results and conduct test result analysis

Problem Reporting
- Report program errors using a systematic solution.

Test Management and Measurement
- Manage software testing activities, control testing schedule, measure testing complexity and cost

Test Automation
- Define and develop software test tools
- Adopt and use software test tools
- Write software test scripts and facility

Test Configuration Management
- Manage and maintain different versions of software test suites, test environment and tools, and documents for various product versions.
7Bug Writing tips
12 Bug writing tips:

1.Be very specific when describing the bug. Don’t let there be any room for interpretation. More concise means less ambiguous, so less clarification will be needed later on.

2.Calling windows by their correct names (by the name displayed on the title bar) will eliminate some ambiguity.

3.Don’t be repetitive. Don’t repeat yourself. Also, don’t say things twice or three times.

4.Try to limit the number of steps to recreate the problem. A bug that is written with 7 or more steps can usually become hard to read. It is usually possible to shorten that list.

5.Start describing with where the bug begins, not before. For example, you don't have to describe how to load and launch the application if the application crashes on exit.

6.Proofreading the bug report is very important. Send it through a spell checker before submitting it.
7. Make sure that all step numbers are sequenced. (No missing step numbers and no duplicates.)

8.Please make sure that you use sentences. This is a sentence. This not sentence.

9.Don’t use a condescending or negative tone in your bug reports. Don’t say things like "It's still broken", or “It is completely wrong”.

10.Don’t use vague terms like “It doesn’t work” or “not working properly”

11.If there is an error message involved, be sure to include the exact wording of the text in the bug report. If there is a GPF (General Protection Fault) be sure to include the name of the module and address of the crash.

12.Once the text of the report is entered, you don’t know whose eyes will see it. You might think that it will go to your manager and the developer and that’s it, but it could show up in other documents that you are not aware of, such as reports to senior management or clients, to the company intranet, to future test scripts or test plans. The point is that the bug report is your work product, and you should take pride in your work.

Be aware of SOA application security issues

2009-01-04T10:33:00.000-08:00

Extensible Markup Language (XML), Web services, and service-oriented architecture (SOA) are the latest craze in the software development world. These buzzwords burn particularly bright in large enterprises with hundreds or thousands of systems that were developed independently. If these disparate systems can be made to work together using open standards, a tremendous amount of time, money, and frustration can be saved. Whether or not we are on the verge of a new era in software, the goal alone is enough to make security people cringe. It might be easy to glue System A and System B together, but will the combination be secure?

Today, most discussions about Web services security focus on standards: from WS-Security and SAMLdown to SOAP and even XML. While the standards are important, measuring the security of a Web service requires looking past the standards and considering all of the ways the system might fail. Attackers focus on a system's weak points, and a security assessment should do the same. Consider the following:

Complex protocols might seem bloated or redundant. Skipping or changing steps might not create any problems for normal operation, but it could but jeopardize the security guarantees the protocol offers.
Application platforms that support Web services are incredibly flexible. In practice, that means they have incredibly complex configurations. Mistakes in these configuration files become vulnerabilities in the service.
Services are often built on top of legacy software that was not originally designed to operate as a service. Vulnerable code that's never been exposed to the network before offers attackers new opportunities.

· Eliminating problems like those is part of any comprehensive approach to SOA security. We'll take a look at an example of each and then talk about the business software assurance techniques you should use to prevent them.

· Most people know better than to make up their own cryptography, but a surprising number of programmers are still willing to try their hand at a home-brewed authentication protocol. Last month Google found out just what a bad idea that is. The single sign-on protocol they used for Google Applications was derived from SAML, but it left out a few seemingly unnecessary pieces of information from two protocol messages. The result was a severe security flaw that allowed a dishonest service provider to impersonate a user at another service provider. The moral to this story is if you adopt a standard, don't stop at "good enough." Subtle vulnerabilities creep in where standards compliance falls off.

· Web service containers such as WebSphere, WebLogic, or .NET WSE are amazingly versatile when it comes to Web services support, but what at first might seem a blessing quickly turns into a nightmare for the people who have to configure these systems. People stitch together configurations from sample projects or flail until somehow their service starts working. The result is service configurations that don't do exactly what the author intended. The example below shows an Apache Axis 2 Rampart client configuration that does not require inbound messages to be encrypted (since the tag does not contain an Encrypt directive).

·

·         ...

·

·

·            Timestamp Signature

·            ...

·

·

·

· When this configuration is buried in an enormous configuration file, the mistake is hard to catch. The result is a Web service that doesn't offer all of the security guarantees its author would.

But there are a tremendous constellation of security errors that aren't related to standards or to configuration. In fact, Web services don't introduce new types of security concerns as often as they provide new opportunities to make old mistakes. If a legacy system gets a brand new Web services interface, all sorts of problems that used to be buried deep in the system might now be able to find their way out. By exposing what used to be the internal workings of a program directly to the network, programmers may inadvertently bypass input validation or access control mechanisms, expose too much access to the internal workings of the program, or provide a new forum for making session management mistakes.

The situation is exacerbated by the fact that the security requirements for a Web service are at least somewhat ambiguous. Web services are supposed to be flexible so that they can be used by other programmers to assemble applications that the creator of the Web service may not have envisioned, but this makes it difficult for a Web service to understand the security needs of its callers. What sort of output validation should a Web service perform? If the Web service is intended to be used directly by a Web browser, then it should take precautions to prevent cross-site scripting (XSS). But if the author of a Web service doesn't know how it will be used, then it is hard to make the right security decisions.

Consider the code below. It's a method taken from DionySOA, a project that advertises itself as a Reseller/Broker service platform built using SOA and Web services. The method is exposed through a Web service. (You get a hint that it might be externally accessible when you see that it throws java.rmi.RemoteException. Knowing for sure requires looking at the application's configuration files.) The method contains a blatant SQL injection vulnerability. It takes its attacker-supplied parameter, concatenates it into a SQL query string, and executes the query. Although it is possible to make this kind of mistake without any help from Web services, we can't help but believe that the Web services setup made it easier to forget about input validation.

public supplier.model.SupplierProduct[ ]

searchName(java.lang.String in0) throws

      java.rmi.RemoteException {

System.out.println("searchName("+in0+")");

    String query="SELECT * FROM products " +

               " WHERE name like '"+in0+"'  ";

  return this.doSQL(query);

Similarly, if a newly exposed method used to rely on another part of the application to perform access control checks, the Web services interface might now bypass those checks, making it easy to lose track of the trust boundary.

There's nothing fundamentally wrong with making it easy to create a Web service, but creating a good Web service is really not so easy. The Web services frameworks we are aware of do not give a programmer any guidance about the security implications that might be involved in exposing the insides of a program.

We don't mean for these examples to discourage the use of Web services, XML, and SOA frameworks, but rather to make users aware of the missteps that can occur when deploying these technologies. These examples will hopefully be a catalyst for taking security into account as part of the software development process. Business software assurance requires building security into every step, rather than trying to tack it on at the end. Just as quality assurance is the operational solution for managing product quality, business software assurance is the operational solution for managing software risk. The following activities should be part of any software project:

Think like a bad guy. How would you attack this system? What would you hope to gain?
Automated vulnerability detection -– source code analysis and dynamic security testing -– to identify vulnerabilities before the code is deployed.
Real-time monitoring and active protection to defend the code once it's in production.

As companies become increasingly dependent on software to run the business, and as new technologies enable old applications to find fresh new uses, companies must make security a priority. A more proactive and systematic approach towards software security will be -- in the end -- less costly and more predictable than an ad-hoc security strategy.

How to specialize in performance testing

2008-08-06T02:17:00.000-07:00

How to specialize in performance testing

Q-I have been learning performance testing for the past eight months. However, I was not given any opportunities in performance testing in my present company even though I performed well in the internal interviews. I really want to become a good performance tester -- it is my dream to become a performance tester. Please guide me as to how I can make this happen.

Expert’s Response: I think this is a great question. It's specific, which makes it easy to start answering, but it's also general enough that anyone who is interested in specializing in something within software testing should be able to pull something from the answer. I think there are three overarching dynamics to your question:

· How can you best structure your future learning to support your goals?

· How can you best market your abilities to get the opportunities you want?

· How can you best structure the work you're currently doing to support your work and learning objectives?

Continuing to learning about performance testing
The activity that you have the most control over is your own learning. I've been studying and doing performance testing for eight years, and I honestly still learn something new about performance testing almost every week. It's a deep and rich specialization in software testing. There's a lot to performance testing that still needs to be formalized and written down. It's still a growing body of knowledge.

If your dream is performance testing, then you need to continue to learn. Reading articles, blogs, books and tool documentation is a good place to start. Attending conferences, training, workshops and local groups is a great place to meet others who have similar passions. If you don't have opportunities like those, then join one of the many online communities where performance testers have a presence. Depending on your learning style, dialog and debate can be as great a teacher as reading, if not greater.

Finally, no learning is complete without practice. I'm so passionate about the topic of practice that I wrote an entire article on it. Many of the materials you read will include exercises. Work through them. Many of the conferences, training, and workshops you attend will show examples. Repeat them. Going through the work on your own, even if you already know the outcome, provides a different kind of learning. Some people learn best when the experience is hands-on.

For performance testing, I think a great place to start practicing is in the open source community. Given the nature of performance testing, most tool knowledge is transferable to other performance testing tools. Learning multiple open source tools will also give you different ideas for how you can solve a performance testing problem. Many times, our available tools anchor our thinking about how to approach the problem. If you've practiced with multiple tools, you're more likely to have variety in your test approaches and solutions.

Once you know how to use a couple of performance testing tools, if you can't seem to get the project work you need at your current employer, and you're unwilling or unable to leave for another opportunity, then I recommend volunteering your time. There are a lot of online communities that help connect people who want to volunteer their technical talents to nonprofits or other community-minded organizations. Finding project work outside of your day job can be just as valuable as formal project work.

Start marketing your skills and abilities
If you're serious about performance testing as a career, I recommend you start pulling together some marketing material. A resume is the place most people focus their limited marketing skills. That could be a good place for you to start as well. What story does your resume tell a potential employer? Is it that you're a performance tester? How has each of your past experiences helped you develop a specific aspect of performance testing? Remember, one of the great challenges performance testing presents to practitioners is its variety. That makes it easy to relate a variety of experiences to the skills a performance tester needs.

Don't forget to include your training on your resume. I've had to remind several people of classes they've attended, workshops they participated in, or people who have been an active member of an online community for years and have not included that on their resume. If it helps you tell the story of your expertise, get it on there. Include anything that shows an employer that you're passionate about performance testing and you're continuously learning more about it.

Depending on the types of companies you want to work for, or the types of projects you might want, a certification might be appropriate. Certifications relevant to performance testing aren't just performance testing tool certifications. Appropriate certifications may also come in the form of programming languages (e.g., Java certification), networking (e.g., CCNA), application servers (e.g., WebSphere administrator certification), databases (e.g., Oracle certification), or even a certification in the context you want to work in (e.g., CPCU certification if you want to work in the Insurance industry). I'm not normally a big fan of certifications, but they are clear marketing products.

Finally, I think the best way to market yourself is to write. Start by being active in an online community. Answer questions on forums or debate ideas on mailing lists. As you learn, catalog your learning in a blog so others can benefit from your hard work. If you feel you're really starting to understand a specific aspect of performance testing, try writing an article or paper on it (for example, email your idea to an editor at SearchSoftwareQuality.com -- they'll point you in the right direction for help if you need it). Present your idea at a conference or workshop. The more of a public face you develop by writing, the more you learn. My experience has been that people are very vocal in their feedback on what you write. You should get to learn a lot. Even if you don't become the next Scott Barber, when a potential employer Googles your name, they'll quickly see that you know something about performance testing and have a passion for it.

Align your project work with performance testing activities
Even if you can't get performance testing projects at your current employer, you can still get project work that relates to performance testing. Does your team test Web services? See if you can get involved; it will get you experience with XML, various protocols and, often, specialized tools. Does your team test databases? See if you can get involved; it will get you experience with SQL and managing large datasets. Does your team write automated tests? See if you can get involved; it will get you experience programming and dealing with the problems of scheduled and distributed tests. Does your team do risk-based testing? See if you can get involved; it will get you experience modeling the risk of an application or feature and teach you how to make difficult choices about which tests to run. I could go on with more examples. Take your current opportunities and make them relevant for learning more about performance testing.

If you can't get your own performance testing project, ask if you can work with someone else. What if you volunteer some of your time? What if you work under someone else's supervision for a while? Work with your current manager to understand what factors are preventing them from giving you the opportunity. Perhaps they can't give you the opportunity for a number of reasons out of their direct control. Perhaps they can, they just haven't given it enough attention. After a conversation where you try to figure it out with them, you should have an idea of what opportunities are available at that company. Just recognize that sometimes you have to leave for different opportunities. If you do that, make sure you're clear with your new employer as to what your expectations are.

I hope that's helpful. Your question is a great one, and I feel like it covers a general concern software testers have. The general form of the answer is the same for people who might want to specialize in security testing, test automation, Web service testing, test management, or any other aspect of testing where there can be specialization. Stay focused on your learning and development, actively market your knowledge and abilities, and work to align your work with your goals -- even if that means taking projects outside of the specialization to help you develop a specific skill.

How to do integration testing

2008-08-06T02:16:00.000-07:00

How to do integration testing

Q- How do testers do integration testing? What are top-down and bottom-up approaches in integration testing?

Expert’s response: Ironically, integration testing means completely different things to completely different companies. At Microsoft, we typically referred to integration testing as the testing that occurs at the end of a milestone and that "stabilizes" a product. Features from the new milestone are integration-tested with features from previous milestones. At Circuit City, however, we referred to integration testing as the testing done just after a developer checks in -- it's the stabilization testing that occurs when two developers check in code. I would call this feature testing, frankly…

But to answer your question, top-down vs. bottom-up testing is simply the way you look at things. Bottom-up testing is the testing of code that could almost be considered an extension of unit testing. It's very much focused on the feature being implemented and that feature's outbound dependencies, meaning how that feature impacts other areas of the product/project.

Top-down, on the other hand, is testing from a more systemic point of view. It's testing an overall product after a new feature is introduced and verifying that the features it interacts with are stable and that it "plays well"' with other features.

The key to testing here is that you are in the process of moving beyond the component level and testing as a system. Frankly, neither approach alone is sufficient. You need to test the parts with the perspective of the whole. One part of this testing is seeing how the system as a whole responds to the data (or states) generated by the new component. You want to verify that data being pushed out by the component are not only well-formatted (what you tested during component testing) but that other components are expecting and can handle that well-formatted data. You also need to validate that the data originating within the existing system are handled properly by the new component.

Real-world examples? Well, let's assume you are developing a large retail management system, and an inventory control component is ready for integration. Bottom-up testing would imply that you set up a fair amount of equivalence-classed data in the new component and introduced that new data into the system as a whole. How does the system respond? Are the inventory amounts updated correctly? If you have inventory-level triggers (e.g., if the total count of pink iPod Nanos falls below a certain threshold, generate an electronic order for more), does the order management system respond accordingly? This is bottom-up testing.

At the same time, you want to track how well the component consumes data from the rest of the system. Is it handling inventory changes coming in from the Web site? Does it integrate properly with the returns system? When an item's status is updated by the warehouse system, is it reflected in the new component?

We see constant change in the testing profession, with new methodologies being proposed all the time. This is good -- it's all part of moving from art to craft to science. But just as with anything else, we can't turn all of our testing to one methodology because one size doesn't fit all. Bottom-up and top-down testing are both critical components of an integration testing plan and both need considerable focus if the QA organization wants to maximize software quality.

Test coverage: Finding all the defects in your application

2008-08-06T01:40:00.000-07:00

Test coverage: Finding all the defects in your application

Q-If trace matrix does not meet the requirement for test coverage, what would you suggest for the same? How can I assure the coverage of all functionalities by a team member as a team leader?

Expert Response: The trace matrix is a well-established test coverage tool. Let me offer a quick definition -- the purpose of the trace matrix is to map one or more than one test case to each system requirement, the trace matrix is usually formatted in a table. The fundamental premise is that if one or more than one test case has been mapped to each requirement, then all the requirements of the system must have been tested and therefore the trace matrix proves testing is complete.

I see flaws with this line of reasoning and here are my primary reservations on the over-reliance of the trace matrix:

A completed trace matrix is only as valuable as the contents. If the requirements are not complete or clear than the test cases designed and executed might fulfill the requirements but the testing won't have provided what was needed. Conversely if the requirements are clear but the test cases are insufficient then a completed trace matrix still doesn't indicate the testing coverage and confidence that is being sought by a completed table.
The trace matrix design relies too stringently on system requirements -- that is the primary design of the trace matrix -- to ensure all system requirements have been tested. But all sorts of defects can be found outside of the system requirements that are still relevant to the application providing a solution for the customer. By looking only at the system requirements and potentially not considering the customers' needs and real life product usage, essential testing could be overlooked. Testing only according to specified requirements may be too narrowly focused to be effective in real life usage -- unless the requirements are exceptionally robust.

Overall I feel the trace matrix might provide a clean high level view of testing but a checked-off list doesn't prove an application is ready to ship. The reason some people value the trace matrix is the matrix attempts to offer an orderly view of testing; but in my experience testing is rarely such a tidy task.

So how do you call the end of testing? And how can you assure test coverage?

To be able to assure coverage at the end, I'd start with reviewing the beginning -- look at the test planning. Did your test planning include a risk analysis? A risk analysis at the start of a project can provide solid information for your test plan. Host a risk analysis either formally or informally, gather ideas by talking with multiple people. Get different points of view -- from your project stakeholders, talk to your DBAs, your developers, your network staff, and your business analysts. Plan testing based on your risk analysis.
As a project continues, shift testing based on the defects found and the product and project as it evolves. Focus on high risk areas. Adapt testing based on you and your testing team's experience with the product. Be willing to adjust your test plan throughout the project.
Throughout testing, watch the defects reported. Keep having conversations and debriefs with hands-on testers to understand not just what they've tested but how they feel about the application. Do they have defects they've seen but haven't been able to reproduce? What is their perception of the current state of the application?

In my view, there is no one tool including the trace matrix that signals testing is complete but the combination of knowing how testing was planned and adapted throughout the project, a thorough review of the defects reported and remaining, and the current state of the application according to you and your team's experience should provide you with an objective assessment of the product and the test coverage.

Don't mistake user acceptance testing for acceptance testing

2008-07-29T22:54:00.000-07:00

Don't mistake user acceptance testing for acceptance testing

If you think software testing in general is badly misunderstood, acceptance testing (a subset of software testing) is even more wildly misunderstood. This misunderstanding is most common with commercially driven software as opposed to open source software and software being developed for academic or research and development reasons.

This misunderstanding baffles me because acceptance testing is one of the most consistently defined testing concepts I've encountered over my career both inside and outside of the software field.

First, let's look at what Wikipedia has to say about acceptance testing:

"In engineering and its various subdisciplines, acceptance testing is black-box testing performed on a system (e.g. software, lots of manufactured mechanical parts, or batches of chemical products) prior to its delivery…

"In most environments, acceptance testing by the system provider is distinguished from acceptance testing by the customer (the user or client) prior to accepting transfer of ownership…

"A principal purpose of acceptance testing is that, once completed successfully, and provided certain additional (contractually agreed) acceptance criteria are met, the sponsors will then sign off on the system as satisfying the contract (previously agreed between sponsor and manufacturer), and deliver final payment."

This is consistent with the definition Cem Kaner uses throughout his books, courses, articles and talks, which collectively are some of the most highly referenced software testing material in the industry. The following definition is from his Black Box Software Testing course:

"Acceptance testing is done to check whether the customer should pay for the product. Usually acceptance testing is done as black box testing."

Another extremely well-referenced source of software testing terms is the International Software Testing Qualifications Board (ISTQB) Standard glossary of terms. Below is the definition from Version 1.3 (dd. May, 31, 2007):

"Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system."

I've chosen those three references because I've found that if Wikipedia, Cem Kaner and the ISTQB are on the same page related to a term or the definition of a concept, then the testing community at large will tend to use those terms in a manner that is consistent with these resources. Acceptance testing, however, is an exception.

There are several key points on which these definitions/descriptions agree:

In each case, acceptance testing is done to determine whether the application or product is acceptable to someone or some organization and/or if the person or organization should pay for the application or product AS IS.
"Finding defects" is not so much as mentioned in any of those definitions/descriptions, but each implies that defects jeopardize whether the application or product becomes "accepted."
Pre-determined, explicitly stated, mutually agreed-upon criteria (between the creator of the application or product and the person or group that is paying for or otherwise commissioned the software) are the basis for non-acceptance.

Wikipedia refers to this agreement as a contract, and identifies that non-compliance with the terms of the contract as a reason for non-payment. And Kaner references the contract through the question of whether the customer should pay for the product.

The ISTQB does not directly refer to either contract or payment but certainly implies the existence of a contract (an agreement between two or more parties for the doing or not doing of something specified).

With that kind of consistency, you'd think there wouldn't be any confusion.

The only explanation I can come up with is that it is related to the fact that many people involved with software development only have experience with "user acceptance testing," and as a result they develop the mistaken impression that "user acceptance testing" is synonymous with "acceptance testing."

If my experiences with software user acceptance testing are common, I can understand where the confusion comes from. All of the software user acceptance testing that I have firsthand experience with involves representative users being given a script to follow and then being asked if they were able to successfully complete the task they were assigned by the person who wrote, or at least tested, the script.

Since the software had always been rather strenuously tested against the script, the virtually inevitable feedback that all of the "users" found the software "acceptable" is given to the person or group paying for the software. The person or group then accepts the software -- and pays for it.

I have no idea how many dissatisfied end users, unhappy commissioners of software and unacceptable software products this flawed process is responsible for, but I suspect that it is no small number.

There are several flaws with that practice, at least as it relates to the definitions above.

The "users" doing the acceptance testing are not the people who are paying for the development of the software.
The person or group paying for the development of the software is not present during the user acceptance testing.
The "users" are provided with all of the information, support, instructions, guidance and assistance they need to ultimately provide the desired "yes" response. Frequently this assistance is provided by a senior tester who has been charged with the task of coordinating and conducting user acceptance testing and believes he is doing the right thing by going out of his way to provide a positive experience for the "users."
By the time user acceptance testing is conducted, the developers of the software are ready to be done with the project and the person or group paying for the development of the software is anxious to ship the software.

If that is the only experience a person has with acceptance testing, I can see why one may not realize that the goal of user acceptance testing is to answer whether the end users will be satisfied enough with the software -- which obviously ships without the scripts and the well-meaning senior tester to help out -- to want to use and/or purchase it.

I have no idea how many dissatisfied end users, unhappy commissioners of software and unacceptable software products this flawed process is responsible for, but I suspect that it is no small number.

In an attempt to avoid dissatisfied end users, unhappy commissioners of software and unacceptable software products, whenever someone asks me to be a part of any kind of acceptance testing -- whether qualified by additional terms like "user," "build," "system," "automated," "agile," "security," "continuous," "package," "customer," "business process," "market," or something else -- I pause to ask the following:

"For what purpose is who supposed to be deciding whether or not to accept what, on behalf of whom?"

My question often confuses people at first, but so far it has always lead to some kind of acceptance testing that enables decisions about the software and its related contracts. And that is what acceptance testing is all about.