The Automated Tester

Testing Should Be Elegant

2009-11-02T18:00:00.000Z

When you think of elegant what is the first thing to pop in to your mind? Is it a ballerina during "The Nutcracker"? That's one of the first things that pops into my mind.

When you think about elegant technology what is the first thing that appears in your mind? My first thought is an Apple iPod. Actually a lot of Apple products are very good examples of elegant. Other examples of elegance in technology are Amazon's "1-Click Buy" and Google Wave.

In all of these examples the elegance has given them a very unique selling point and made them leaders in their respective fields. The other thing that they have in common is that they have a lot of work that is done in the background. For example, with Amazon's "1-Click buy" it needs to handle if I buy 5 things in 5 minutes and put them in the same box. As a customer, do I care how it does it? No. As Amazon's COO, do I care? yes! It saves money on postage that the customer will probably refuse to pay because they just spent some money.

Now what does this have to testing? Well, in testing what would you consider elegant? I will make it easier by splitting it to test automation and manual testing. In test automation, what framework would you consider elegant? For me, nothing comes to mind. There are some BDD frameworks that are close but to be honest they aren't elegant. In manual testing what technique for testing would you say is elegant? Exploratory testing...no! Scripted Testing...no!

Why is software testing not elegant then? Well, the thing that makes elegance in the items that I had at the beginning of this post is the removal of decisions, especially redundant decisions. The iPhone goes on to silent with a flick of switch, my Android takes 2 or three steps asking me questions as I go along. Every time I get a question I need to make a decision. Nokia's are even worse with the changing profiles that make something slightly less silent each time.

So now get a developer to test their code and they look for the elegant approach because its out of their comfort zone. Developers write tests so they can produce high quality code but they need to decide which framework will fit the best. Decision number 1. Then when they start working with it they realise that there is something that it can't do against their code so they need to swap it out with a new framework. This means they need to update a number of tests which can take a lot of work. Decision numbers 2 and 3. 2 being the need to change the framework and 3 being the new framework that they chose.

If you are a UI Test automator I am sure that you have had a play with Selenium. Selenium is a perfect example of something that could be perfectly elegant but isn't. Have you ever tried to type something into an input box and failed with Selenium? Have you ever tried to press a button with Selenium and failed? Why do you fail to get it right? It was the decisions that you were forced to make at the beginning and you got it wrong. But because you made the decision you are less likely to blame Selenium than yourself.

Then you get into the situation that a developer, either yourself or a colleague, changes the layout of something that didn't have an ID you suddenly see your Continuous Integration server goes red. The XPath is different so you need to update it in each of the broken tests. Elegant? Far from it!!!

So what can we do about this? Well, most developers have started using Domain Specific Languages(DSLs) to write their tests and let all the hard work be abstracted out. It allows developers to now write tests that are now readable and leading towards testing being elegant. The formation of DSLs is starting to make testing more elegant but we need to do more to make things elegant.

In manual testing we also need to make it more elegant. Take a form that has 6 input fields and you can fill in 1 or all 6 of these fields. There are 720 different combinations and then we have to assume that it only takes 1 type of input and that it doesn't go down a number of different code path because that will add to the number of decisions that a tester has to make. Testing will never remove the decision making but cutting them down to make testers a lot more productive.

There are techniques to help you cut the decisions down, like equivalence partitioning, but there are still lots of decisions to be made. If partitioning suggests we only do 10% of the combinations because those are the combinations that are important we still have 72 decisions to make. If each of those take 10 seconds to execute you suddenly have 1 form that can 12 minutes to do. Elegant? Far from it. This is possibly the other reason why management think that test automation is the be all. Test automation is slightly more elegant but to be honest it isn't as elegant as it should be!

What can we as testers do about this? Well unfortunately I can't answer it but we as a community need to do something to make it all a lot more elegant! So when you start developing you next load of test cases, try think about how you can make the process of testing elegant.

GTAC Day 2

2009-10-22T11:18:00.004+01:00

Day 2 of GTAC started out with something that always fascinates me in the computing world. Something more than test automation or the user experience that users are given. The day started by a talk given by Dr Alberto Di Meglio about grid computing.

Opening talk for Day 2 - Dr Alberto Di Meglio

Alberto works for CERN working on the ETICS project. This is part of a project to get Grid computing working so that scientists can use it to complex calculations. This grid is also going to be use by many other scientists around the world use the E-sciencE grid. Alberto started explaining that with new technologies, like the LHC, we need something to process a large amount of data and we need to do it fairly quickly. The LHC can produce in the region of 15 Petabytes worth of data.

All of this work needs to be done and needs to tie in with the emerging standards that are appearing for Grid computing. The next challenge that they have seen is that they need to find a way of doing Quality Assurance on the system. ETICS is a continuous build, test and QA verification system. This system is designed so that developers can work in many different languages but they all access a standard API to get builds and tests done. ETICS is developed against a QA model called A-QCM, which stands for Automated Quality Control Model. They also implement a number of ISO codes and have submitted how they do their work to potentially become its own ISO code.

Tests are done in a multinode environment. This environment is built dynamically for what is needed and when it is needed. This is done by creating VM images on the fly because maintaining a set of VM Images can be quite the prohibitive in maintaining all of this. He also mentioned that there has been a major shift in culture with the ETICS project because the scientists that develop now do so thinking about the quality of their code.

A very enjoyable talk about the issues of grids.

Testing Applications on mobile devices - Doron Reuveni

Doron, the CEO of uTest, started by explaining what his company does and how goes about doing these things. He talked about the story from England where an entire village tried to guess the weight of a bull. Not one person got the correct value but the average of all the entries worked out the weight of the bull to within 2 Grams. This is called the Wisdom of the Crowd.

He then started explaining about the differences between crowdsourcing and in house testers or outsourced testers. He said that crowdsourcing fitted in somewhere in the middle between in-house and outsourced. He then also started describing the differences between scripted manual testing and exploratory testing. He said that a lot of the uTest testers fell in the exploratory testing realm. These testers are really good at finding the edge cases that no scripted tester would find and that they are a lot more creative.

Doron said that all of this is really good when testing mobile apps because the smartphone that will win the market will be the one that has the best apps and has the best entry to market for these apps. He said that the crowd is really good with mobile apps because they can use a lot more providers, phone types and data types than an in house team could.

It was a good insight to crowdsourcing

JSTestDriver - Jeremie Lenfant-Engelman

Jeremies talk was discussing the new JavaScript Testing framework that he and other Googlers have created. The new framework allows people to develop JavaScript in a Test Driven Development environment. The new framework was designed with the need to make tests run really quickly in the same way that JUnit tests do and also to be run in the same way. It needed to remove the need to move from an IDE to a browser and press refresh.

Developers like to see that their code is doing a lot of work and that it is doing this work quickly. JSTestDriver does this by having a server that captures browsers and the browsers then poll the server very regularly to see if there are bits of information out there. When there is it runs it and reports the results. This means that we can either run this really quickly or we add this to a continuous integration server.

It also has a very good code coverage model that is being developed so that we can see what code has been executed. Overall it is a very good framework and since it came out I have been playing with it and have even submitted a number of patches. This is definitely something that we need to watch in the future.

Selenium: to 2.0 and beyond - Simon Stewart and Jason Huggins

This is probably the most anticipated talk of the entire conference. It was quite funny to see the entire conference room get all filled up just before the start of the talk. Jason and Simon started talking about the benefits of Selenium and the benefits of web driver. It is great for doing the work that you want when testing the browsers. They then discussed the different issues with doing this as well.

So the solution? Well to merge the two frame works and make one "Uber" framework. Simon and Jason have been doing a lot of work to get the two development branches merged resulting in the code now being moved over to http://selenium.googlecode.com and the next step was to start merging the bug tracking for this. They are going to be creating a hybrid system that allows developers to write WebDriver tests and run them in Selenium and vice versa.

Looking at Selenium 2.x there is going to be a lot more work on browser support for Webdriver natively and work to get it doing a number of this so that we can move away from the "Uber-jar" that is going to be released shortly. There was also a talk by a guy from Opera, who's name I can't remember, who said that they use WebDriver to help them do all of there rendering tests for new versions of Opera, including mobile versions.

This was a very entertaining talk.

Score one for Quality - Joshua Williams and Ross Smith

This was one of the talks that I was looking forward to for the entire time in GTAC. Joshua and Ross, who both work on Windows 7, talked about how they introduced a number of games to do quality assurance. They started out by showing that some people are Actively Disengaged with there job. This means that they don't always enjoy what they are doing and how they are doing it. They then started introducing games into the work people did but they needed to work out a few things like working out what would excite people. They went through a number of games that testers and developers would play until they found the right mix.

During their talk they even got the group involved by doing a competition for who had the "Best Bug Story" and sharing that with everyone. The showed with this that people are competitive and want to compete all the time and its a natural thing. They also mentioned that new developers and testers are coming from the Gaming Generation so love playing games all the time.

The said that we need to do a lot more to get people engadged with their work and they need to start feeling like they are controbuting to this.

A very good talk all in all from the Microsoft guys!

Automated Performance Test Data Collection and Reporting - David Burns and David Henderson

This, unfortunately, you will have to wait for a seperate blog post for this.

GTAC Day 1

2009-10-21T10:47:00.006+01:00

Today was the first day of Google Test Automation Conference. There have been some really good talks and below I am going to give my thoughts on the talks.

Keynote by Prof. Nicklaus Wirth

Prof. Nicklaus Wirth opened the first day of GTAC. Prof. Wirth is a winner of the Turing award and currently working at ETH Zurich. Prof. Wirth worked through the years of Computer Science showing how things have improved over the years. he explained that through those years you couldn't just pull up a debugger and find the issues.

Prof. Wirth also used the famous Dijkstra quote that "Testing shows the presense of bugs, not the absence of them" so we can't just assume that we are making high quality code. He also had a complaint that universities are not teaching programming to the students. He likened computer programming to playing the piano, its easy to learn to play with 2 fingers but to play with all 10 fingers is extremely difficult so its easy to create programs but hard to make really good applications.

Precondition Satisfaction by Smart Object Selection in Random Testing - Yi Wei and Serge Gebhardt

Yi and Serge did a really good talk about creating random objects to go through testing objects. They were talking about how they go about generating random objects and how they are used within testing.

They talked about how they generate lots of objects and generate lots of object very quickly so they needed to work out what they needed exactly so started working out what objects meet the pre- and post-conditions.

They then started talking about how the objects start working in tests and were left to run for many hours. They found over 500 bugs found in 2 open source libraries by applying this random object testing. They found minor bugs to bugs within a lexer which can be quite scary.

Fighting Layout Bugs - Michael Tamm

Michael Tamm gave a very good presentation on how to fight layout bugs. Michael started by talking about all the basic things that we can do and should we be applying them to the our continuous integration servers. This can be done by collecting the page structure,parsing it and passing it to W3C. This can be done for the both HTML and CSS. Unfortunately CSS validator can be too strict and doesn't like browser specific styles.

Michael then moved on to a project that he has been working on about an Open Source project that he has created. This can be found at http://code.google.com/p/fighting-layout-bugs. This project works by firing a WebDriver instance and then takes a number of screen shots and does a bitwise operation to work out where things are overlapping. It was a very a good presentation and will be watching this project very carefully.

Even Better than the real thing - Lessons learned from GWT application - Nicolas Wettstein

Nicolas gave a very good talk about Google Web Toolkit. He talked through all the different issues that can be found when developing software. He showed what can be really bad code examples and how this can be cleaned up.

Nicolas also talked about that there should be a very good design structure such as MVC/MVP. That way people can start creating very good unit tests because a developer can then fake out the right items. Favourite quote from this talk is "Developers are responsible for the quality of the code"

Automatic Workarounds for web applications - Alessandra Goria and Mauro Pezze

This talk was a very interesting talk about having a system that sat between the browser and the backend server. The have developed a system that when a user encounters a bug you can tell the proxy and it will go off and find a solution from a repository.

They were using bugs in the Google Maps API bug tracker and analyzed the issues and if there were any potential workarounds apply them to the page. Unfortunately this is very academic work and does not meet any of the potential psychological needs of users. It also doesn't differentiate between bugs that could be happening.

The talk was very good but academic so don't think it will ever become a real world application.

Achieving Web test automation with a mixed skills team - Mark Micallef

Mark gave a very good presentation on how to get Test Analysts and Test Engineers working very well together. Mark was showing how you start with a team with lots of tasks but they are all manual and how, with the right tools, you can get them writing basic tools.

Mark said that he got the teams using Ruby and Cucumber to start doing basic ATDD and BDD with the test analysts and then getting the test engineers to fill in the technical aspects. Mark did this by seeing what motivated the team and trying to work with their strings.

Mark said that this is the start of the work but the BBC still has a lot of work to do! It was a very good presentation and makes me wonder what motivates testers out there.

And finally...

The lightning talks were really good. Everyone was very entertained by the talks. When they are out on YouTube I recommend everyone having a watch of them.

Great Expectations

2009-06-11T21:43:00.000+01:00

I recently returned from a well deserved holiday with my wife. We went to an area with mountains, mines and sheep (lots of sheep) and while we were there we wanted to get something nice to remember our holiday. The area where we stayed is extremely famous for its mining and thought that a trinket would be an easy find. We were sadly mistaken. We had set our expections and they were not met.

This is something that we all do every day. We have certain expectations that we expect and are expected from us. Working in testing, expections are always high for the quality of work we let through but these expectations come from an age where testers were the gatekeepers. The days where my profession was made up of people trying to get into IT because it was the next big thing.

The main people to have certain unrealistic expectations are our end-users. Or are they just normal expectations? These expectations can mean the difference between a "feature" and a "bug". End-users can spot blatant things that are wrong, like wanting something gorgeous from the mine not just a coaster for my tea.

Using this analogy in software we would say thats a bug. But when an html editor that doesn't correct really bad html for you when it corrects mildy bad html, is it a bug or just they way it is? I would tend to lean towards the "just they way it is" as long as the editing software didn't crash the browser/computer. That reminds me of a professor at university who had the saying "The rubbish you put into a computer is the rubbish you get out". So trying to make the rubbish that the end-user creates before they do helps create good software.

Out thinking the end-user is something that a tester does naturally...but who is the end-user? Are they customers? They are both internal and external customers and the tester needs keep them both sets happy. External customers are rarely spoken to by the tester. We tend to speak to the Support teams and the Product Owner making sure that they have reasonable expectations. This is because they are very important internal customers to the test team. Spotting the customers of the tester is easy but who are the test team customers of?

I like to think that the test team are the customers of the development team but then in the same breath the development team are customers of the test team. When the developer is the customer of the tester they expect to receive a decent bug report with the error report and a way to recreate the issue. When the tester is the customer of the developer, the tester expects a certain amount of testing to have been completed before it getting to the tester. This is the creation of unit tests and test harnesses to make sure that their code works. If you work in an Agile environment then you should expect that if a developer breaks the build then they need will fix it within a certain time. With Continuous Integration servers when it goes red I expect someone to be looking into the error fairly quickly and then resolve the issue. Maybe I have unrealistic expectations but I like to keep the bar high in that area for the quality of the product.

The approach of treating everyone around you as a customer will hopefully mean that you can set their expectations properly. This will lead to less disappointment and help with the Total Quality Management in the organisation.

Designing and Testing with Behaviours rather than requirements

2009-04-23T21:12:00.002+01:00

A few months ago I decided that I wanted to learn something new. I decided that I would have a go on the Google App Engine, learning Python and jQuery. I decided that I would start working on something that would be useful, at least for me, and go from there. I also thought that I would give Behaviour Driven Development a try

I decided that I would work on a project management application. So the first thing I needed was a way to store my backlog and visually see where items were on the stack.

Since I have a stack on bits of paper and want to do this by describing my behaviors for each of my requirements I need a way to describe them that I can easily translate into a test. I went looking through all the different BDD Frameworks (NBehave, Behaviour, JBehave, RBehave,StoryQ, and many many more!) to look for the common phrases and it came back with the phrase that I think is brilliant. It flows off the tongue so easily that getting product owners involved to write them can make an agile project work like clockwork.

The phrase is below. NOTE: The bold items make the main phrase up and the italics make it more readable.

As a product owner
I want functionality
so that something is done
with Scenarios

Scenario 1:
Given start point
When do something
Then expections will be met
.
.
.
Scenario n:
.
.
.

This can easily be coded into different languages and in some of the languages the natural language above is all that is needed but just a little tweaking to get the functions calls sorted. If you are a Selenium nut like me you could even drive with something like FitNesse. Gojko Adzic has explained it really well on his blog so suggest you have a look there.

The other plus point for BDD in Agile environments is its ability to change the BDD test very quickly if the product owner changes their mind. And since product owners change their minds every time they have a cup of tea, and in the UK it can be very often, you need to be able to keep your tests up to date.

Testing through the Credit Crunch - Part 6 - Innovation

2009-03-26T21:13:00.000Z

This is the last in the series and I thought that I would finish it off discussing the innovative culture that is needed in today's climate. If you work in an Agile environment trying to create the next big thing you know not all the components you need have been created. And if they haven't been created you have very little to zero chance that you will find a testing tool to automate your application.

You can automated the units of the code, obviously, but there are parts of the application that you can't test with a unit/integration testing framework or your continuous integration server is taking too long to build and test. I know we have all seen the last issue.

So what do we do then? We need think outside of the box to get a solution. A lot of the testing frameworks have grown from the need for something to fit a certain issue.

The image on the left shows the increase in testing projects on sourceforge.net over 9 years. As you can see there has been a nice growth of testing frameworks. A lot of these have been grown from the need to test something and there are no tools out there. One example of this is Selenium.

Selenium started out as an in-house testing framework for ThoughtWorks. Now its one the worlds best functional testing tool. Next tool to talk about is a Selenium derivative. Selenium Grid was born out of the need to make the tests run quicker. The build was taken forever so Philippe Hanrigou and a few other ThoughtWorkers extended Selenium RC to get it to runs tests in parallel without having to worry the infrastructure out there.

I am currently building a system to pull in usability stats from our web application with a developer using 3 free tools that anyone out there can find and making a nice hybrid test tool. I will document this in a future blog post with my colleague. But this has been born out of the need to do something that no other tool seems to do at the moment.

The skill that is needed by testers all across the board is the ability to tackle projects by thinking laterally. Tools are not always going to be available for everything and record-and-replay will not solve all of the worlds issues. There are a number of harder issues that need to be solved like the Testers-Heads-Up-Display that I discussed in my last post.

All of these items can easily be solved with a little innovation by all the testers out there. I guess the question is, what can you offer to the testing world.

Testing through the Credit Crunch - Part 5 - Visualization

2009-02-11T18:56:00.004Z

Visualization of testing is one of the new and exciting areas of testing. It is being born out of the need to see results and coverage of testing quickly. Remember the old days of writing test strategy documents at the beginning of a project, then rewriting them a week later, and then a week later until you get to the end of the project and the document looks nothing like it did at the beginning of the project. Oh and did I mention that the document never gets completed?!

Now you have stakeholders just assuming you did the right thing, documents that will just pass an ISO audit and a slight resentment to your project manager for allowing scope creep! Then came along strategy documents in a state diagram, like Graphical Test Strategy I discussed before, that allows you to get your document done in a day and sent out to stakeholders. All of a sudden you a hero to your "customers" as they become more involved in the quality of the end product.

You have drawn a couple pictures to show people what you are going to do but now lets have a look at drawing pictures to actually do the testing. If you carry on breaking down the state diagram you are given a beautiful test case diagram. Translating that into a automated test can be done with a tool called CubicTest. CubicTest is an Eclipse plugin that allows you create abstractions for test cases and then gives you Selenium or Watir Tests using a state diagram. Its a good tool if you are constantly using Eclipse but I never use Eclipse

Visualization also offers a lot of value to code metrics. Unit tests are only of value if they test different scenarios and if you are testing different scenarios you will inheritently get good code coverage. Code Coverage is a good way to see that you are testing properly. Well, that you are testing in the right areas. My tool of choice is NCover but thats because I work in a .NET shop.

All of this can be placed into a CI server and all it can then be reported on easily. I use CruiseControl.NET to do my CI and I love that it shows all the passes,ignores and fails of tests from the web front end. I can check up on the developers without having to get their code out of the source repository. The manual testing guys make their scripts update a database. Thanks to simple charting APIs( Google Charts ) I can see how projects are going from one page. So without too much cost and fuss we can see how testing is going and the stakeholders can feel attached to QA of the product from Development to Delivery! As always tools to do what I have talked have open source alternatives.

Unfortunately none of these tools make the lives of testers easier. It works is showing that we are doing well, or not if thats the case, but how do we know that we have a decent amount of code coverage on core libraries? The Microsoft XBoX team analyse their logs and apply Heat Maps so that they can see where their testers have been doing. But what if you took your normal code coverage report and put that through a heat map but put a weighting on the function. The weighting relates to how bad a bug would be if found in this area and if we are to work it how well its tested by the code coverage and the number of tests we can actually get a feel for what we are developing and testing.

Finally the thing that is desperately needed is a Testers Heads Up Display (T.H.U.D). James Whittaker discussed this in his series on the Future of Testing and in his Keynote at GTAC 2008. It takes the idea from games where the H.U.D. allows gamers to be really good at the game. The T.H.U.D will give you little notes when you are working saying that there are potentially bugs in this area and telling you about previous bugs in that area. My idea of the T.H.U.D would give you the information and then record where you have been so that you get a heat map over your application when you want to check weather you are finished in this area.

The faster the iterations get the more testers will need something like this. The average ratio of developers to testers seem to sit around 3:1 and it takes just one developer not to unit test their code to slow a tester down since they have to look for smaller bugs as well as those hard to find that we testers look forward to finding.

This leads me onto my final post in the series: Innovation

Sorry about the spam

2009-01-26T21:24:00.000Z

I would like to apologize for the RSS Spam that everyone has been getting lately. I have moved my Feedburner account to Google. It appears that Google did not test the migration process. I spammed everyone when sorting my CNAME so I can use http://feeds.theautomatedtester.co.uk/TheAutomatedTester as my Feed URL again instead of it returning 404 messages that Google had decided would be better after the migration. I spammed you guys then and then on the weekend Google decided to spam you with a fix that they added.

Sorry for the annoyance that everyone has suffered. Hopefully everything is now sorted and it won't try resend everything again!!!!

Sorry again for the spam!!!!

Testing through the Credit Crunch - Part 4 - Value Open Source

2009-01-13T19:09:00.001Z

By now you will have hopefully realised that running a Test Department doesn't have to be expensive. We have seen that by making the entire company responsible for the quality of the product we take the strain off us doing the developers jobs and lets us find those show stopper bugs.

In this post I am going to discuss why you should value open source tools to automate finding those showstoppers.

Lets start at the small thing to be tools we need at the beginning:

Continuous Integration

How many time in the past have you released something and then suddenly have to roll back the release because not everything works together. Continuous Integration Servers get code out of the source repository. I am a big fan of Subversion as a source reprository, which just happens to be open source!

The top CI server application out there are CruiseControl, Cruise Control.NET, Hudson and TeamCity. 3 of those 4 are Open Source. The one that is not open source, TeamCity, requires a you pay for a licence if you want all the bells and whistles. You should be able to manage on the free version of TeamCity if that the one you decide on.

One of the most important features of all these tools is the ability to automatically run the unit tests that have been created by the developers.

Unit Testing

The de facto testing tools for Unit Testing are NUnit, JUnit and TestNG for .NET and Java. There are loads more out there, like unittest for python, that are used all the time. These tools are great for integrating into Continuous Integration Servers.

In the 2nd post of the series I mentioned that having unit tests allows us to have a safety net to work against. The CI servers will run every developers unit test. This allows developers to do a check-in/commit, once their tests pass, and then get the rest of the tests run. This means that if you make a change you will find out if you have by accident broke something. You will learn to appreciate this safety net but make sure that you don't become lazy and use that as your only testing.

Functional Testing

Web applications have the largest amount of Automation tools for functional testing. Since I also work in this industry I will be very biased towards the following applications.

Selenium
Selenium is slowly becoming the test tool of choice for Rich Internet Applications. Selenium allows you to write functional tests against AJAX applications. Selenium is used by the top Internet application companies and Agile Consultants.

Its developed using JavaScript and can work in all Browsers on any OS. As you would have noticed I am a big fan!
Watir
Developed in Ruby its has a very good following. Test Scripts need to be developed in Ruby and you just import Gems for the Browser that you want to test.

I found that it needed a lot of work to test multiple browsers compared to Selenium. I know some people would say its easier to use that Selenium. I guess its down to the user.

Acceptance Testing

More and more companies are starting to use acceptance testing tools to capture their requirements. Acceptance test frameworks can all integrated into Continuous Integration Servers. They take a plain text requirement and turn it into a Test Fixture.

The main tool for acceptance in my opinion is a manual tool commonly known as the customer. Remember that customers are both internal and external to the company! End users are the best acceptance testers as they are the subject matter experts and spot bugs quite quickly.

In the next post of the series I will discuss Visualization of Testing

Testing through the Credit Crunch - Part 3 - Virtualization

2008-12-18T21:10:00.001Z

So far in the series we have learnt how to save money by becoming more Agile and by implementing Test Driven Development. The move to these can take a little while to implement because of the learning curve involved and/or the inability of some people to move out of their comfort zone.

This post is going to discuss virtualizing our test environments. Virtualisation is fast becoming the norm in data centers because it allow infrastructure managers to build up disaster recovery machines fairly quickly. They can also utilize the hardware of the host more efficiently. Good thing when everyone is trying to become 'greener".

So if the infrastructure people think its a good idea then it must be a good idea for testers. Getting real machines for testing is expensive, not only in terms of the hardware and software, but also the human resources to maintain them. Think about having 10 computers with 10 Windows licences that then need to be maintained. Having a quick look at Dell for a basic computer it would cost at least £2990 for the 10 computers and windows licences and just to make it a round number lets say that it will take £100, £10 per machine, to maintain them for a test cycle.

So we now have a cost of at least £3000 to get all the computers and maintain them for one cycle. That doesn't include the electricity to run all of this hardware. Since its the credit crunch we don't that amount of money to waste. The best thing to do is to virtualise the whole lot.

Spend the same amount of money on a low range server . So now we only have one machine that needs to be maintains. Maintenance cost goes from £100 down to £20. I say £20 because maintenance on servers can be a little more because the hardware is slightly more expensive. http://www.vmware.com/go/calculator will give you a better calculation. I did it and for 500000 machines, defaults in the calculator, I saved £350000+ over a time frame.

So whats the best Virtualisation software available? This is a very hard question to answer. I am big fan of free software so use Microsoft Virtual Server 2005 and VMWare ESXi. I have heard good things about XEN if you have a Linux. The main thing is to chose an application that you feel that you can work with quite well. Both MS Virtual Server and VMWare ESXi have APIs that allow you to manipulate the machines that are running on it. This means that you can clone machines and create a new machine by just running a script. As with Test Automation you won't see the saving straight away but it will save time, hence money, in the long run.

In the next post I will discuss visualisation of testing

Testing through the Credit Crunch - Part 2 - Test Driven Development

2008-11-30T21:50:00.000Z

In the last post of this series I discussed briefly about Technical Debt and how Test Driven Development can help in dropping the debt and dropping the amount of interest gained over the life of the product.

Test Driven Development is the concept that you write your tests before you write your code. Once your code is finished you refactor with the safety that your newly created test will prevent any stupid bugs that you could have introduced. It's what the TDD community call Red-Green-Refactor. Now to most people writing tests first sounds stupid, "How do you know what your code is going to do until you have finished coding it?"
My retort tends to be "You know what you are expecting back from the call. Your test needs to be really small and since an Assert is a test, Assert what you are expecting your method to return! Image that your code exists and test against it." Below is an example of a small test and when writing tests we should try keep them as small as we physically can.

[Test]
public void creditcrunch_Test(){
int FutureTechnicalDebt = 100;
Assert.LessThan(FutureTechnicalDebt,TechnicalDebt.Current);
}

So we know that we need tests first to give us a good safety net for writing our code.In Technical Debt terms we have spent a little less to create/run and report on the test and we prevent anyone else breaking our code. Cost-- and Interest-- which is what we want. Running the test above will fail because there is no code to run it against. Write some code to make it pass if you have the time.

The other reason why developers are starting to like TDD, other than the removal of the ear ache that testers would give them about throwaway builds, is that challenges them to think about the interface into their new object. We have all sat down with the keyboard in front of us, tea/coffee next to us and found that someone has changed the interface your were developing against! Queue the expletives! $*@%*$£ !@)@)£)&! Luckily for us we have good Test Driven Developers who have have discussed the interface and we by proxy prevented interest being added to our debt. Interest--! Communication is key in Test Driven Developments, especially when it comes to interfaces because there may be some integration bugs that will creep in without you realising!

Knowing what the Interface is going to look like is has another major benefit. Mocks and stubs! If we know what the interface to an assembly is and how its going to react we can write tests to test our code and when it needs to speak to the other assembly, which may not have been created yet or makes our tests run slowly like accessing a database or filesystem, we can return the results back that we want our code to handle using one of them. This could be exceptions,dodgy data or good data. Testing those code paths means we can increase our code coverage therefore increasing the size of our safety net! It will also mean that our tests run a lot quicker because we don't have to interact with slow drives or have anyone elses tests interfere with our data. Cost-- and Interest--.

So by creating this safety net we have prevented stupid mistakes creeping into our code. Its these stupid mistakes that could cost our team/company a lot of money. Since companies don't have much money it could mean your job!

The graph below shows how by doing the, what I call, simple things we have brought down the technical debt that we could potentially owe. NOTE: The values used are just arbitary values I found on the internet with a little searching.

In the next post in the series I will discuss Virtualization and what the it can mean to you as a tester!

Testing through the Credit Crunch - Part 1 - Development Environments

2008-11-20T20:31:00.003Z

The credit crunch has gripped the world and not so long ago Gartner suggested that the investment in technology was going to drop. They have dropped the increase in spending from 5.8% down to 2.3%. This may not seem a lot but when companies spend millions on IT each year it adds up very quickly. Unfortunately past experience has shown that testing departments are the ones who suffer first.

More employers are predicting a rise in redundancies in the UK and the other week a politician predicted a run on British Pound.

This doesn't mean all doom and gloom for IT professionals. Over the next few weeks I am going be writing a series of posts with some best practices that I have found. Lets start this series by discussing the Environments we build and test software.

Environments

This is a very ambiguous term so lets split it into the 2 main things I mean. Development Methodology and Testing Environments.

Development Methodology

When creating products it is starting to become common practise to follow Total Quality Management (TQM). The idea is to make sure that quality is in every process of the organisation. By moving to your company to this principle you will be adding confidence to your product to all stakeholders. More confidence in your stakeholders will mean that your department will get a little more funding and since its the credit crunch it must be good!

A lot of companies actually practise this without realising it. The main thing to take away from TQM is that everyone in the company has a part to play in the quality of the product that is delivered!

Agile or Not!

My first few jobs were at financial institutions and thought it was normal to do testing at the end. I worked as a WinRunner Automated Tester. I then started working on some Open Source projects in Java and was asked "Can we see your unit tests? Did you use JUnit?".
I answered quite sheepishly "I tested it by checking that it did what it needed. I don't have any JUnit tests."

Obviously my code got rejected because it wasn't neat and didn't have proper unit tests. There was a few other issues that they raised and to be honest if I was doing the code review I would have rejected it too!

Now I work for an ASP/ SaaS and we do a little Agile. I have been able to see the value of being agile while developing software. The main thing that I have learnt is that Technical/Design debt is something that is inherently low in an Agile Evironment.

Technical debt is the idea that when you release software you go into debt. If you have cut corners it will affect the level of debt you have. If you leave these items in your code it will start gaining interest on your debt. So if you have any TODO's in there that is interest on your debt. If you have poorly designed interfaces then you have added interest on your debt. Putting testing at the end of a project will definitely add interest to your debt! Doing tests at the beginning of the development process will bring down the "interest" accrued against your debt in the development process.

So to help us get through this credit crunch we need to bring down the debt in our software. The best way to do this in my opinion is to achieve this is to move your development team to a Test Driven Development way of thinking. As I said above if you place your tests earlier in the process you will lessen your technical debt. The simple rule when it comes to testing is the earlier you test the cheaper it is to solve bugs. So putting in automated tests at the beginning of the development process will go far in bring down your actual development costs for that project but also for future developments.

In the next post I will discuss further the value of Test Driven Development.

GTAC 2008 - Day 2

2008-10-24T22:29:00.005+01:00

Day 2 of GTAC 2008 has been quite interesting. The talks that have interested me were the talk on Context Driven Testing.

This talk, by Pete Schneider of F5, dealt with what context that tests were being created. It dealt with a common issue that testers and deverlopers have and most of them ignore. It was really interesting in seeing that they were 11 different applications that testers had created and were maintaining but there was a lot of overlap. They found that creating tests in the right context and with the right owners of tests has dramatically made a difference in getting their application tested.

The main thing that I took away from it is; When creating a testing framework ask yourself the following questions

Who is going to write the framework
Who is going to build and maintain it
How are you going to use it
How long will the tests live

The next talk that was really interesting was on Automated Model-based testing by Atif Memon and Oluwaseun Akinmade of University of Maryland. It was a really good talk on creating models for test cases, similar to my Graphical Test Strategy post from the beginning of the year. The difference between my post and this talk was the model is created on the fly by the test code that then builds a WebDriver test case. There is one draw back that I noticed and I know I wasn't the only one. This modelling works well on websites that don't have AJAX. Its a big draw back and they did mention that they are working to get it sorted.

Its value for Desktop applications is really good and I would definitely recommend it. It is able to recreate states for the test and make sure that works well and has good coverage.

The next talk was on the value of unit tests by Christopher Semturs. He was advocating the use of Mocks in your unit tests so that you can get better granularity of your tests. This is something that really interests me because I like making my tests fast because if a test is fast a developer is more likely to run the tests. He was advocating the idea of small,medium and large tests. Its an idea from some Google guys where you do unit tests in isolation and then do pair-wise integration tests. This means that you drop your large end to end tests number dramatically.

GTAC 2008 has been really cool with a lot of clever people talking about different aspects of testing and describing what is needed from testing in the future. Its definitely been worth the jetlag!

GTAC 2008 - Day 1

2008-10-24T00:34:00.005+01:00

Day one of GTAC has been really cool. There have been a number of really good talks. I will put links to the YouTube videos when I know them.

James Whittaker's opening talk was a really good start to the day. One of the main things he talked about was the visualization of testing for the future. He discussed the way that people can visualise when there are new code differences. This can be very useful in making sure that testers can see what is new and what should be concentrated on. I am always keen on using visual ways to test and make sure that the quality is high.

The next talk that caught my attention was "Advances in Automated Software Technologies". This talk discussed the idea of Autonomous Computing in software testing and the idea that you can auto generate test cases against APIs. It is an interesting idea but the way that it was put across was that you can only use this on items where the requirements are rock solid. This is not really something that can be applied to Agile developments.

The talk on Groovy was really good. I have started playing with Groovy when trying to automate SOA testing using SoapUI. Its a nice language that can be strong typed or weak typed at the same time and really useful for scripting. Since I haven't played with it a lot I did learn a lot like you can hook into all languages that use the JVM.

The next talk that really interested me was the last talk of the day. It was on the testability of code. It was looking to see how easy is it to test and making tests simpler. This is important because it means that your tests become manageable and supportable. Vishal Chowdhary was advocating SOCK:

Simplicity of your code
Observe how things work and interact
Control of your tests
Knowledge of what it should be done

He was also saying that we need to make sure that we don't overdesign software because this can add extra complexity when there doesn't need to be!

The talks have been very good and have stimulated some ideas that I want to take back to work! I am now off to the Google Seattle office for a tour the lightning talks.

.Net Gem - How to Unit Test Internal Methods

2008-10-16T22:17:00.001+01:00

I have recently started redoing unit tests for bit of code at work and came across a internal methods and classes that needed to be tested.

This posed a question of: how do I access the internal methods and classes to test them properly from an external assembly?

I found this little gem that a lot of the developers where I work had not heard of, this doesn't mean that its not common knowledge but thought I would share it a little more.

I have put a scenario below of how it might work.

Being a good TDD developer you decide that you want to write your unit tests for an internal method. You create your .Net Class Library structure so you know what your tests need to call to do your asserts. It may looks something like the image below.

So now you want to start writing your tests and notice that your intellisense is not bringing up your internal class. So what do you do next?

Open the AssemblyInfo.cs file that is in your Properties folder. Add the line [assembly:InternalsVisibleTo("Unit.Tests")] somewhere near the top.

Now your Internal Classes and methods are only visible to Unit.Tests assembly.

Your start writing your unit tests and intellisense should be playing Mr. Nice Guy and we should see something like below.

I hope that it has been useful!

Are regression packs still worthwhile?

2008-09-21T21:59:00.010+01:00

Recently I was on forum and noticed the thread "Why do we do Regression testing?". In it the person was complaining about their company's regression pack and asking if it was worth the effort of fixing it?

My answer was a little quick off the mark. I replied saying "Definitely! Its the backbone for each and every build!". But then this got me thinking, How many testers out there DO NOT practice Agile development? How many testers DO NOT work in an environment where automation of tests is 2nd nature?

I do not ask this question in a malicious way, I genuinely would like to know. When I go to SIGiST there are a number of questions in the presentations/workshops by people who have not worked in an Agile environment and, to be honest, seem scared by it. These are the same people when you mention something like "Continuous Integration" and they get that deer-in-headlights look on their face.

This leads me to my next question; Is there a "Rich get richer, Poor get poorer" divide happening in testing? People in conferences mention that the lines between development and testing is blurring. It's called grey-box testing. But your average manual tester in a large non-software institution (Banks, Insurance Companies, etc) have got their "break" in testing by being seconded to the test team for a project and never wanted to leave their secondment. They may not have been given opportunities to learn how to program or may not even have heard about concepts like Agile, TDD and all the other new ways testers do their jobs.

To these people; the answer to my original question is still the same: "Yes! Definitely! Otherwise things that used to work may not work anymore. I would also suggest trying to automate the regression pack as much as possible so that you can concentrate on making testing more efficient. Try automate as much as possible so that you can concentrate on doing testing on those hard to test areas. Quality of software is still the main objective of testing."

Google Test Automation Conference 2008

2008-09-18T15:35:00.006+01:00

I have just got back to work from my summer holidays to find a nice little gift waiting for me in my Inbox!

I have been given an invite to the Google Test Automation Conference(GTAC) in Seattle, USA! GTAC for those who don't know is the annual conference that Google hold. Its an invite only conference where people can just let their geekyness flow!

This year its being held in Seattle, USA. Last year it was held in New York and the year before that it was in London.

Below is a list of the presentations at the conference

Scheduled Presentations

* Atom Publishing Protocol, Testing a Server Implementation by David Calavera
* JInjector: a Coverage and End-To-End Testing Framework for J2ME and RIM by Julian Harty, Olivier Gaillard, and Michael Sama
* Advances in Automated Software Testing Technologies by Elfriede Dustin
* Taming the Beast: How to test an AJAX Application by Markus Clermont and John Thomas
* Automated Model-Based Testing of Web Applications by Oluwaseun Akinmade and Prof. Atif M Memon
* Boosting Your Testing Productivity with Groovy by Andres Almiray
* Practicing Testability in the Real World by Vishal Chowdhary
* The New Genomics: Software Development at Petabyte Scale by Matt Wood
* The Value of Small Tests by Christopher Semturs
* Deployment and Test Automation: Extending the Notion of 'Done' to 'System Tested on a Production Deployment' by Marc-Elian Begin

I am going to try my best to update my blog while I am there so people know how its going. If anyone reading my blog is going conference drop me an email so we can meet up and talk shop!

Tester 2.0 - Definition of an Agile Tester

2008-07-03T21:56:00.002+01:00

I have been to a number of SIGiST Conferences over the last year and have always been impressed by the calibre of the speakers there. There has been a theme lately talking about Agile Development and Agile Testing but the thing that always seems to missing is someone giving their interpretation on Agile Testing. To actually describe what an Agile Tester is! And what they should be doing.

Below I am going give my opinion of what an Agile Tester by describing skill set needed.We are after all only as good as our skill set! I am going to rate them from 1 to 10 with 10 being needed and 1 being desirable.

Skill	Score	Reason
Creativity	10	A good tester needs to be extremely creative when trying to test applications. Developers like to think that they are clever and get all the bugs before its released. Unfortunately this is not true when a tester starts being creative in the way they test the application and breaks it very quickly!
Innovation	10	A good tester needs to be able to innovative with the tools that they have. An example would be in SaaS(Software as a Service) companies who constantly striving to be the next big thing and competing against the likes of Google, Microsoft, Salesforce, etc. This drives innovation in development which means that it drives innovation in testing!
Test Automation	9	Every Agile tester needs to be able to do some form of automated testing. I am not saying that the tester must know how to use every tool that is out there but they need to understand how to create the automated tests.
Exploratory Testing	7	Agile Testing tends to mean Exploratory testing. I know that some people will disagree with this but because testing tends to happen on the fly and appears ad hoc but it isn't. Exploratory testing does have a set of rules that need to be adhered and will mostly bring a number of bugs to the surface just as well as normal scripted testing.
Communication	7	One of the main pillars of Agile Development is communication. This is very important in my opinion because a good tester must be able to communicate bugs to developers and requirements to end users. A good tester will also be able to challenge technical architects on their ideas. I know the last sentence can be very hard to achieve in large organisations but it doesn't mean that you can't send them an email to make suggestions.
Development	5	An Agile Tester should not be afraid to look at a developers code and if need be, hopefully in extreme cases, go in and correct it. I am not suggesting for one minute that they must be able to code entire applications but must have the confidence to look at a bug, spot the error in the code and either write a unit test to break it or point out the line where they think the issue is. An example of this is where doing Paired Programming with one developer we shaved of nearly 70% off the time it took to run a unit test test suite just by doing a one line change to their tests which got them to make one other small change.
Ability to work in pairs	3	I am a fan of being able to knock ideas off each other. I wouldn't do paired programming all the time but I like the concept and how it seems to get things done properly with little fuss.

This is not a comprehensive list but these for me are the most important things. Tester 2.0 is something that a number of testers need to strive for if they do not want to get left behind. There is a definite blurring of lines between development and testing. Gone are the days of "I have finished my part, over the wall it goes!". There is also a move to more "Grey Box" testing because of this blurring.

Interpretations of the V-model

2008-06-09T19:54:00.000+01:00

I have recently been doing interviews and one question that I like to ask interviewees is "Where do you feel testers should sit within the V-Model?"

The V-Model (as pictured below) is a representation of how most "development" tasks have an equal "testing" task. I like to think of it as testings equivalent to Newton 3rd law.

This should be a good thing because it shows that there are different aspects of testing that need to be done during the software development lifecycle.

All the interviewee's seemed to give a different answer to the question above. Being the interviewer I was hoping for one of 2 answers.

The first answer would be one that made me rethink my thoughts on the V-Model. To be honest as an interviewer I am always looking forward to someone making me rethink a topic.

The other answer is that testing should encompass the entire V-Model. From the beginning of the Business Requirements stage to the end of the Acceptance Testing stage. The reason why I think this is because Quality Assurance should be in every aspect of the software development lifecycle.

I know that this is something from the Waterfall methodology but its a good way to see what needs to be done in the lifecycle and can easily be transformed into an Agile development.

ISEB/ISTQB - Is it really worth it?

2008-05-07T21:53:00.000+01:00

I have been reading James Bach's blog for a while and in the latest post he has commented on Alan Richardson's commentary on ISEB/ISTQB.

Finally I have someone that agrees with me that ISEB/ISTQB while being good in theory is pretty useless in real life. I know that most people reading this will be saying "That is a very bold statement!". I know that it is and below I will explain why I agree with Alan.

Background
Testers have been fighting for years to remove the stigma that testing software is easy and that any person can do it. This, I feel, is because software testers used to be glorified data inputters with a little technical background. No wonder developers saw a secondment to testing as a demotion. Then Kent Beck came along and wrote Test-Driven Development: By Example. A developer who came along and made testing fun for developers! Something that testers could never get right! So lets have a look at the pros and cons of having a ISEB/ISTQB certificate.

Pros

Lets start why I think ISEB/ISTQB is good. I just said that testers have been fighting for years to remove a stigma that testing software is easy. ISEB came along and people who didn't work in the industry suddenly thought that testing was a little harder because you needed a certificate.
It also sets out standard names for things so that if one tester speaks to another they are speaking in the same tongue. Its also the latest buzzword so HR people will more than likely to give you an interview when you apply for a job.

Cons

The ISEB/ISTQB foundation course teaches the exact same things that I was taught during my first year of university. So not only do you have to pay your way through university but then have to pay from £140 to £700 to get the certificate.
I met a ISTQB examiner in my last job and he did not seem to understand testing in the real world. So are the testers that are getting these certificates also getting a working understanding of testing? I honestly don't think so!
The other thing, and this is a big thing for me, is that the foundation course book mentions Agile Development 3 times. Those three entries are no more than a quick mention. Is it worth getting a certificate that doesn't give a chapter what is becoming the most common development methodology.

Results
ISEB/ISTQB is unfortunately going the way of the the MSCE certificate of the late 90s. Its becoming no more than a buzz word and I feel is lowering the standard of testing. I know that I am not the only person who thinks this. I read a blog entry not so long ago where testers in India are seen as the poor cousins. Something that we've been working against!

This has to stop and we as testers are the ones who can stop this! Come on testers, lets start a revolt and get certificates to mean something!

Selenium Grid - Why is this the way of the future?

2008-04-27T23:04:00.002+01:00

Update: If you are looking for a Selenium Grid Tutorial I have put it here

I have been working with Selenium Grid since Philippe Hanrigou and others at ThoughtWorks released it at the end of last year. I knew then that it was going to become the standard way for automated GUI web application testing.

As an application it has great potential since it is has all the best bits of Selenium but it does multiple browsers in parallel. Suddenly your testing that used to take n units of time to run are taking 1/n to run. As a practitioner you can get things done quicker, as a manager you can get things done for cheaper! Everyone is a winner.

The other thing that makes the grid work well, and this is a thing that makes Selenium Remote Control work brilliantly, is the ability to use any language to control the test. People use Java, Ruby, .Net, Python and many more to create their test scripts and run it with the appropriate client. If you have the appropriate client application like TestNG for Java or DeepTest for Ruby or PNUnit for .Net you can start running tests with one script in parallel. The one limitation toof using these test tools is that you only really have one browser application, like Firefox, being tested at a time. This is because you only have one instance of the Selenium object in your code.

The way that I have got around this is to have many instances of the Selenium object in my code that each control a certain type. This type could be Internet Explorer 7 on Vista or Firefox 2 on XP.

An example of this would be (Note: I use c# to create my tests and then run them with NUnit)


[Setup]
public void setup(){
//Vista IE7 Instance
VistaIE7selenium =

new DefaultSelenium("hub", 4444, "IE7 on Vista", "http://testserver");

      //XP Firefox2 Instance
XPFirefox2selenium = new DefaultSelenium("hub",4444 , "Firefox2 on XP",

"http://testserver"

);
XPFirefox2selenium.Start();
}


[Test]
public void GridTest(){
VistaIE7selenium.Open("/index.html");
            XPFirefox2selenium.Open("/index.html");
}

As you can see it has the ability to create multiple instances of Selenium and then run them. This then leaves the handling of the Selenium Remote Control up to the Grid. As Philippe says, "It allows you to gridify your tests".

The negatives of Selenium Grid is that its still a work in progress. I know, thats hardly a reason to call it a negative. There are also a number of bugs but its growing section of OpenQA.org so give it a chance! I have 7 Selenium Remote Control instances running against my Grid (all virtualised with MS Virtual Server and WMWare) and I haven't had many issues. I have hooked Selenium Grid into our CruiseControl.NET server to test all the different browsers. I still have a lot of work to do with this but I know that it will be worth it in the end!

If you want to automate web testing and you want to do it efficiently use Selenium Grid! It will make your life a lot easier!

What makes a good automated test?

2008-03-26T09:02:00.000Z

I have been asked to try automate the testing of a wider range of for the group. This is going to be quite challenging since I have not automated desktop applications for about 15 months and the software we make is quite unique. I will also hopefully be teaching the other testers how to maintain the automated tests and eventually create their own.

This challenge has made me think back on my automated testing experience and ask "What makes a good automated test?". A good automated test has all the hallmarks of a good manual test except that its comparatively quick and cheap to run. This means that an automated test is successful when it finds bugs and repeatable where ever the software is installed.

In terms of web testing it would be a test script that can be created and then stored in a central place and be run from the central place. Ideally this would be also run on the continuous integration server when doing builds.

The thing that makes a good automated test great is scalability. I recently watched a YouTube video of the Selenium Users Meet-up. There were Google staff there saying that their testing farm handles 51000 tests and most Friday's all of these tests run. They then went on to complain that Selenium struggled to scale to that size. To be honest at that size I am not surprised!

P.S. To those who are waiting for the next Selenium Tutorial I am hoping to have it complete in the next week for everyone to use.

What would people like for the next Selenium Tutorial?

2008-03-07T20:41:00.002Z

Thanks to everyone who has been using my site and all those who are subscribed to my RSS Feed. I haven't released another tutorial for couple of weeks and have started thinking of what would people like next for a Selenium tutorial.

I am struggling to think what people would really like. I would appreciate it if people would take the time to let me know, via the comments section of this blog, what tutorial they would like next. Please don't think that your questions are too novice. I would like to help as many people out as possible!

I look forward to your replies!

Is documenting test processes a bad idea in an Agile Development Environment?

2008-02-15T16:07:00.010Z

I work in a smallish development team in Southampton as some of you know. There are other development teams in the group in United Kingdom and in France that all fall under the same group of companies of different sizes.

The unfortunate problem that we have is that there isn't a properly documented process for getting everything in place and ready for each testing cycle. Once we have everything ready we have another problem that I will hopefully discuss in a future posting.

This is where I had the thought "Is documenting test processes a bad idea in an Agile Development Environment?". If you have worked in an office with ISO9001 accreditation you will know that everything needs to be documented or if you have worked in a Six Sigma office you will have noticed that you have to put everything you do in a database so the "Black Belts" or "Master Black Belts" can see areas that need improvement.

So back to my question, "Is documenting test processes a bad idea in an Agile Development Environment?". My answer is definitely! And I am going to admit this is a fairly biased answer having started my career in large financial institutions where I was taught that everyone is expendable but their process knowledge isn't. This means that everything needs to be documented. However my thoughts on this is that things need to documented and followed to a point as long as that point does not impact on the creativity of the software developers and the testers.

Its this creativity that allows people to come up with new and exciting ways of developing and testing. Tools like the xUnit and Selenium and many more have come from this creativity. This then needs to be balanced with some form of Total Quality Management (TQM) so that everything can be reproduced if and when needed.

Documented processes can't be completely wrong, if they were then large companies like IBM, Citrix and Microsoft would have never got to where they are now!

What has changed in the last year at work

2008-02-14T21:31:00.001Z

Not that long ago it was my year anniversary with my employer. As with most people it just seemed to blow over but I have had a chance to reflect over what
changes there have been in my time there. The one thing that I can definitely say is that I has been a good year.

When I started there was no testing team which means that there were no definitive test processes and no definitive automation testing.

There was however a team of developers eager to make sure that they delivered a
good quality product.

In the last year I have managed to automate around 40% of the site with Selenium. This has then been installed on the development machines so that the developers can have a

way of developers to unit test thier front end code.

I have managed to get the development team to start using NUnit to create their unit tests instead of developing their own applications to unit test their work. This has meant the number of bugs that get into the builds has dropped dramatically.

These two changes have created a mindshift for all of the developers to a more
agile development cycle. This shift has not been that difficult and the reason
for this is the management team that I report to. They are always open to ideas
especially ones that save money. And agile development on web projects
definitely has this this appeal.

So next thing in my plan is to create an integration server so that all the
different development teams all over the UK and France can check in their work
and be confident that their changes will work on another environment.