Ewert Technologies Blog

Note: this article first appeared in the September 2010 Issue (Volume 23, No. 9) of ASPects, The Monthly Newsletter of the Association of Shareware Professionals.

Test Automation is often seen as the Holy Grail of Software Testing, and anyone who has manually executed the same test case over and over again on various builds and releases of a software application, will attest to the usefulness of automated tests.  If automating software testing is such a good idea, why isn't it implemented more often, and why, when attempts to implement test automation are done, they often end in failure?  In this article I will start by giving a brief history of test automation, followed by how to decide whether test automation is for you.  Next, I will give some benefits of automation, and describe how to begin test automation.  Finally I'll end with a few tips to help you avoid some common pitfalls.

Brief History

The earliest forms of test automation (sometimes called first generation test automation) were accomplished using basic click recording software.  This software could record all user inputs, including mouse clicks and key strokes.  Mouse clicks were recorded using the screen coordinates and key strokes were entered into whichever component currently had the focus.  Tests that were recorded using this kind of software were not very robust; changes in screen size, and timing problems (for example, a dialog box not being displayed quickly enough) could very easily cause the test script to fail.  There was also very little chance for reuse of scripts between various tests, and verification of results would need to be done visually, either by watching the script as it ran or examining screen shots that were taken while the script ran.

In second generation software automation, record and playback software began to be introduced.  This software was a step up, since instead of mindlessly clicking on screen coordinates, this software can recognize the various objects (buttons, text fields, etc.) that make up the application, and be able to interact with them (e.g. click buttons, enter text, etc).  The basic procedure was to manually execute a test case, recording it as you went and then you could play back the test case.  Since application objects were being acted on, changing the screen size was no longer a problem, and the software was smart enough to wait for certain objects (e.g. a button on a dialog box) to appear before trying to click on it.  These new tools also helped in automating test verification because the state of an object could be checked at various points (e.g. the text value in a text box could be checked after the application completed some calculation).

While this was a major improvement, problems still occurred when an application was changed, e.g. a button is added, removed or renamed, or the UI redesigned.  In this case the test script might need to be re-recorded to account for the changes.  Some of the tools began abstracting the application object information to a Object Repository so all the information about a particular object (e.g. button) was stored in one place and could be edited if the object changed.  Many of these second generation tools also began introducing more sophisticated scripting languages allowing testers to organize their scripts into routines and functions, making them more robust, and more resistant to changes in the applications.  One problem with this, is that writing test scripts became a software development project on its own, which would require testers that have programming experience.  In response to this, some companies began marketing so-called third generation testing tools.  These tool claim that comprehensive, robust tests can be created without any programming.  Most tools available today fall into the area of either advanced second generation, with object management and advanced scripting, or third generation (scriptless) tools.

Should I automate?

When deciding whether to automate software testing, several things need to be considered.   Firstly, you must accept that implementing test automation, is a long term investment; a lot of up front work will need to be done before any gains can be realized.  For a larger organization, this means that one or more of your test resources will need to be dedicated to the work of automation, and won't be available for regular testing tasks such as test planning and test execution (I believe that test automation is not something that can be done as a 'side' project, or done in a tester's 'spare' time).  For very small organizations or independent software developers, deciding to automate your application means taking time away from developing new features from your product or developing new products.  For some, this cost may be too high.

Another consideration, is whether you have right resources and skill sets to take this on.  While some automation tools claim that no scripting or programming knowledge is needed, I believe test automaters need to be a hybrid of a tester and a programmer; able to understand the software testing processes and goals, but also possessing technical skills to develop any necessary scripts, and to understand the architecture of the application being tested.

Finally you need to consider the nature of the project or application you are thinking of automating.  Is it something that will be around for awhile and that will be continually worked on?  Automating testing brings value when it can be used often.  If a software application will only have a short lifespan, or is very stable and not being worked on or updated anymore, then it would not make much sense to automate testing for it.

Benefits of automation

With all of the challenges and limitations mentioned, why would anyone want to embark on the process of automating their testing?  If automation is done on the right types of projects and is implemented correctly, it will save a lot of time in testing and/or will help you release higher quality software, with fewer defects.  The main advantage comes from being able to quickly run a set of tests after some changes have been made to your application.  Consider the following example.  You have released your product out to the world, but a defect has been found.  The changes needed to fix the defect are relatively minor but affect several areas of the application.  After making the fix you have several options, you can:

1. Do a quick test and make sure the defect is fixed, then re-release the software.
2. Execute the test cases you have, that you feel tests the areas that were affected by the change, then re-release you software.
3. Run a complete regression test of the of software, then re-release the software.
4. Re-execute all of your test cases to make sure everything is still working, and the re-release the software.

Going from 1 to 4 the amount of time needed increases, but also your level of confidence in the quality of the release increases; so the decision come down to time vs quality.  If, however, all of your test cases were automated, you could choose to re-execute all of you test cases and the time impact would be minimal.  This is the big advantage to automation.

Another advantage is the consistency of the testing.  You know that testing will be done the same way every time; you will know exactingly what was tested and how it was tested.
A further advantage is that it takes the tedium out of testing.  Anyone who has done testing for a while knows that when testing the same features or same application again again, it becomes difficult to remain sharp and vigilant find any defects.  A test script, however, will never get bored!

Selecting a Tool

When choosing a tool to help implement test automation there are several things to look for.  Firstly, the tool needs to be compatible with the type of application you want to test.  Some are for Web applications, some for VB, some for .NET, some for Java, etc.  You will want to make sure the tool will be able to recognize and interact with the objects (buttons, tables, text boxes, etc.) in your application.

In addition to being compatible, the tool should have a strong scripting language, as this will allow you to  modularize your code making your test scripts more robust.  The tool should also have a good reporting of results, to make it easy to determine the status of tests (pass or fail) and to easily determine the reason for any failures (many tools can be configured to take a screen shot of failures).

Some tools also come with test case management software that allow you to create and manage your test cases.  While this is a very nice option, in terms of automation it is not a must.

There a many tools on the market, some commercial, and some free.  The commercial ones can be quite expensive but will often contain many more features to make creating tests easier.  With the free tools, be prepared to do a lot more of the work yourself.  I have listed several tools in the References section.

Implementing Test Automation

The exact details of how to implement Test Automation will depend heavily on the tool you are using, and to some extent will depend on the Application you are testing.  I will present a generalized strategy that I use when implementing test automation.

One of the key things to having a robust test automation system, is to abstract and separate the various actions on GUI components, so that each action is only in one place.  For example, a screen might have a button on it that gets used in multiple test cases, but the code to press the button should only be in one place, so if the button ever changes, the code only needs to be changed in one place.
The approach that I take, is to look at a screen, dialog, or web page, and try to think about all of the small, discreet actions a user could take.  These actions could be filling text into a textbox, selecting a cell from a table, picking an item from a combo box, etc., and then writing a function to do that action.  All of these functions for all of the screens/pages, form an object function library.

Next I look at the test cases that need to be automated.  Often there are groups of test cases that are very similar, e.g. they all perform the same basic actions, but with different data and may have different results.  Using the object function library I create a second level of functions (action function library) that perform these common tasks.

Finally, using the various functions from the object function library and/or the action function library, I take each test case and implement it into a test script.  By taking this layered approach, if and when something in the application changes, hopefully my code will only need to be changed in one or two places.
A final, but important piece that also needs to be considered, is logging.  Because there is coding involved when writing test automation scripts, there most likely will be bugs in the code.  When a test case script fails it is important to find out if the failure was caused by a bug in the application or a bug in the test script.  This is where logging becomes important, so you can trace back and see what happened while the scripts were running.

Tips for starting Test Automation

1. Start small.  If you have a well defined Sanity Test, or moderately sized Regression Test, start by automating these before trying to automate all of you individual test cases.  This will give quicker return on your investment.
2. Don't try to automate everything.  Some tests are not very easily automated, and may cause more headaches than it is worth; start with the easy stuff.  Also, if there are some test cases that you rarely execute, don't bother automating these.
3. Keep the automation up to date.  The bulk of the work in test automation is at the beginning, but after that, the test scripts will need to be maintained, and new scripts added, as the application changes.  If not, the automated test suite will become obsolete and will no longer be useful.
4. Thoroughly evaluate tools.  Most commercial test automation tools, allow you to try them before purchasing.  Make sure you do this to make sure that the tools will work for you application, and meet your needs.  Set up a proof of concept, identifying some of the more challenges tests to automate.  If you can automate these tests with the tool, it should be adequate.
5. Remember test automation is not a silver bullet.  Automated tested won't necessary find all bugs.  For example, general usability is nearly impossible to automate since it is subjective.  Automation is not a straight replacement for manual testing, but should be seen to assist and complement manual testing.
6. For larger organizations, make sure the person to team responsible for test automation documents what they are doing, so other can take over if they leave.

References

QuickTest Pro - https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24^1352_4000_100__  (Commercial for Web, Windows, Java)

IBM Rational Robot - http://www-01.ibm.com/software/awdtools/tester/robot/ (Commercial for Web, Windows, Java)

IBM Rational Functional Tester - http://www-01.ibm.com/software/awdtools/tester/functional/ (Commercial for Web, Windows, Java)

Test Complete - http://www.automatedqa.com/products/testcomplete/ (Commercial for Web, Windows, Java)

Worksoft Certify - http://www.worksoft.com/products/worksoft-certify.html (Commerical for Web, Windows, Java)

Squish - http://www.froglogic.com/ (Commercial for Java)

Marathon Tester - http://www.marathontesting.com/Home.html (Open Source for Java)

Watir - http://watir.com/ (Open Source for Web)