For successful Test Automation – Presence of a Good Tool is not enough

In today's fast paced business environment, test automation has increasingly become a mission-critical activity. Information systems are no longer back room systems. Today's systems impact the bottom line of a business, give it a competitive advantage and more importantly, are frequently responsible for increasing sales revenue. Consequently, the risks associated with developing these systems have escalated.

How can we adequately test a system not only during its development, but also over the life of the system? Frequently, the answer to this question is "all we need is the right tool".

Unfortunately, while automation may be the answer, a tool is not enough. A Company also needs a proven testing process and a skilled test team structured to meet the challenges of automation. The purpose of this article is to examine how to address the automation challenge and explain how the critical elements of the tool, process and the test team work together to provide the necessary foundation to achieve the payback, cost savings and reduction in defects promised by the tool vendors.

To many IT professionals, test automation is synonymous with capture / playback tools. However the traditional capture / playback method of automation so prevalent in vendor demonstrations is not adequate to address the needs of complex, multi-environment systems. Both manual and automated scripts are required to provide the robustness, maintainability and reusability so crucial to long-term automation success.

Furthermore, multi-tiered object-oriented systems require monitoring of the communication among components during testing. This is beyond the capability of the automation tool. A tester has to do hands-on programming to perform this task. Like all development efforts, the tester must base his scripts and programs on sound requirements - in this case the testing needs of the application. Due to the need for manually coded test tool scripts and supporting programs, successful test automation requires a test team to do development work in addition to their traditional testing work.

The Standard Testing Process

Successful automation mandates a testing process. Just as a developer needs a system development process, testers need a testing process to successfully use the testing tools. The testing process provides the steps, guidelines and techniques, which will ensure practical & successful automation. To achieve the testing and risk management goals of the project, a solid testing process is essential to focus the test automation effort where it can do the most of the good.

Organization needs a testing methodology, to consistently focus testing resources and tools where they are needed to meet the system’s coverage objectives. This methodology can be used to determine the areas most appropriate for automation and to provide an effective framework to accomplish the automation. No matter what testing methodology is utilized, however, the most important point is that a company has a methodology. Otherwise, the full benefits of testing automation will never be successfully achieved.

How to Manage Automation Expectations

In order to implement a successful automation effort, testers need to educate management on a number of different issues. Educating management on these issues could mean the difference between a successful automation effort and a failed automation attempt.

One such important issue addresses integrating test automation into the entire development process. Testers need to educate senior management as well as script writers on how a specific tool will fit into their software development environment and software development life cycle. It is equally important to remind everyone that manual testing does not end when automation is implemented.

Another issue concerns the purchasing of tools and their impact on test planning. Test planning, planning what to test and how to test, becomes much more complex and more important when a tool is purchased. For example, test automation is rarely justified for all testing. The tester needs to determine what tasks make sense to automate and what does not make sense to automate.

Moreover, the organization needs to understand that automation does not eliminate manual efforts on the part of testers. Testers need to maintain automation scripts and verify the execution of automated scripts. This is manual work that needs to be factored into the test team's testing activities.

In addition, in order to automate successfully, testers need training and time to master the test tools. Many managers view the test tools as simple capture / playback programs which a tester can learn and implement in his spare time. This is a misconception. Successful test automation frequently requires the use of complex tools — tools which are much more complex than the just capture / replay tools. In order to use these tools effectively, testers need considerable training. Management needs to be committed to training testers on these tools and to establishing the infrastructure in which the tool operates.

Also successful test automation requires the involvement of the software development team as well as testers. In the past, developers and testers had minimal contact. Frequently, the philosophy was to "throw code over the wall and see if it works". This type of interaction does not work when automating the testing process. Developers and testers need to work much more closely together. Developers have to provide support personnel and technical information on their development methods. They can be asked to use the automation tools during their unit and integration testing. More troubling to the developer will be that test automation may raise concerns about the testability of their code. This especially will be the case if standards are not followed, or if developers use odd, homegrown or even very new libraries / objects.

The organization’s project management team must have a clear understanding about the types of roles and responsibilities required for a successful automation effort. The creation of the test environment starts when an organization purchases hardware and installs a tool. Then the test team and development team need to work together to build and maintain a test automation environment that may include dedicated servers, workstations, databases and the like. Management needs to include development of the test environment in the overall project plan. They should also budget for the resources needed to successfully automate testing.

Successful tool automation depends not only on the test tools but also on a standard testing process and the right test team roles, duties and skills. Tools, process and test teams are the three essential legs of the test automation stool.

Moreover, the automation test team needs to have a blend of testing, programming, and tool knowledge. If an organization wants to reap the automation benefits promised by tool vendors, it needs to use the tool as a complement to manual testing. It also means adopting a strong test methodology and training the test team on the ins and outs of the selected tool in an organization's unique development environment.