Use of test automation tools especially for Graphic User Interface testing is increasingly becoming a trend. This increasing demand of such tools which are generally called Record and Playback tools is due to emergence of new technologies & increased productivity of the development tools and of the programming languages. It may be a Herculean task to convince the client with an argument that four weeks would be needed to manually test a new web application designed & updated within a span of two weeks by the developers.
Gradually large numbers of test teams are turning towards test automation tools. Although testing teams have great expectations from automation tools like miracles during shortage of testing time, scarcity of available resources and complete automation; such expectations rarely get fulfilled. Thus purchasing expensive test automation tool may not be enough & it is likely to become just a showpiece, unless a scientific methodology is adopted for timely implementation of the tool.
Have Realistic Expectations from Automation:
While moving to test automation it is important that the entire team must develop realistic expectations from automation efforts right from their beginning.
Expectation – 1: You should have enough patience & belief that initial tests might be slower while automation test design features are getting designed. Such automation test design features shall be able to save the overall automation time later.
Initially manual testing is bound to be less time consuming as compared to the time taken for developing effective automation scripts. Selection & procurement of the appropriate automation tools itself is a challenging task, because your automation tool shall have significant effects on the overall productivity of your automation effort.
Remember that even the best of the best automation tool can fail to be productive unless a set of effective tool scripts are planned, designed, coded, tested and maintained. Tool scripts in themselves have their own software development life cycle.
Expectation – 2: Don’t expect to be able to automate all the tests. Remember that through a sufficiently good automation testing effort, you will be able to automate only 40 - 80 percent of your total testing, while for the balance you may find manual testing to be more suitable.
An intelligent tester should understand that on many a times it will be quicker to run the test manually rather than automating. In cases of once-only tests it is not economically viable & wise to think of automating the test.
Expectation – 3: Don’t believe that just by having automated testing using a complex & the best tool, you can give a back seat to the effective testing methodologies already in practice. Remember that in the absence of sound testing methodologies and proper planning, implementation of the automated testing tool is bound to fail.
You may be able to run your tests faster, and may be able to execute more tests, but you might not be able to find many bugs.
Expectation – 4: Don’t expect to have miraculously large number of software bugs by automated testing right from the beginning. It might happen that you might find much lesser number of bugs compared to manual testing.
This is a sheer myth that automated tests find more number of bugs compared to manual tests. In fact, many bugs are found while doing manual testing or while doing manual sequencing operation for a Record and Playback tool. In case such a bug is fixed there & then, the automated tool might not even detect this bug even during its first automated run.
That is why automated testing justifies its worth during 2nd, 3rd or subsequent releases of the application under test. Record and Playback tools prove excellently beneficial for regression testing and are extremely useful in detecting unexpected bugs arising out of changes made to the code under test.
Expectation – 5: Don’t expect to have quick return on your investments made in the automation tool.
It has been seen that creation of good test scripts take around 3 – 5 times more time compared to the manual creation of similar tests. However such increased timing may be able to be recovered during subsequent re-runs of the tests. But the more time spent during the initial phases is bound to have effect on the overall project schedule. Thus ROI may be as high as few years for a fully implemented automated testing system.
Expectation – 6: Don’t expect amazingly fast change management through Automated testing.
Remember that whenever changes take place in the basic code of the application due to any reason, the entire test scripts would require similar changes. Here had it been a case of manual testing the software tester would have easily managed changes, but the same might not be true for automated testing. Hence one-to-one changes vis-a-vis code to automated testing scripts is mandatory.
Remember that in case of automated testing the test script shall execute every time in the same way as it has been designed.
Expectation – 7: Don’t expect all testers to program well and all programmers to test well. Record and Playback tools are comparatively easy to handle simple automated tests involving data capture etc. However, they need skilled programming efforts for robust & reliable tests especially where we need to handle features like comparisons, error handling and repetitions etc. etc.
Best underlying Testing Strategies:
a) Test automation tools should be used to give testers more time to design new and more effective tests and to run manual tests.
b) Let the automation tool be used for regression testing to check the performance & the basic functionality of the application
c) Permit more & more time for the testers to create more & more effective test scripts for automation and to concentrate on the areas best suited to manual testing.
d) Since automation generally simulates the manual interactions with the system, manually execute & refine most of the test paths, before selecting them as a prospective candidate for automation. Such manual interactions in the beginning shall act as an effective guide for the automation efforts.
e) Concentrate on small number of properly selected test cases rather than hundreds of randomly or poorly chosen test cases for an effective testing effort.
Keyword: Automation, Automated testing, Testing Work, Tips, Manual Testing
Read more...
Friday, June 6, 2008
For successful Test Automation – Presence of a Good Tool is not enough
In today's fast paced business environment, test automation has increasingly become a mission-critical activity. Information systems are no longer back room systems. Today's systems impact the bottom line of a business, give it a competitive advantage and more importantly, are frequently responsible for increasing sales revenue. Consequently, the risks associated with developing these systems have escalated.
How can we adequately test a system not only during its development, but also over the life of the system? Frequently, the answer to this question is "all we need is the right tool".
Unfortunately, while automation may be the answer, a tool is not enough. A Company also needs a proven testing process and a skilled test team structured to meet the challenges of automation. The purpose of this article is to examine how to address the automation challenge and explain how the critical elements of the tool, process and the test team work together to provide the necessary foundation to achieve the payback, cost savings and reduction in defects promised by the tool vendors.
To many IT professionals, test automation is synonymous with capture / playback tools. However the traditional capture / playback method of automation so prevalent in vendor demonstrations is not adequate to address the needs of complex, multi-environment systems. Both manual and automated scripts are required to provide the robustness, maintainability and reusability so crucial to long-term automation success.
Furthermore, multi-tiered object-oriented systems require monitoring of the communication among components during testing. This is beyond the capability of the automation tool. A tester has to do hands-on programming to perform this task. Like all development efforts, the tester must base his scripts and programs on sound requirements - in this case the testing needs of the application. Due to the need for manually coded test tool scripts and supporting programs, successful test automation requires a test team to do development work in addition to their traditional testing work.
The Standard Testing Process
Successful automation mandates a testing process. Just as a developer needs a system development process, testers need a testing process to successfully use the testing tools. The testing process provides the steps, guidelines and techniques, which will ensure practical & successful automation. To achieve the testing and risk management goals of the project, a solid testing process is essential to focus the test automation effort where it can do the most of the good.
Organization needs a testing methodology, to consistently focus testing resources and tools where they are needed to meet the system’s coverage objectives. This methodology can be used to determine the areas most appropriate for automation and to provide an effective framework to accomplish the automation. No matter what testing methodology is utilized, however, the most important point is that a company has a methodology. Otherwise, the full benefits of testing automation will never be successfully achieved.
How to Manage Automation Expectations
In order to implement a successful automation effort, testers need to educate management on a number of different issues. Educating management on these issues could mean the difference between a successful automation effort and a failed automation attempt.
One such important issue addresses integrating test automation into the entire development process. Testers need to educate senior management as well as script writers on how a specific tool will fit into their software development environment and software development life cycle. It is equally important to remind everyone that manual testing does not end when automation is implemented.
Another issue concerns the purchasing of tools and their impact on test planning. Test planning, planning what to test and how to test, becomes much more complex and more important when a tool is purchased. For example, test automation is rarely justified for all testing. The tester needs to determine what tasks make sense to automate and what does not make sense to automate.
Moreover, the organization needs to understand that automation does not eliminate manual efforts on the part of testers. Testers need to maintain automation scripts and verify the execution of automated scripts. This is manual work that needs to be factored into the test team's testing activities.
In addition, in order to automate successfully, testers need training and time to master the test tools. Many managers view the test tools as simple capture / playback programs which a tester can learn and implement in his spare time. This is a misconception. Successful test automation frequently requires the use of complex tools — tools which are much more complex than the just capture / replay tools. In order to use these tools effectively, testers need considerable training. Management needs to be committed to training testers on these tools and to establishing the infrastructure in which the tool operates.
Also successful test automation requires the involvement of the software development team as well as testers. In the past, developers and testers had minimal contact. Frequently, the philosophy was to "throw code over the wall and see if it works". This type of interaction does not work when automating the testing process. Developers and testers need to work much more closely together. Developers have to provide support personnel and technical information on their development methods. They can be asked to use the automation tools during their unit and integration testing. More troubling to the developer will be that test automation may raise concerns about the testability of their code. This especially will be the case if standards are not followed, or if developers use odd, homegrown or even very new libraries / objects.
The organization’s project management team must have a clear understanding about the types of roles and responsibilities required for a successful automation effort. The creation of the test environment starts when an organization purchases hardware and installs a tool. Then the test team and development team need to work together to build and maintain a test automation environment that may include dedicated servers, workstations, databases and the like. Management needs to include development of the test environment in the overall project plan. They should also budget for the resources needed to successfully automate testing.
Successful tool automation depends not only on the test tools but also on a standard testing process and the right test team roles, duties and skills. Tools, process and test teams are the three essential legs of the test automation stool.
Moreover, the automation test team needs to have a blend of testing, programming, and tool knowledge. If an organization wants to reap the automation benefits promised by tool vendors, it needs to use the tool as a complement to manual testing. It also means adopting a strong test methodology and training the test team on the ins and outs of the selected tool in an organization's unique development environment.
Read more...
How can we adequately test a system not only during its development, but also over the life of the system? Frequently, the answer to this question is "all we need is the right tool".
Unfortunately, while automation may be the answer, a tool is not enough. A Company also needs a proven testing process and a skilled test team structured to meet the challenges of automation. The purpose of this article is to examine how to address the automation challenge and explain how the critical elements of the tool, process and the test team work together to provide the necessary foundation to achieve the payback, cost savings and reduction in defects promised by the tool vendors.
To many IT professionals, test automation is synonymous with capture / playback tools. However the traditional capture / playback method of automation so prevalent in vendor demonstrations is not adequate to address the needs of complex, multi-environment systems. Both manual and automated scripts are required to provide the robustness, maintainability and reusability so crucial to long-term automation success.
Furthermore, multi-tiered object-oriented systems require monitoring of the communication among components during testing. This is beyond the capability of the automation tool. A tester has to do hands-on programming to perform this task. Like all development efforts, the tester must base his scripts and programs on sound requirements - in this case the testing needs of the application. Due to the need for manually coded test tool scripts and supporting programs, successful test automation requires a test team to do development work in addition to their traditional testing work.
The Standard Testing Process
Successful automation mandates a testing process. Just as a developer needs a system development process, testers need a testing process to successfully use the testing tools. The testing process provides the steps, guidelines and techniques, which will ensure practical & successful automation. To achieve the testing and risk management goals of the project, a solid testing process is essential to focus the test automation effort where it can do the most of the good.
Organization needs a testing methodology, to consistently focus testing resources and tools where they are needed to meet the system’s coverage objectives. This methodology can be used to determine the areas most appropriate for automation and to provide an effective framework to accomplish the automation. No matter what testing methodology is utilized, however, the most important point is that a company has a methodology. Otherwise, the full benefits of testing automation will never be successfully achieved.
How to Manage Automation Expectations
In order to implement a successful automation effort, testers need to educate management on a number of different issues. Educating management on these issues could mean the difference between a successful automation effort and a failed automation attempt.
One such important issue addresses integrating test automation into the entire development process. Testers need to educate senior management as well as script writers on how a specific tool will fit into their software development environment and software development life cycle. It is equally important to remind everyone that manual testing does not end when automation is implemented.
Another issue concerns the purchasing of tools and their impact on test planning. Test planning, planning what to test and how to test, becomes much more complex and more important when a tool is purchased. For example, test automation is rarely justified for all testing. The tester needs to determine what tasks make sense to automate and what does not make sense to automate.
Moreover, the organization needs to understand that automation does not eliminate manual efforts on the part of testers. Testers need to maintain automation scripts and verify the execution of automated scripts. This is manual work that needs to be factored into the test team's testing activities.
In addition, in order to automate successfully, testers need training and time to master the test tools. Many managers view the test tools as simple capture / playback programs which a tester can learn and implement in his spare time. This is a misconception. Successful test automation frequently requires the use of complex tools — tools which are much more complex than the just capture / replay tools. In order to use these tools effectively, testers need considerable training. Management needs to be committed to training testers on these tools and to establishing the infrastructure in which the tool operates.
Also successful test automation requires the involvement of the software development team as well as testers. In the past, developers and testers had minimal contact. Frequently, the philosophy was to "throw code over the wall and see if it works". This type of interaction does not work when automating the testing process. Developers and testers need to work much more closely together. Developers have to provide support personnel and technical information on their development methods. They can be asked to use the automation tools during their unit and integration testing. More troubling to the developer will be that test automation may raise concerns about the testability of their code. This especially will be the case if standards are not followed, or if developers use odd, homegrown or even very new libraries / objects.
The organization’s project management team must have a clear understanding about the types of roles and responsibilities required for a successful automation effort. The creation of the test environment starts when an organization purchases hardware and installs a tool. Then the test team and development team need to work together to build and maintain a test automation environment that may include dedicated servers, workstations, databases and the like. Management needs to include development of the test environment in the overall project plan. They should also budget for the resources needed to successfully automate testing.
Successful tool automation depends not only on the test tools but also on a standard testing process and the right test team roles, duties and skills. Tools, process and test teams are the three essential legs of the test automation stool.
Moreover, the automation test team needs to have a blend of testing, programming, and tool knowledge. If an organization wants to reap the automation benefits promised by tool vendors, it needs to use the tool as a complement to manual testing. It also means adopting a strong test methodology and training the test team on the ins and outs of the selected tool in an organization's unique development environment.
Read more...
Introduction to Test Automation
Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.
Over the past few years, tools with graphical user interfaces (GUI) that help programmers quickly create applications have dramatically improved programmer productivity. This has increased the pressure on testers, who are often perceived as bottlenecks to the delivery of software products. Testers are being asked to test more and more code in less and less time. Test automation is one way to do this, as manual testing is time consuming. As different versions of software are released, the new features will have to be tested manually time and again. But, now there are tools available that help the testers in the automation of the GUI which reduce the test time as well as the cost; other test automation tools support execution of performance tests.
Many test automation tools provide record and playback features that allow users to record interactively user actions and replay it back any number of times, comparing actual results to those expected. However, reliance on these features poses major reliability and maintainability problems. Most successful automators use a software engineering approach, and as such most serious test automation is undertaken by people with development experience.
A growing trend in software development is to use testing frameworks such as the xUnit frameworks (for example, JUnit and NUnit) which allow the code to conduct unit tests to determine whether various sections of the code are acting as expected under various circumstances. Test cases describe tests that need to be run on the program to verify that the program runs as expected. All three aspects of testing can be automated.
Another important aspect of test automation is the idea of partial test automation, or automating parts but not all of the software testing process. If, for example, an oracle cannot reasonably be created, or if fully automated tests would be too difficult to maintain, then a software tools engineer can instead create testing tools to help human testers perform their jobs more efficiently. Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion.
Test automation is expensive and it is an addition, not a replacement, to manual testing. It can be made cost-effective in the longer term though, especially in regression testing. One way to generate test cases automatically is model-based testing where a model of the system is used for test case generation, but research continues into a variety of methodologies for doing so.
Read more...
Over the past few years, tools with graphical user interfaces (GUI) that help programmers quickly create applications have dramatically improved programmer productivity. This has increased the pressure on testers, who are often perceived as bottlenecks to the delivery of software products. Testers are being asked to test more and more code in less and less time. Test automation is one way to do this, as manual testing is time consuming. As different versions of software are released, the new features will have to be tested manually time and again. But, now there are tools available that help the testers in the automation of the GUI which reduce the test time as well as the cost; other test automation tools support execution of performance tests.
Many test automation tools provide record and playback features that allow users to record interactively user actions and replay it back any number of times, comparing actual results to those expected. However, reliance on these features poses major reliability and maintainability problems. Most successful automators use a software engineering approach, and as such most serious test automation is undertaken by people with development experience.
A growing trend in software development is to use testing frameworks such as the xUnit frameworks (for example, JUnit and NUnit) which allow the code to conduct unit tests to determine whether various sections of the code are acting as expected under various circumstances. Test cases describe tests that need to be run on the program to verify that the program runs as expected. All three aspects of testing can be automated.
Another important aspect of test automation is the idea of partial test automation, or automating parts but not all of the software testing process. If, for example, an oracle cannot reasonably be created, or if fully automated tests would be too difficult to maintain, then a software tools engineer can instead create testing tools to help human testers perform their jobs more efficiently. Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion.
Test automation is expensive and it is an addition, not a replacement, to manual testing. It can be made cost-effective in the longer term though, especially in regression testing. One way to generate test cases automatically is model-based testing where a model of the system is used for test case generation, but research continues into a variety of methodologies for doing so.
Read more...
Subscribe to:
Posts (Atom)
