Crucial Decision – Whether to Automate my Testing or Not

Let us firstly delve upon some of the role of Man versus Tool in a Testing Project.
# Analysis of the Product – Done by Human
# Designing of the Test – Done by Human
# Execution of the Test 1st time – Done by Human
# Evaluation of the Results – Done by Human
# Reporting of the 1st Bug – Done by Human
# Saving of the Code – Done by Human
# Saving of the Results – Done by Human
# Documentation of the Test – Done by Human
# Re-run the Test – Done by Tool
# Evaluation of the Results – Done by Tool as well as human in case of a mismatch
# Maintenance of the Results – Done by Human

Is it not interesting to note that in a software project human intervention plays a major role & can’t be dispensed with? We may be living with a notion that machine or a tool will lighten the burden of humans to great extent, it might be a fair illusion. In practical scenarios, ratio of human effort versus tool effort may be as high as 90:10.

In spite of full knowledge of such odds, test automation eases out the burden of testers to a great extent. That is why many automation tools are getting popularity.

Now Let us see the situations when to automate our Testing Process:
Decision of automating the testing does not remain an intelligent one in all situations. There can be many situations wherein automated testing may not be viable. Thus to decide whether and when to automate or not is a crucial one. A decision in favor of automation requires proper analysis and near drawing of clear-cut boundaries between automated test plan and the manual test plan. Planning needs to be done for the use of programming language like Visual Basic or C# because writing of test scripts itself is a software development activity consuming great amount of time.

There is no hard & fast rule to judge as to what should be tested manually and what should be tested with the help of automated test scripts. It is a matter of experience which itself is a best judge. However there are a few preliminary questions, which a team leader can raise before taking a decision in favor of automated testing of a project. The following considerations shall be of great help to us in ascertaining, as to whether it is worthwhile to automate the testing process of the particular project or not.

1) Consideration of Manpower Skills & Project Complexity:
There are many occasions when Project Managers are found to go in for automated testing projects giving due consideration to the skill levels of the manpower available with them. Availability of personnel of desired skill levels is extremely important for the success of any project.
Few key aspects needing careful consideration are:

a) Understand the scope of the automated testing:
If we are aiming at completely automating all the tests, then the scope of testing is totally unrealistic.

However for introducing automated testing in some of the existing projects or into an absolutely new project – it is better to begin with little but manageable targets. We should not adventure into unrealistic or unmanageable targets. Since the skill levels of all testers & programmers are bound to be different, we should be sure of the capability & experience level of these personnel. To check this aspect, we can ask the team members to write down a few simple utilities which can support our testing project using Visual Basic or C#.

b) Know the automation skill level of the testing team fully well:
If the existing team is quite new to the area of automation, we should not make unnecessary haste. We should allocate sufficient time & resources for imparting proper training to them.

Before starting an automation project for the first time, we would need to reinforce the present team of testers by the induction of experienced automated testing personnel. The level of experience of testing personnel shall be the measure of level of automation the organization shall be able to handle. Thus skill & experience of the testing personnel is of prime importance for the overall success of the automation project.

c) Check the real availability of the experienced testers:
In case experienced & skilled testers are present in the organization, still it is important to ascertain as to whether these people are actually available or not. Many a times it would happen that the projects would take off with strong teams with the participation of experienced members, but over a passage of time such people will migrate to other newer projects.

2) Consideration of Product related aspects:
All applications being tested are not created equal. Generally, when automated test scripts are written, we work behind the GUI, meaning thereby we just create & polish various bits & pieces of the software. We need to do thorough planning to ascertain as to whether these activities are in line with the requirements of our product.
Few key aspects needing careful consideration are:

a) Whether the application is sufficiently stable or not:
We should not attempt creation of automation scripts unless & until the application has acquired a sufficient degree of stability. Otherwise what will happen is that every day you will be engaged in updating your test script to match the changes happening in the unstable application. Hence if we start ambitiously too soon, we might land up wasting the precious time & money. Automated testing is best suited to products which have acquired enough stability of structure and elements.

b) Are we going to test the User Interface:
Some of the automated testing tools are specifically tailored to suit the GUI. Thus for the testing of the GUI of an application under test, we should try to deploy such specifically tailored automated testing tools as compared to others. Some of the cheaper / open source tools may also be helpful.

c) Frequency of repetition of Test Runs:
If we are forced to repeat the test runs on large frequency – say more than ten times, it is worthwhile to go in for automated testing. Computers are best suited to repetitive tasks perfectly without any strain. Regression testing to resolve high-priority bugs needing repetition of the similar test time & again can be called a perfect case for automation. Doing Build Verification Test on a software product to verify its robustness, build after build is also ideal for going into automation due to being of repetitive nature.

d) Compatibility over multiple platforms:
Majority of the software product are expected to be used over different versions of OS. In MS Windows family itself so many versions are available like Vista, XP, Win 2K, WinNT, Win9x etc.. Compatibility aspect of the application needs to be given due consideration & automated testing scripts should be designed to take care of such compatibility aspects.

e) Cost economics justifying the automation:
Automated testing can be time consuming & a bit taxing on the pocket initially, but can lead to addition of a lot of value to the project. Still for smaller projects it may not be economically viable. But for large projects it would certainly prove to be a cost saver.

3) Test-Management Aspects:
Few key test management related aspects needing careful consideration are:

a) Availability of proper Software tools relevant to the project:
Ready availability of adequate number of seats of all relevant software tools is very essential.

b) Introduction of automated testing in between existing testing process:
When automation is to be inserted between the existing testing process, it calls for proper integration of the test scripts with manual tests & ordinary commercial tools. This can become extremely time consuming. Hence important aspect of introduction of test automation in between the other testing process should be given due consideration that it should not have any adverse impact on the budget & overall project time Schedule

c) Time Constraints:
Creation of automation test scripts is nothing but an activity being a part of software development. Adequate consideration needs to be given as to whether we have sufficient time for writing, debugging & maintaining the test scripts, otherwise we can land up in wasteful consumption of time & money.

d) Allocation & Management of Resources:
Testing efforts & their consequent results like test scripts once developed for a particular project must be reusable for future projects as well. Such scripts become valuable assets for the organization for use in future. Proper identification of such assets together with allocation of responsibility (among an individual or to a group) for their maintenance & archiving is very important, otherwise all efforts shall be become a sheer waste.

Based upon the individual experience & vision of the testers many such considerations evolve from project to project.

Key Word: Automation, Testing, QTP, Automated Testing