Guidelines to select an Appropriate Automation Tool

If your organization is currently using manual means to test your software applications & is believing that it can derive tangible benefits by automating its software testing process, then simply jumping to a conclusion of buying some tool simply because it is popular among many, may not be wise. What should be done now is to take a judicious decision through a scientific process to find out as to which tool will best be suited to your needs. Since this is an capital intensive move aimed at taking your company to the world of Automated Testing, needs careful examination.

When you are shopping for a proper automation tool, you will come across several people ambitiously marketing their products, which may provide solution to variety of automated testing needs. Now the question arises as to whether a particular tool is really suitable to your needs or not. Are you not inclining your choice for a particular tool by making lot many compromises in features. Think that in times to come, your testing needs may become more complicated by the variety of applications coming across for testing & that too under variety of operating systems.

Thus following guidelines shall be helpful in evaluating & zeroing down your choice of an appropriate tool for the job at hand, out of a bunch of many testing tools sold by different vendors.

Guideline – 1: Understand your True Requirement

First of all don’t look at & form any type of opinion about any XYZ tool available in the market. This is high time for doing deep introspection of your real needs. It is wise to prepare a comprehensive list of your requirements of software testing at the present moment. Identify the time consuming problems, which you want to solve with the new tool. Identify the technical capabilities your prospective tool should have to be compatible with the environment of your application.

Following checklist can be helpful in a judicious compilation of your requirements:

a) List down the Compatibility issues: Remember that the tool selected by you has to be compatible with:
# The operating systems supported by your application to be tested

# The development environments under which the application shall be created

# Third party software if any with whom your application needs to be integrated at some stage

b) List down the Users of the Tool
# List down the people who will be actually using the prospective tool. Keep the skill levels of the available persons at the back of your mind.

# Remember that more powerful tools are bound to be more complex as well. If the skill level of the available manpower (who will be expected to use the prospective tool everyday), does not match the complexity level of the tool, believe me, you are likely to land into many problems in smooth implementation of the tool in your organization.

# Think as to whether there is enough time for training your staff within the prevailing time & budgetary constraints, if any.

c) List down the Testing requirements
Technical requirements like the following needs to be listed down before zeroing down your choice on a particular automation tool.
# Identify the types of your own testing problems you wish that your new tool should solve for you.

# Identify the problems faced by you during manual testing.

# Identify the time constraints coming across while making minor changes to your system.

# Identify the shorter regression testing time frames.

# Identify the Test data setup requirements.

# Identify the Defect tracking requirements you are aiming at.

# Identify the Increased test coverage you are looking for.

# Identify the Increase in efficiency of the testing process you are looking for.

Guideline – 2: Understand the constraints you have

You need to understand various factors, which may compel you to drop down some of the tools from your initial selection list. Such crucial factors need to be identified during the early stages of your tool selection process.

a) Environmental constraints

# Environmental constraints can be either related to hardware or the software itself.

# The prospective tool must be able to work on the desired operating systems

# The prospective tool must not dictate the terms of having some specialized hardware for its working.

# Up-gradation of existing hardware like providing more hard disk / More Ram etc. to cope with the requirements of additional scripts and test data likely to be added.

# Consider your likely objections to your new tool running under some specific environment, while the software application might be required to run under different environment or operating system. This issue may gain importance from the consideration of the future use of the new tool.

b) Credentials of the Vendors & their Clientele
It is certain that you won’t desire to hang on with problems with your new testing tool. Certainly anyone would like to have a quick, competent & professional solution to the tool related problems arising may be occasionally.

Following checklist may come handy in such a situation.
# The tool supplier must represent a genuine company.

# The tool as a product & its supplying company must be matured enough.

# The tool may not be worthwhile at all unless there is enough technical support available.

# Find out the clientele of the prospective tool & try to obtain the feedback from such organizations if possible.

# Find out the past history of the prospective tool?

c) Understand the Quality related characteristics
Following quality related constraints of the prospective tool should be helpful
# Identify the skill level required for using the prospective tool

# Verify as to whether it is possible to have multiple user access

# Identify the support and help documentation required.

# Verify as to whether it is possible to integrate the prospective tool with other tools.

# Ensure that there should not be any possibility of getting your data corrupted by the tool.

# Frequency of failure during realistic use

# Identify the budgetary constraints if any. Such financial constraints can restrict your choice of buying a particular tool.

Remember that it is not the question of just purchasing a particular tool by spending some money. In fact it has been seen that in many cases, the cost of fully implementing the tool can be much higher than the cost of the tool itself. Budgetary constraints shall be applicable not only to the tool purchase cost, but shall cover costs of licensing / AMC’s, costs of training and cost of tool implementation etc.

We can be in a position to evaluate various tools available in the market after compiling a comprehensive list of our requirements and various constraints.

Guideline – 3: Shortlist the most likely suitable tools

This is the stage when an extensive research is needed to identify various types of tools available in the market. WWW can be a good place to explore the tool. Preliminary study of the technical brochure of every tool shall reveal the capabilities of the tool fitting your requirements & constraints.

We can identify various features of every tool by classifying them like a) Essential features b) Desirable features c) Less relevant features etc.

a) Essential features: are the ones, which are extremely necessary to meet your requirements within the defined constraints.

b) Desirable features: are the ones, which will make a particular tool standout among many of its competitors. Based upon the presence of variety desirable features, your decision can be favor of a particular tool among many more.

c) Irrelevant features: are the ones, which are not of any great significance & may not be able to provide some tangible benefit to you in your present requirements.

Evaluation of the above types of features is the next exhaustive & iterative exercise. At this stage you should evaluate as much number of tools as possible and try to zero down your focus on around 5-6 tools any one of which could qualify to be the final tool fitting your ultimate choice.

The next step in the process should be establishing a contact with the tool suppliers & organizing a practical demonstration. If possible you can ask for an evaluation or trial version of the tool for refining your decision. At this stage you should clearly explain all your requirements of testing along with set of constraints to the tool supplier, who shall be in a better position to clarify many points left out by you during your short-listing process.

Guideline – 4: Making a Final Choice

Having gone through the above-defined rigorous process of evaluation, the time has come for you to take a decision in favor of one particular tool, which suits best into your slot of requirements & constraints.

a) Final Comparison of Features:
# With the help of all available data now you should be able to draw a clear comparison of the performance related features of the tools as desired by you vis-a-vis features claimed to be present in the technical literature provided by the persons marketing the tool.

# This is stage to go on for verification of the credential of the particular tool from the present users by visiting their web sites or making contact through other channels. This shall help you in refining your choice of a particular tool.

b) Practical demonstration at your site:
If a particular tool has already caught your attention, you can ask the supplier to organize a live demonstration of the tool under your actual working environment. This way you will be able to judge the technical capability of the supplier as well in providing support later-on.

c) Maintenance of Test Script
Proper maintenance of automation scripts & ease of handling them is very important aspect of the tool under evaluation. Although we may not be able to make an on the spot judgement about this capability, but this aspect gets confirmed during the second or third release of testing after implementing the new tool.

d) Final Comparative Trial
Now this is the time for conducting same test on one particular application time & again on each & every short-listed tool independently. This way you shall be in a position to make a final decision in favor of a particular tool out of a box of many.

Key Word : Automation Tool, Tool Selection