Am I doing Too Little or Too Much Automation.

We shall be able to count a large number of possible test cases for any system although fact may be that we might be having too little a time to execute only a few of them. Even if this number may be small, still we would expect to detect majority of our bugs out of these whatever small number of test cases.

Hence identification of test cases to be created & to be executed is an important activity. Random selection of our test cases is not a good approach for testing. Hence for the development of good test cases, a carefully designed approach is necessary

Am I trying to do too much of Automation:
Testers usually commit this common mistake of doing too much of automation & that also too soon. Intuitively, testers would be tempted to feel that having more number of automated tests better it would be. But this would be wrong. This belief would leave us with plenty of poorly automated tests, which shall become quite difficult & uneconomical to maintain.

If we go on adding more & more automated tests we shall land up with unwanted duplication, redundancy & increased cost of their maintenance. Hence it is better to start with small numbers of good but diversified tests and automate them. Here it is ideal to begin with say 10-15 tests involving 2-3 hours of interactive testing.

Am I automating the Wrong Tests:
It is not wise to attempt to automate every test case just because the benefit of automating some of the tests is outweighing the cost of automating them. In fact some of the test cases cannot be automated however if some dedicated tester pulls up his strings, can get success in automating them at a very high cost but with no tangible benefit.

However after gaining some reasonable experience of automation testing one can predict with reasonable accuracy the time it will take to automate a particular test. The decision as to which of the test cases to be automated and which of these to be automated first, is quite crucial & is based upon the likely pay back.

Attributes of test case, which can be likely candidate for automation, are as under:
1) Any test which can be expected to run several times. It is quite natural that more the times a test case is made to usefully run the more beneficial an automated version of it will be. Regression tests, which are made to run every time a new version of software is created, are best suitable for automation.

2) Input verification tests, for example checking that an edit box accepts values in a valid range only. Such tests are quite boring by nature & prone to error for manual operation.

3) Tests, which are expensive to execute manually, for example multi-user tests, which take a long, time to run.

4) Tests, which are difficult to execute manually, for example test cases which are timing critical or are complex to execute.

5) Tests, which require persons having special knowledge, like business knowledge or system’s knowledge can be good candidates for automation.

Attributes of test case, which are unlikely to be candidate for automation, are as under:
1) Any test which will not be expected to run many times. If a test case is not run many times just because there had been no need to run it often (rather than because it could be costly to run it often manually) then it should not be automated.

2) Test cases, which are not important, will definitely not find important bugs. Naturally, If a bug is not important it is not wise to invest time & money in finding it, especially when such bugs are not going to be fixed.

3) Usability test, which cannot be automated because usability, is an issue related to human interactions.

4) Any test which is difficult to automate. Test cases, which would take a lot of effort to automate, are generally not worth automating.

5) Any test which is expensive to maintain. Even if a test case is quite simple to automate but if it is vulnerable to changes in the software it would not be worth automating, because it would be very costly to maintain it.

6) A test case which might have a lot of value in itself, but if it duplicates some or all of the value of an existing test case then the value of the new one gets reduced to almost zero. Such test cases are also not worth automating.

Best Lessons learnt:
1) Where full automation is not warranted, consider partial automation.

For example, it may be difficult to automate the execution of a particular complex test case but it may be possible and beneficial to automate the comparison of some of the results of the test case with the expected results. Alternatively, where the execution of test case cannot be automated may be due to some technical reasons, but it may be possible and beneficial to automate some parts of it like data preparation and clear-up.

2) There is a great learning curve in the beginning, hence it is best not to automate many test cases to start with because they may not be as good & fit for automation, compared to the ones we could automate later on after having learnt more about good practices.

3) It is better to focus on relatively few tests, trying out different implementations and assessing their relative strengths and weaknesses before automating large numbers of tests.

Remember that many case histories have made startling revelation, that over 70% of bugs are found by manual testing & not by automated testing in-spite of the fact that the automated tests had been designed & developed with great efforts of many years. Hence think well before automating your tests.

Keyword: Test Automation, Best Practice,