Automated testing tools can save time and resources in the rapidly expanding agile world. TestPlant senior systems engineer Allen J Fisher looks into the future of testing tools.

Test automation is not going away. The faster turnaround of the web and the rapid development of new technologies such as the Apple iPad and Google’s NexusOne are requiring the tester to do more with much less. Added to this challenge are new technologies that are rapidly being developed to satisfy the need for not only functional, but usable applications.

As technology becomes more and more mainstream, graphically rich applications are required as users demand usability as well as functionality. This makes testing the true end-user experience even more important. Skinnable and customised user interfaces also add to the challenge. For example, iGoogle allows the user to add modules based on different technologies and then rearrange them to suit the individual user’s comfort.

It used to be you had Windows which ran on a big lumbering machine sitting on your desk. Now, with Apple and Linux gaining market share, testing more operating systems becomes crucial. Each time you add an OS without dropping one, you’re doubling your testing time.

A large chunk of my testing career has been spent testing a 20+ year old codebase that has to run on several different versions of Windows and the Mac OS. With a huge codebase with lots of interdependencies, adding a platform is no small investment, even a new version of the OS could introduce many problems that could triple testing time. Add to that things like smart phones, netbooks, iPod Touch, and the iPad and the amount of testing time can start to spiral out of control.

Another challenge in automated testing is the initial investment. The initial investment is really hard to justify as release dates loom. Managers may be willing to part with the money for an automated tool, but often the tool becomes shelfware—destined to sit on a shelf and maybe be used occasionally instead of being a critical part of any testing strategy. This is in large part due to an attitude of ‘we need it done now, not done right.’

My experience is that testers just want to test (I am no exception). They want to hit the date, and tend to fall back into their comfort zone of working a few extra hours to meet the deadline. Also, a condition I’ve called ‘QA blinders’ can creep into the mix. This happens when a tester has looked at an application for so long that they do not see problems because they are doing the same test that they’ve always done and don’t see side effects. Tests need to be repeatable, to guarantee an accurate report of the quality level of a given release to management, regulators, certification boards, and users.

Also, cost to train a tester to use an automated tool can be quite high. The automated tester often has to either have a computer science degree or needs to go through two to three weeks of training in order to have success with the tool that they’ve been tasked with. After that, it may still require developer intervention to ensure that the objects are available for the tool to work with. This plays to team dynamics. Testers and developers often have adversarial relationships that make working together or getting information difficult. A tool should be easy enough for a lower level tester to learn, but powerful enough for the journeyman tester to use successfully.

Lastly, there’s platforms; many tools are Windows-centric. My testing career has been spent in a cross-platform, multi-environment world. It is difficult to find tools that test older systems as well as cutting edge ones. This means that you will not get the full value of an automation effort if your tests only run on a single platform. In my specific case, I had to cover six operating systems: three versions of the MacOS and three versions of Windows. The code base was 20+ years old and largely the objects were custom, there was not good model separation, and much of the code was dependent upon which OS you were running. We could not assume that if a test passed on five platforms that it would pass on the sixth.

I’ve been using our eggPlant on projects for the last seven years. I find it flexible and easy to use. To start with, it is a two-computer system, consisting of a controller machine connecting to a system under test using VNC technology.

This makes it able to connect and test any system that supports VNC, including smart phones. It is a truly black-box automation tool, testing what the user sees, by using images of different screen elements to move about the screen as opposed to having to learn what objects are part of the codebase before being able to test it. This means that technologies that other tools can’t reach such as Flash, Flex, or Silverlight are well within its reach. Even if you already have a set of automated tests, it can piggy-back onto that effort and fill in the gaps.

New platforms and technologies are also not a challenge due to this flexibility. Many tools require updates to support emerging operating systems or technologies. Since VNC support is the only requirement, new operating systems and platforms can be inserted into the testing process as soon as they are available, saving valuable time for other efforts.

This flexibility also lends itself to the problem of highly customisable, graphic rich user interfaces. If a screen is rearranged, such as the widgets on an iGoogle page being moved around, it will adapt and locate the image, wherever it is on the screen. As long as it can ‘see’ something, it can interact with it—exactly like a human would.

The image matching schemes are flexible enough to allow you to write a single script and be able to execute it on several systems, regardless of the OS.

Having a single script do the exact same test on six platforms gives confidence that the code is clean regardless of platform. As an example, we had a test that took four hours to run manually. The test consisted of opening hundreds of proprietary files in a desktop publishing system for music and pressing the ‘play’ button to hear the song play. If the program did not crash while preparing to output sound, the file would begin playing back and the test was successful. By spending eight hours automating this tedious procedure, the time to execute this test dropped to 20 minutes, and could be run at any time at the press of a button.

Ease-of-use is very important, especially for the tester new to automation. The software has its own proprietary scripting language called SenseTalk. It is a high-level, readable scripting language with all the features you’d expect from a language such as Perl or Python, but with the readability you’d expect to find in a manual test script. I like to say that SenseTalk is so easy, even a manager can read it.

The time to ramp up is very short. When I first evaluated eggPlant for my project, I was able to write a simple smoke test of my application in a couple of hours. By the end of a week, I had a meaningful set of tests that I could show as a proof of concept to the R&D team. Other, single platform tools that I had evaluated had a much more significant learning curve, in the order of a couple of months.

In the future, I see the need for automation of tests expanding, not contracting, needless to say. teams are going to have to become leaner and faster, and the advent of things such as cloud computing are only going to make the demand for high-quality, quick turnaround only higher. Money and lives can be lost if software is not released at its highest quality.

The progression of software and technology from a select few to everyone having a computer in their home has brought with it a demand for highly visual, highly customisable experiences. The tools of the future will be those that can adapt to a rapidly changing world.

www.testplant.com