Although testing is gradually being recognised as a profession and natural career path within the IT industry, not all know the real art of testing. It’s one of the reasons why e-testing Consultancy has adopted the industry respected British Computer Society’s ISEB tester certification scheme to run a series of structured courses leading to recognised certificates in software testing. Senior tester Don Mills reports.

The initial ISEB (ISTQB) Foundation Certificate course has been a resounding success. More than 48,000 people have now completed these three day courses, run by ISEB accredited training providers. The certification scheme is effectively backed up by the British Computer Society (BCS) and Special Interest Group in Software Testing, which defines career paths for IT professionals.

QC, QA or risk management?

I used to teach people that testing was all about software quality control. This would often start arguments about whether it was quality control or quality assurance (see below), but the point was that I saw its purpose as enabling a project to keep the quality of its work products under control. This was and is done by periodically measuring (testing) work-product quality, and reporting back on aspects that fall below par. And this is necessary because, if you don’t keep a constant eye on it, quality will rapidly drift below par.

But there’s certainly something to be said for the point of view that testing can mean quality assurance. One of my mentors in testing, almost twenty years ago, was Boris Beizer, from whom I learnt that: “The proper use of testing isn’t finding bugs, but preventing them.” This is indeed a ‘quality assurance’ activity: work done to assure quality, by assuring a development environment which doesn’t create bugs, but prevents them instead. From early days, my testing courses have identified several ways in which testing can contribute to creating such an environment.

Later on (I’m a slow learner), I came to associate testing with risk management. It’s a happy fact that a well-designed test case that finds a bug, thereby eliminates a risk, just as surely as one that doesn’t find a bug (since a risk is an uncertainty: something that might go wrong, but hasn’t yet). Testing is important, I tell people, because of the many risks that users of untested software may face. If testers can do nothing else, we can turn those risks into certainties, known problems! (But mostly it’s even better to get them fixed.)

But why do testing?

All of these interpretations of testing carry a degree of truth, but none of them really has the force of a business imperative. As we’re all only too well aware, we are in straitened economic times in 2009 (belt-tightening, and all that). Why should businesses throw scarce money into testing their software before releasing it? And why should they throw yet more money into training people how to do testing? Isn’t it something that anyone can do, from clerical staff taken from behind their desks, to failed developers that you want to put somewhere where they can’t do any more damage?

Well, we’ll deal with that set of ideas below. But first, let’s come to the real purpose of doing testing. Why does software quality need controlling? Why do we need development environments that minimise bug-generation? Why do the risks of faulty software products need to be managed by testing them?

Quality, bugs, and risk can certainly look after themselves without help from testing, but the result will be low quality, bad bugs, and high risk, as sure as eggs is eggs and night follows day. And what really matters to businesses is that poor quality, harmful defects, and high failure rates, are all likely to cost money, whereas testing can save huge gobs of it.

“We compromised on the testing”

Queen Elizabeth II officially opened Heathrow’s Terminal 5 in a ceremony on 14 March 2008. On that day it quickly became apparent that the new terminal was not operating smoothly, and British Airways cancelled 34 flights and was later forced to suspend baggage check-in.

In the first five days of operation, BA is reported to have lost sixteen million pounds. Part of the blame can be attributed to overruns on the construction work, and other problems not directly related to software. But, according to an item in Computer Weekly (“Lack of software testing to blame for Terminal 5 fiasco, BA executive tells MPs” 9 May 2008), significant contributors to the loss included: “a software filter being left on the BAA baggage system,” which had apparently been included to ensure that only test data would be processed (!), and the inability of the servers to “cope with the number of messages the baggage system generated”.

British Airways’ chief executive, Willie Walsh, commented: “If I were to pick one issue I would have done differently, it is that, having recognised the importance of [software] testing and having designed six months of testing, we subsequently compromised on that.”

A context for good testing

Over the past five years, I’ve spent a lot of time guiding certification exam candidates through the material necessary to obtain the ISEB/ISTQB Foundation Level Certificate in Software Testing. As a testing course, it’s something of a disappointment, if you were expecting to be trained in “how to test”. Its real objectives are revealed by its target audience, which includes testers, for sure, but also includes “people in roles such as … project managers, quality managers, software development managers, business analysts, IT directors and management consultants,” not to forget “software developers”.

The truth is that software testing is a much-misunderstood set of activities, and the people listed above, generally speaking, understand it even less well than testers do. This is a pity, since they are the people who create the context within which good testing happens—or doesn’t happen, as is more usually the case. Willie Walsh may have had an excellent team of testers waiting to leap into action, but (like the testers on the infamous Ariane V project, which was a 40-second-long seven billion dollar software disaster) they never really got the chance. It’s significant that Computer Weekly filed their article under “Project Management”, not under “Software Testing”.

In some respects, then, it’s the “project managers, quality managers, software development managers, business analysts, IT directors and management consultants” who need some basic training in testing: what it is, what it can and can’t do; how to use it rather than misuse it, and so on. Sadly, we see very few of them on our training courses. The industry needs to find some way to get the message to them in a clear, unmistakeable, and above all helpful, way.

But what about the testers?

Meanwhile, I had about 250 testers, or would-be testers, in front of me last year for training at the ISEB/ISTQB Foundation level, plus around 100 others for different varieties or levels of testing. The costs of ISEB/ISTQB Foundation training differ from training supplier to training supplier, and from time to time (they’ve mostly seen a respectable decrease this year), but a typical price might be between £675 and £900, including the examination fee. The obvious temptation, in times when funds are scarce, is to save money by eliminating tester training. After all, anyone can do it, no?

But as Tom Millichamp pointed out (Can you afford not to train, T.E.S.T, October 2008), “a test team that is not properly trained to do its job is at best not working to its optimum capacity, and at worst a financial disaster waiting to happen.” Many, many other organisations besides BA and the European Space Agency can attest to this. Testing isn’t something that just anyone can do, or at least, not if it’s going to be successful.

It’s certainly true that schemes such as ISEB/ISTQB Foundation certification have brought about “an accepted formal training structure for learning the key foundational skills of testing” (Millichamp again), and that the ISEB Intermediate and Practitioner, levels to some extent build upon them. And there is no doubt that well-applied testing—especially, testing applied to the prevention of bugs—can generate huge savings both in the development of software, and its use in live environments.

And what about the future?

The ISEB/ISTQB Foundation syllabus is not (I believe) really intended to teach the skills that Millichamp refers to, so much as to make all participants aware that such skills exist, are very advantageous, and need to be learned. This is useful stuff indeed, but I think that some valuable opportunities have been missed, which the future might bring in to a tester training programme.

For example, the original Foundation syllabus, included mention of “cyclomatic complexity”. This has been dropped from all current syllabuses, Foundation, Intermediate and Practitioner —partly (I suspect) because their authors didn’t really understand it. (I beg their pardons if there other, better, reasons.)

This is a great pity, because it’s not really a difficult concept to understand, nor a difficult measurement to make, and neither is it just an attribute of program source code, as the syllabus portrayed it. Rather, it’s a fundamental property of “test models” such as flowcharts, decision tables, state transition diagrams, cause-effect graphs, activity diagrams, and even use case narratives; a property which can be used to help design powerful test sets that achieve high levels of coverage, and disclose high numbers of bugs, with relatively low numbers of test cases.

Such techniques have been shown to find approximately the same number of bugs, as found by double the number of test cases developed by less ‘scientific’ means. Studies of this phenomenon have been reported for both code-based testing, and specification-based testing. The techniques are collectively known under the label, High-Yield Testing (google it if necessary), because they return a high yield of bugs for a relatively low investment in the number of test cases run.

Conclusions

What can we conclude from this? Here are the highlights:

• Software development, and the business use of software, are both risky undertakings, where the risk is that the software may let you down badly, and expensively.
• Failures of software projects, or of deployed software, can be very expensive indeed, and can usually be traced back to failures of or in testing.
• Good testing practices can be learnt the hard way—through raw experience—but it’s much less uncomfortable, safer, and cheaper, to learn the lessons from someone else.
• Tester training costs money, but it’s an investment that can repay itself many times over, provided it hits the right nails on the head.
• But even the best-trained testers can’t work their magic to real effect if the project structure is against them. Those who create the environment in which good or bad testing takes place also need some training.

And finally,

• The ISEB tester certification programme, adopted by e-testing Consultancy, have made a big difference, but not necessarily big enough. More opportunities for providing testers with skills that are of direct value to the business need to be created.

Let’s look to the future, folks.

Don Mills

Senior Tester

e-testing Consultancy Limited

www.etesting.com