What does a quote from a 1940s movie starring Charlie Chaplin have to do with testing and well performing testing centres of excellence (TCE)? More than one would think according to Archie Roboostoff. In fact, some would say it forms the core of some of the best TCE implementations around the world.

A testing centre of excellence (TCE) provides a framework and methodology for bringing specialists and toolsets together that better leverage knowledge, technology and resources. A basic TCE structure brings together test managers, architects, automation engineers, subject matter experts, quality assurance and testers. While TCEs vary across verticals and companies, the basic premise is the same, create an organisation that delivers the highest quality software in the most efficient manner.

Given that mainstream, commercial test automation tools have been around since the late 1980s, organisations have had plenty of time to fine tune and adjust a wide variety of TCEs. From outsourced offshore to localised TCEs, there have been many derivatives that have both failed and succeeded. Every TCE methodology puts a strong emphasis on testing (ie, Feature-Driven Development, Test-Driven Development, Behaviour Driven Development, Insert you methodology here-Driven Development, etc) yet, most development organisations still cannot seem to get it right.

Product deliveries across the globe are still delayed and those that aren’t sometimes have quality issues. When a quality issue is discovered after a product has gone to market, the cost to recover is exponential. Having to pull development resources off future products to fix existing quality issues can severely impact not only the ongoing cost of the development team but will adversely affect the revenue generating capabilities of that organisation.

Now, back to that Charlie Chaplin quote and what it means to productive TCEs. The bottom line, it’s more important to focus on collaboration than it is to focus on the technology within a TCE. Collaboration leads to two benefits within a TCE, predictability and productivity. The impetus of TCEs was to foster collaboration yet as market and product cycles shrink, silos within TCEs are starting to emerge. It is human nature after all, that when under stress, most people shut down to try to recover.

In the past, this lack of collaboration remained hidden given the long delivery cycles of most software applications. Those TCEs that do not collaborate well will not have the luxury of hidden recovery given that the voracious technical appetite for social media driven applications, mobile devices, on demand content, wireless services and connected appliances is rapidly shrinking product lifecycles. Now, more than ever, TCEs are being pushed to the limits to get products out to market that work, perform, and function.

Building a collaborative TCE step 1: Requirements

Evolving a TCE into a collaborative ‘well-oiled machine’ that brings instant productivity to the people within the TCE and constant predictability to the management layers above the TCE is actually easier to accomplish than most organisations make it out to be. The tip of the spear, so to speak, is getting everyone on the same page with requirements. It is notoriously difficult for business users to communicate needs to development teams. As a matter of fact, up to 70 percent of all production defects stem from this stage when requirements are poorly defined.

Requirements come from numerous stakeholders that may be globally distributed and in a variety of formats. This is further complicated by the fact that users often don’t know what they want until they actually see it and the fact that the only constant with requirements is change. This leads to wasteful cycles where development teams try to implement what they understood and the TCE is trying to verify quality on misaligned goals. This results in missing capabilities, unnecessary features, and – down the road – a lot of rework. And that rework quickly adds up, especially when issues are discovered late in the development process.

Visual business scenarios are an effective means of communicating business process flows and concepts but are usually only propagated to the various business and development stakeholders. By sharing process visualisations coupled with simulations with all functional groups within the TCE, this helps reduce ambiguity and presents an instantaneous way to recognise sequencing issues, broken decision points, missing process steps, and opportunities for innovation. The overall benefit to the TCE is that the TCE now has the ability to create test cases while products are still in the planning phase.

Effort put into defining the paths that users can take within an application is a wise investment because this information is used to create test cases for each path. With the ability to calculate the magnitude of requirements risk and complexity, QA staff can identify high risk areas in an application and in doing so help set testing priorities. This in turn ensures that effort is effectively spent and not wasted on over- or under-engineered functionality. Generating tests cases directly from requirements also means that QA staff becomes active participants in the development process from inception and not as an afterthought. By doing this, an environment where test planning and effort estimation can become much more realistic. And thus, they become more predictable and productive because they have been validated early in the process.

By bringing the TCE into the loop during the requirements phase, and linking that information with testing assets, any change to the requirement and any affected policy changes are instantly communicated across the entire SDLC and TCE. This aspect of technology integration is a prime example of focusing on collaboration rather than pure technology that will ultimately drive efficient testing practices and an overall better product.

Building a collaborative TCE step 2:  Collaborative testing framework

Building a successful test automation practice within a TCE is a challenging task for many development organisations. Mature development organisations realise that simple UI test automation techniques like record/playback do not provide the collaboration required for the speed of today’s product delivery cycles.

There are many frameworks for test automation that have evolved over the years. Keyword-driven Testing (KDT) is a widely used test automation technique that many mature TCEs rely on to overcome the disadvantages of simple record/playback test automation. There are, however, some major disadvantages with managing the complexity inherent in KDT. Many applications require that thousands of automation keywords be developed to make use of KDT. Navigating and constructing test cases based on these keywords can be cumbersome and unpractical.

Acceptance Testing Frameworks (ATF) use a similar approach to structuring test cases (acceptance tests) using keywords that are implemented using coded fixtures. These frameworks do not support users in navigating all available actions (keywords). As with KDT frameworks, ATFs do not support collaboration since the structured fixtures (the implementation of the actions) cannot be easily reused and maintained.

Keeping in mind the end goal for a productive and predictable TCE organisation is to bring collaboration to the forefront; advanced testing frameworks should be considered. Again, focusing on people vs. technology, frameworks should be considered that allows testers to create automated tests as early (and change as often) as possible. The ideal state is to allow a test automation engineer to create test cases based on a product that still may be on the whiteboard.

State-driven Testing (SDT) is a framework that drives collaboration and addresses the maintenance and complexity issues of KDT by providing a UI-state transition model. By defining the given state at any moment of the user interface, the test automation engineer instantly knows what steps the application can take from there. SDT reduces the test case possibilities from thousands of allowable actions down to a manageable list of between 10 and 20 allowable actions. SDT uses a domain specific language (DSL) to define the test automation framework which provides a highly maintainable and simple productive approach to structuring a test framework.

SDT is the next evolution of collaborative frameworks with an approach that provides the ease-of-use for creating tests, enabling non-technical subject matter experts to write tests while eliminating the complexities and maintenance issues inherent in previous frameworks that created test silos.

Choosing a highly flexible framework that brings people together from the inception of an application all the way through to the testing endpoint overcomes many issues that are present in today’s frameworks. There are numerous reasons why many TCEs still struggle with test automation projects:

• Lack of required skills (test automation requires technical staff);

• Maintenance effort (test automation is sensitive to application change and volatility);

• Inconsistency of execution (test automation tools can deliver inconsistent results due to unresolved synchronisation points between test scripts and AUT);

• Lack of relevance/test documentation (without associating tests with specific requirements and/or code and without clearly documented test cases, it is not clear what actually has been tested);

• Lack of situational awareness (if a requirement changed, who informs the test automation engineer?).

While no framework is going to be the ‘holy grail’ of test automation, SDT does provide answers to many test automation challenges that remain within TCEs. SDT is an approach that suits organisations that want to deliver productivity through collaboration to their test automation practices without the pain and effort and limitations of relying on less capable commercial and open source tools.

Building a collaborative TCE step 3: Increase agility

As product lifecycles continue to shrink, software development practices are evolving to keep pace. It has become increasingly important that TCEs test and modify software from the perspective of the group that will ultimately decide the fate of an organisation’s business – their customers. With the customer perspective in mind, Agile methods are becoming a more common development approach. Companies value the flexibility, transparency and speed of these Agile development procedures, which are getting adopted more and more even for large development and testing projects.

Agile development enables organisations to deliver products early while making any necessary changes for future roll-outs along the way. Agile testing plays a critical role in this. Collaborative testing at an early stage – and in parallel with always visible requirements – ensures that the quality of the software satisfies all necessary requirements regardless of change. Most importantly, it enables the TCE to find potentially costly errors earlier in the development process, saving time, money and resources.

As stated before, quality assurance teams have typically lacked visibility into what is being tested, the actual test results, the business requirements driving testing and the extent to which they are reflected in the testing requirements. The movement toward Agile adoption promotes environments where organisations can deliver and benefit from real-time visibility into testing activities and the quality status of a project. This is the driving reason to start fostering collaboration early so that old habits can be eliminated. In order to deliver quality software, TCE using Agile testing methods collaborate more effectively with the business to improve project delivery outcomes.

As organisations continue to turn to Agile methods to speed up time-to-market, reduce risk, and improve the overall quality of their software projects, we have pulled together some practical tips for TCE organisations on how they can optimise their software testing procedures:

Test early and often: Not just some, but all relevant test procedures for a given project must be carried out comprehensively as early as possible in the lifecycle of a project. This means that unit tests, functional tests and load tests must be built into a single visible portal so that management, developers, and TCE members are on the same page.

Integrate unit testing: Unit testing is a critical step in the development process that ensures code being used will actually do what it is designed to do and will behave in the way it is intended.

Load test daily: As more software is designed for massively scalable deployments, load testing has emerged as an important step in the development process. The best product in the world will be useless if it crashes under the weight of too many users.

Know the start points: Test starting points must be clearly defined throughout the testing process, so it is important to take starting points into consideration even in the development stage.

Be flexible: One of the benefits of Agile testing is the flexibility it delivers to the development teams. Software development projects are not static – priorities, goals and requirements can be fluid and often change throughout the development lifecycle. Having an infrastructure goal that fosters collaboration can ensure that the team is always testing against the most up to date set of requirements and living up to the flexible environment that Agile development was designed to deliver.

The trend in technology is to bring people together. The first iPhone was laughed at by resident technocrats for not having as much technology as its competitors at the time. By focusing on how people work and bringing them together, Apple has continued to keep the iPhone the market leader in smartphones. Testing practices and TCEs have not followed suit. Rather than focus on how people work together, TCEs have compensated by throwing more technology and tools at the problem. By taking a step back and understanding that open and flexible communication through the tools being used will make sure everyone is on the same page. This collaboration delivers productivity through shortened development and testing cycles while at the same time bringing predictability into the environment. The TCE will be able to see and track how subtle changes from the business that once created uncertainties, are now understood, corrected, and learned from.

Borland solutions are different than anything else in the market. Where competitors might focus on throwing complicated technology at a problem, Borland focuses on fostering collaboration between the people developing the technology. Borland’s first priority is providing customers with tools that turn whiteboard ideas into customer facing products faster than anyone in the industry. From cost savings to faster product delivery cycles, organisations using our tools will continue to build better software faster, regardless of framework, platform or system.

Archie Roboostoff

Director of product management, Borland Solutions

Micro Focus

http://www.microfocus.com/

Apart from our testing top 20, another thing you can depend upon is change. When I first interviewed Axel Naumann for the CERN feature on page 16 of this issue, details had just been released, literally in the same week, about how the astonishing results of the OPERA experiment carried out by CERN in which 15,000 beams of neutrinos were fired from Geneva to the Gran Sasso Laboratory in Italy seemed to show that the neutrinos had travelled faster than the speed of light.

CERN researchers advocated prudence in the face of the early results, although at the time they believed their observations – in which the neutrinos made the 730 km journey 60 nanoseconds, ie, faster than light – were accurate.

Since late September when the initial details were publicised, another OPERA experiment was run, further data were processed and the same results were observed, seemingly confirming the initial flabbergasting results that the neutrinos were indeed travelling faster than light – and thus undermining the whole idea of cause and effect by theoretically allowing scientists to send data back in time. The second set of results were publicised just as I was working on the feature.

In the latest response to the research however, published just yesterday as I write this, a team of scientists led by the Gran Sasso National Laboratory claims to have refuted the OPERA faster-than-light neutrino result. The latest experiment used much shorter pulses to try and remove a potential systematic error from the set-up. To make sure that they could prove the time correlation between the received neutrinos and the originating event, the latest test used pulses three nanoseconds long, with more than 500 nanoseconds between pulses. This experiment claims that the neutrinos arrived 57.8 ns (±7.8ns) too early, seemingly corroborating the original OPERA data.

The Gran Sasso group however says that in attaining faster than light velocities the neutrinos should lose energy by producing photons and electron/anti-electron pairs. This, they assert, has not been observed; their spectrum seems to demonstrate that they didn’t lose that energy, therefore they can’t be passing the speed of light…

If this is the case, then perhaps the guys at CERN need to run the data processing software through a bit more static analysis. Whatever the facts, it’s clear that life on the cutting edge of theoretical physics is never simple – and by the time you read this, there will probably be a whole new set of results to confound us.

Until next time…

Matt Bailey

Why is it often a good idea to employ people with Aspergers Syndrome as testers or developers? This was the question TEST editor Matt Bailey put to Garry Burge, an Asperger’s advocate and campaigner in Australia, who in turn sought the help of Gavin Bollard. The question ultimately found its way into my inbox in a rather grey Newcastle upon Tyne in the UK.

If you type software testing autism into Google you’ll find many much more educated answers to the above question, but just to maintain your attention, Gavin has kindly summarised it for you here with a few of my own edit’s.

The trouble with any article like this is that we walk on dangerous territory, since we are purposefully labelling people, and that’s troubling. It’s important to remember that people are people, and people who are different as a result of a condition are still people. All people with Aspergers Syndrome are different and most have significant differences in their co-conditions. Not all of the information here will apply to all people with Asperger Syndrome. I’m also sure that those of you already working in the testing community probably work with some people with Aspergers, knowingly or unknowingly and already think that the condition doesn’t matter as long as the job gets done.

People with Aspergers Syndrome are often particularly suitable for software development for a number of reasons:

Excellent rote memory

People with Aspergers Syndrome often have a very good memory for facts and events. They may be terrible with people’s names but can often easily remember product keys, IP addresses and even entire routines. In testing, this can mean that they remember a lot of detail about their particular data set and it can result in both increased efficiency and early detection of problems.

In the coding world, this ‘gift’ often means that people with Aspergers Syndrome can develop without having to constantly refer to help and manuals. API calls and their parameters are easily recalled and whole routines used in other projects can easily be remembered without having to open old projects. It makes for much speedier coding.

Hyperfocus

When a person with Aspergers is really concentrating on something, particularly if it’s closely aligned with their ‘special interest’, they can go into a state referred to as ‘hyperfocus’. In this state, the person tends to shut off most input around them and concentrate solely on the task.

It’s this single-minded drive which enables many people with Aspergers complete large amounts of work in a seemingly small amount of time. It also ensures great clarity of thought. The main drawback with hyperfocus is simply that the person can be so absorbed in the subject that they will accidentally miss meeting appointments, lunch and even knocking off time. While this is usually a great feature for employers, care must be taken to ensure that long durations do not cause harm, such as keyboard overuse injuries.

Attention to detail

Software development and testing is one of those places where attention to detail is actually helpful if not essential. Coding relies upon very strict grammatical rules, and in all languages a clear syntax. It helps to have someone who is obsessive about the way that things are written, displayed and put together. Apply the same trait together with a curious mind to testing and fewer stones are left unturned.

Rules orientation

People with Asperger’s Syndrome like routine and structure in their working environment and are often punctual in getting into work on time and finishing tasks/projects on time. Many also like doing tasks to a set of rules and are conscientious in their work as a result and if not then the rules need to be improved.

Unique point of view

People with Aspergers often have quite a different frame of reference which leads to different points of view. Often the new point of view is the result of looking at excessive details and the big picture at the same time – something that most people find difficult. When data and procedures are looked at in this manner, different patterns tend to emerge and these can result in radical approaches which cut development, testing or processing time significantly.

The different point of view is also quite useful when it comes to debugging. People with Aspergers may try to view the code from the point of view of an object or variable. They can tell you what they can and can’t see at the time, leading to easy tracing of undeclared objects and unpassed variables. It’s quite a good debugging technique.

Motivation

The special interest is a major motivating force in the life of a person with Aspergers Syndrome. An employee with a special interest in computers will enjoy computing to such an extent that they do large amounts of research and testing in their own time at home. This results not only in much better education but also cost savings as the ‘paid for’ time that an employee spends on a problem is drastically reduced.

If all this makes a person with Aspergers sound like a superhero, then it’s probably not telling the entire story. People with Aspergers Sydnrome can make very capable employees and in some fields they are actually much more efficient than neurotypicals. Unfortunately, these people have issues too, chiefly in their interactions with other people. You may find that your ‘aspie workers’ have difficulty in social situations but if that’s not a major job requirement, then they’re probably the best people to have in testing and development.

And this is exactly why we created our company. The reality is that despite all of the above in the UK alone there are half a million people with a diagnosis of Aspergers and Autism, with only 15 percent in employment. All that Autism Works sets out to do is positively discriminate; by that I mean that we just make it easier for someone with Aspergers to work with us. To the best of my knowledge there are only a few companies doing something similar in the world, check out Specialisterne, Passwerk and Aspiritiech.

Hopefully by shining a light on what can be achieved more companies will adopt our policies and we will not only create better working environments but also be much more tolerant of people’s differences.

Peter Macdonald

Managing Director

Autism Works Ltd

www.autismworks.co.uk

CERN is the European Organisation for Nuclear Research. Headquartered in Geneva, Switzerland, the world’s leading laboratory for particle physics has rarely been out of the news in the last few months with stories about how the lab might have detected a neutrino travelling faster than the speed of light – a feat thought impossible a few months ago, but now very much under serious investigation.

CERN is one of the world’s largest and most respected research institutions; it has successfully deployed static analysis to improve the integrity of the source code found across a number of projects analysing data from its famous Large Hadron Collider (LHC), the world’s largest and highest-energy particle accelerator which is addressing some of the most fundamental questions of physics and advancing our understanding of the deepest laws of nature.

The LHC lies in a tunnel 27km in circumference, 175m beneath the Franco-Swiss border near Geneva. It was built to test various predictions about high-energy physics, including testing for the existence of the hypothesized Higgs boson and of the large family of new particles predicted by the supersymmetry theory.

During operation, the LHC produces vast quantities of data that have to be processed in order to detect the presence of these exotic particles. One of LHC’s core software ingredients, ROOT, is a program used by CERN’s physicists to store, analyse, and visualise petabytes of data about the experiments. The lab used static analysis from test automation specialist Coverity to help improve their software’s quality. “Better quality software translates to better research results,” confirms Axel Naumann, a member of CERN’s ROOT Development Team. “Like CERN, Coverity finds the unknown; its development testing solution, Static Analysis, discovers the rare, unpredictable cases that can’t be recreated in a test environment.”

Testing at CERN

The experiments conducted around the LHC generate approximately 15 petabytes data per year, equivalent to 15,000 standard disk drives. Given the size and scale of these experiments, CERN has implemented a number of processes to ensure data generated by the LHC experiments are accurate and as bug free as possible. “ROOT is used by all 10,000 physicists, so software integrity is a major issue,” says Naumann. “A bug in ROOT can have a significant negative impact on the results of the LHC experiments and physicists’ data analyses.”

Within the first week of implementing static analysis, CERN’s ROOT Development team found thousands of possible software defects that could have impacted software integrity and research accuracy, including buffer overflows and memory leaks, with very few false positives. To improve the integrity of its source code, the ROOT team spent six weeks on resolving the errors and continues to use the solution in production daily to prevent further software defects from occurring.

“We used a massive amount of testing before,” say Naumann. “The software has to be working properly to ensure the results you are seeing are right. I would say that the amount of effort we put into testing the software is more than we put into its writing. For each stage in the experiment software we have independent tests like unit tests, memory checkers, and performance checkers. We test coverage, we enforce coding rules – and we were actually doing a pretty good job.”

CERN works in a different way to most software development environments. “CERN is not a company, we don’t have teams which do things, we have physicists who write the code to get the jobs they need doing done and then they write the tests because they want to make sure their code actually works and that it continues to work even if other people change their code,” says Naumann. “The important thing is that we realise that we cannot operate without static analysis. We had a massive testing framework before and we thought that dynamic tests were sufficient, and they’re really not, especially when you want to cover the rare cases when dynamic tests don’t get you all the way. Now we have static analysis checking our code and that by itself help a lot. I believe that CERN is not the only one having fallen into the trap that if you just do enough run time testing you will be covered and that is really not the case. Whether you follow ISO or not doesn’t matter, you’ve got to do static code analysis.

“Two out of the four experiments are now using this in production, so any code change will be checked by static analysis,” says Naumann. “It reduces the probability of new bugs coming into the software. The most important change is that the developers feel watched, because they know if they upload sloppy code to the repository there will be somebody looking at it and slapping their hands if it’s not done properly. My physicists tell me that it is a dramatic change in the way they develop their algorithms and I think that is invaluable. Even though there are procedures that call for testing, you should not assume that these will actually help. It helps controlling the issue, but actually getting things fixed still is based on motivation, you have got to motivate the developers to look at the report and deal with them properly.”

Static analysis tested 50 million lines of software code in the LHC software, helping CERN find and fix more than 40,000 defects.

The ROOT of the problem

ROOT is a data processing software used for the Higgs boson experiment. It runs statistical analysis on the huge amounts of data generated by the search for the particle. “It is one of the ingredients for the experiment,” explains Naumann. “Our physicists have their own code and do their own tests but they all use ROOT for the basics like statistical analysis of huge amounts of data, reading and writing huge amounts of data and utilisation of that data.

“The way we test is very mixed,” he adds, “because there is no over-arching structure that people have to follow. In our ROOT development team we use something very close to an Agile approach. We also have continuous testing; we have code reviews and we have small steps that we test regularly – it’s not a formalised Agile approach, but it is very close. For ROOT we have a strong lead developer who basically decides what goes in and what doesn’t feature-wise. We have users around the world who contribute patches and new developments and it’s our job to make sure it all fits together.”

Testing at the boundaries of science

Even though the realm of theoretical particle physics is as cutting edge as it gets in the scientific community, there are still some known quantities and constants that make testing a lot easier. “We know our test coverage and we know how our software behaves. When we release it, we know it is working correctly if it proves the same physics results as the previous version and if it doesn’t we sit down and find out where the difference comes from,” says Naumann.

“We can even measure the effects of software bugs in the physics results because when we have a bug it can change the measurement and we get a wider distribution of measurements. We don’t just measure things once, we measure them billions of times and then we use statistical analysis which gives us a certain distribution of measurement values and if we have bugs this distribution becomes wider. To get the real value we need to measure more often – taking data longer, maybe two years instead of one year.

“We know and measure the effects of software bugs and we take them into account in our physics results. That said, we don’t want to have bugs that are sensitive to physics. At CERN we are looking for the Higgs boson, a particle we assume exists. If we were to have a bug which is triggered by the Higgs particle being found in the detector, that would be a disaster because it could prevent us from finding it!”

Even though, in principle, Naumann’s team have the bugs well under control, they have to make sure that they find as many as possible to improve the software’s efficiency and the quality of their results. “Testing is done by the people who know what the code is supposed to be doing,” says Naumann. “This situation is good and bad at the same time. But it’s not like the developers are testing their own code all the time. This does happen, but there are also more complex stages where the experiment as a whole is testing the software as a whole to verify the results.”

Testing at the speed of light

The remit of the static analysis solution has now also expanded to include the software underpinning specific LHC experiments, which equates to 50 million lines of code. Static analysis is now used by thousands of developers across CERN. Using Coverity’s web interface, even non-professional developers are able to quickly understand and remediate errors in the software, as well as the reports produced from the analysis. Moreover, Coverity has achieved viral adoption across the CERN developer community within a period of two months.

Given the recent news that an experiment had once again appeared to show neutrinos moving at speeds faster than light, the question of whether the ROOT software was used to number crunch the data had to be asked: “I couldn’t say for certain,” replies Naumann, “but I know of no experiment in our environment which doesn’t use ROOT, so I assume that they did.”

Comedian Groucho Marx may be best recognised for saying that he would “refuse to join any club that would have (him) as a member.” When it comes to the Data Privacy Breach Club, we embrace his witticism readily. Companies that have experienced business ramifications of the unintended exposure of Personally Identifiable Information (PII) include those in credit card processing, clothing and home goods, marketing, and gaming entertainment systems.

At this point in software development history, everyone in the testing community should already understand that using production data for testing is not only wrong, but also violates long-established rules and regulations from both governments and industry.

Still, the numbers are staggering. The San Diego, California-based Privacy Rights Clearinghouse reports that more than 542 billion records have been breached in the U.S. since 2005. And if you’re thinking that you just need a better firewall or intrusion detection system, it’s time to reconsider. Of those 542 billion breached records, greater than 32 million were a result of internal mismanagement, business operation failures or insider data theft. What could be more sadly ironic than concerned citizens in Tampa, Florida who purchased home security systems only to discover later that their personal information was stolen and used to file fake tax returns in their names?

The money involved is also astounding. In March 2011, the Ponemon Institute’s annual study of the cost of a data breach put the price tag at $214 per compromised record. In another case, Ponemon estimated that the total exposure of an entertainment and gaming company resulting from a large data breach could total more than $2Billion.

Despite Americans’ trust in legislative healthcare-related security requirements like HIPAA, another Ponemon report pegs the expense of data breaches in the healthcare system at $6 billion annually. In July, a California-based hospital system paid $865,000 to federal regulators to settle claims of unauthorised access to medical records. The complaints cover 2005 to 2009, a time during which hospital employees were repeatedly caught and fired for peeping at the medical records of dozens of celebrities.

These reported healthcare data breaches from 2005 through 2009 – a four-year time period – point the way towards other disturbing statistics. An April 2010 Accenture survey revealed that repeated security breaches are an ongoing challenge for many organisations. Fifty-eight percent of executives polled said they had lost sensitive personal information, and for nearly 60 percent of those who had a breach, it was not an isolated event during the 12-month period. In sum, companies compromised once are not bridging the security gap quickly enough to prevent additional incidents.

The cost of security

Outright expenditures resulting from data breaches encompass numerous areas, including development and implementation of new internal polices, technical remediation of data systems, increased auditing services, public relations programmes, customer notifications, identity theft monitoring services and legal judgments.

Other costs, such as lost customers, are less tangible. Even companies that pride themselves on outstanding face-to-face customer service suddenly fall short when personal data is exposed. Customers truly fear identity theft and they feel violated when they hear media reports or receive a data breach notification.

The 12 question challenge

Answering the questions below can help your organisation examine the challenges you face when handling sensitive data during software testing:

1) How does information flow across your enterprise?

2) How much customer data do you actually need to collect and process?

3) How long do you hold on to PII?

4) Do you have internal security policies related to who has access to which data?

5) Have you trained company associates about these policies and do you monitor their adherence to them?

6) Are your data privacy policies supported by relevant technology to both prevent and monitor for data leaks?

7) Are your customers aware of your data privacy policies and practices?

Are you using production data for testing or other non-production purposes? If so, is it being masked or obfuscated first?

9) Do you know how many copies of production data exist internally and externally with subcontractors and vendors?

10) Do your subcontractors and other vendors understand and abide by your data privacy requirements?

11) Could you pass internal and external audits?

12) How many of your company’s divisions, departments and executives are involved in keeping sensitive data secure?

If your organisation is conscientious about data privacy, the questions above will not have rocked your world. If a few were difficult, it’s time to do more than create a committee to study the issue.

Technology considerations

Data obfuscation is a set of processes by which production data is extracted and transformed so that it may be used for non-production purposes, such as systems development or testing, without compromising any personally-identifiable information it may contain. The trick to doing it well is to minimise the amount of work necessary, while producing a data set with the same logical relationships and characteristics as the original data. One complicating factor is that processes defined for a particular set of data structures will have to be revised if the structures are changed, which is a likely result of the systems development processes that are a primary driver of the need to obfuscate in the first place.

As you envision a new privacy initiative or work to enhance your current practices, consider this list of technology-related questions that relate to the data obfuscation process:

• Is it easy to deploy and customise, requiring minimal custom programming?

• Is it automated and repeatable?

• Can it create realistic-looking data in sufficient quantities?

• Does it maintain logical relationships identical to the original source data?

• Will it work to select and to subset data from all the types of data stores you’re using, including relational, hierarchical or object databases or flat files?

• Will implementing it force you to restructure existing application data stores?

• Will it work with all releases of your operating platform(s)?

• From a cost effectiveness perspective, will it facilitate the security you’re seeking while minimising business disruptions?

After reviewing the questions pertaining to privacy breaches and data obfuscation above, we have one final piece of wisdom from Groucho Marx regarding the gravity of your prospective membership in the Data Privacy Breach Club. In his 1933 film Duck Soup, Marx opined, “I’ve got a good mind to join a club and beat you over the head with it.”

Howard M. Wiener

Director of Professional Services

Direct Computer Resources

http://www.datavantage.com/

At the start of every project we would be invited to a walkthrough of the new project. At this meeting, the product specialist team will explain what the new product will be, why it is required, what the budget is, and what the drop-dead date for delivery is. The latter being the real date required – no giving us an earlier end-date because a slip is inevitable. Starting off with false information is more likely to cause this than anything that professional software engineers might miss.

As soon as requirements are documented, either as part of the business requirements, functional specification or user stories, we would be invited to review them. This may or may not be a formal affair, depending on the software development model being followed. Whether this is a full-blown statement of requirements or a set of user-stories, the perspective of the test community is essential at this early stage, if overall development costs are to be reduced. The tester perspective here will show up likely issues in functionality well before the system is implemented.

If functionality has been defined up-front, we would be given time to analyse them so that we can apply our tester mindset to what has been written. We would firstly look to apply the cause-effect technique to what has been written. Does every effect have an identified cause – in other words, when this functionality is executed, then this next functionality is activated, and so on. In other words, have we identified the when-then? Also, we would place the new requirements in context with the current system functionality. We would ask the question – do they complement what we already have, or are there likely conflicts? Is the user experience likely to be improved or reduced? Have we increased the number of interactions with the system to effect a transaction or reduced it? In other words, we look for usability issues up-front.

When requirements are not documented, we are not shouted down when we ask for some statement of the business need.

Developers, on releasing code into test, would provide us with a set of release notes. The notes would show the areas of functionality changed; the updated versions of software packages; the extent to which testing has been carried out at developer level; outstanding defects and a risk assessment statement. In this way, we are informed on likely risks to product and project as soon as development has been completed. Thus our risk assessment exercise is timely and relevant.

When testing in earnest, we have agreed defect turnaround times. If a defect cannot be addressed by the agreed time, we are informed, rather than having to chase.

When the system goes live, we take time out to celebrate. It is important to take time out to enjoy life. This brings me to the all important work-life balance.

Work life balance

A long while back, or so it now seems, I recall congratulating myself that I was born in today’s world. The world wars long behind us; oil prices meaning that you put another jumper on, not another war; the great outdoors as a child meaning that I could run around the neighbourhood with my friends all day, without a care in the world. At school, you were streamed. If you wanted to go to university, you were pushed, and encouraged. You were not daunted by fees. You just needed to make the grade, and your parents and the government took care of the rest. However, it didn’t stop the ‘too-much-term-left-at-the-end-of-allowance’ dilemma! This wasn’t all bad though, it made sure you were first to apply for those summer jobs.

I remember vividly a discussion with my colleagues when I first started out as an engineer. It went something along the lines of – work five out of seven days every week (the term 24/7 was yet to be coined); have 22 days paid holiday (but dates to be approved by your boss); do this each and every year for the next 42 years; at age 64 work four days out of seven to prepare you for the apparent shock of being the boss of all 24 hours of your day); then be the boss of your whole day. At 22 you can imagine my bewilderment at this projection of my future.

Add to this the life bit, which is not all about fun, and we can no longer measure our free time in days but in hours. Sometimes, life can throw you a curve ball or two. Real life comes with real problems. You have to man-up and face them. The family springs into action and work provides a welcome break. The work-life balance now takes on new meaning. You need work to balance life. It provides shape and structure to your day. My wish is that life’s curve balls are few enough so that we can keep complaining about work!

This brings to mind a dinner party. My friend’s husband was loading up the dishwasher. The dishes were many as were his frowns. He had a very stern look on his face. I asked if he was ok. Yes, he said, but if I look like I’m happy she’ll ask me to do it again!

Then try counting your ‘me’ time. This is in minutes in today’s world. In today’s world, the technology means that we no longer need to wait for things, we can demand instant responses. Salespeople can call right into our home, just as we’re having dinner with the family; or they can call on a Sunday, when we’ve finally managed to wind down and get into that lazy Sunday afternoon feeling. I can switch off the mobile, but not the landline – my wish is that this channel of communication is for our families, and is respected as such.

When we provide our details for a specific need, my wish is that data protection laws also apply to us.

Encouragingly, I see more and more families on their bikes around town. Our elongated autumn has been a wonderful gift to a southern softie; guess there’s a few of us around. The evening football field was buzzing last Friday night with the happy laughter of children and their parents. My wish is that we remember to switch off the technology and go out and play ball.

As we head into 2012, my personal New Year’s resolution is to resolve not to feel guilty about not making one, yet again.

Best wishes to one and all for a very Merry Christmas and a Happy New Year.

Angelina Samaroo

Managing director

Pinta Education

http://www.pintaed.com/

I am a tester in the team that is developing a testing tool called Twist. In past I have worked on variety of tools including QTP, Loadrunner, SoapUI, etc. I remember using QTP 8.0 in past to automate one of the legacy application developed in .NET 1.1, this was some three to four years ago.  I was reading some blogs where I was reminded of QTP and I decided to download the latest version of it and see what’s new with the tool and also to see how it’s keeping pace with the changing scene of the IT industry where Agile methodology is becoming more popular and replacing traditional Waterfall methodology.

So I downloaded QTP (Trial Version 11) from the website. I have a Dell Machine with Windows 7 installed on it. Browsers installed on my machine are: Firefox (Version 6.0.2); IE (Version 9.0.2); and Chrome (Version 13.0.782.220). As part of my regular testing activity I keep my browsers updated to see whether our tool (Twist) supports latest browsers. I installed QTP with one click, a very easy process. It ships with four default add-ins : Standard Windows Application; Web Object/Applications; ActiveX Controls; and Visual Basic Applications.

QTP allows me to setup the environment before recording the test. So I can navigate to Automation, Record and Run Settings. I can set the application URL there and when I pull the browser drop down, it just shows up Microsoft Internet Explorer.

Though I have Firefox and Chrome installed on my box, they do not shows up in the list. So when I opened the “HP QuickTest Professional Product Availability Matrix” pdf (present in installed location) I was shocked to see the matrix Below (fig 2 & 3). Are you serious?

If I am testing a web application I would want to test it on various browsers available, or  at least Firefox (Version 4, 5, 6), Chrome (Version 12, 13) and IE 8, 9 (on Windows 7).

Read more here about the supported web browsers (www.thoughtworks-studios.com/twist-agile-test-automation/2.3/help/system_req.html)

When it comes to web application testing, QTP still lacks basic browser support. We have so many browsers around and Firefox is releasing new versions aggressively. Also Amazon has recently launched a new browser on their device ‘Fire’ called Amazon Silk and people are expecting the OS version of that browser soon so that they can check compatibility with their websites. So QTP is way behind other functional test automation tool by just supporting IE as the browser available for recording. This is precisely one reason why I would use Twist for web application testing and not QTP.

Once I am past the record and run setting, I start using it to automate test against my sample application which is a project management tool ‘Mingle’. I have used QTP before and I am aware of the basic things like test creation, actions, keyword-driven approach etc. I found the ‘Process Guidance’ really good for first time users. It guides the user as to ‘how to create and organize your test in a step by step manner’ (One thing which is not present in Twist and I would want to add). This is mostly because my first 30 minutes with the tool forms my basic opinion. So a tool should be easy to use with a rich user experience.

I started by creating a layout of the project by creating actions which are logical names for Test scenarios like ‘Login’, ‘Create Project’, ‘Create Card’ etc in QTP. So once the project layout was created in QTP I started recording the test. Recording is very simple in QTP and remains one of the powerful features of the tool.

QTP shows me the recorded script and I can add comments to that script to say why it was recorded. However recorded script doesn’t help anyone apart from me to understand what this test does. So it remains very QA oriented. Action name helps me understand the intent of the test but doesn’t convey much about what my test is doing. It’s not test case based.

Let’s go to the tool I am developing and using: Twist. Once the project layout is created, I can write what my test will do, basically, what the test steps are. I can further break down my test scenario into different test steps and record each of them one by one. Also, all this is in plain English and any one can understand it. So it’s easy for me to write my acceptance test cases and then automate it.

When I record the test for each step, it generates an underlying Java method which looks as shown below. I have highlighted the method name.

I have my tests written in domain-specific language and it can be easily read by any one on the team irrespective of their knowledge about the code. Anyone can check out the code from the repository and read it for their own understanding. This basically takes away the pain of interacting with two testing tools/software: one to write my manual test cases and other one to automate.

This is another reason why I would use Twist for test automation and not QTP.

Analysing the tools

Let’s analyse both the tools on the aspects of test framework and test automation maintenance. When you launch QTP, you will notice that the Process Guidance panel shows you a path to achieve what QTP has popularised in the testing community as ‘keyword-driven testing’.

“Keyword-driven testing is a technique that separates much of the programming work from the actual test steps so that the test steps can be developed earlier and can often be maintained with only minor updates, even when the application or testing needs to change significantly.”

This technique relies on three pillars:

• Test steps;

• Keywords;

• Object repository.

A tester generally writes test steps using the keywords, while a tester with technical knowledge creates the function library using built-in keywords or their own user-defined keywords and maintains the Object Repository. So a slight change in application would require changes in the Object Repository and then the functions.

“The keyword-driven methodology is especially useful for organisations that have both technical and less technical users because it offers a clear division of automation tasks. This enables a few experts to maintain the resource framework while less technical users design and maintain automated test steps. Additionally, after the basic infrastructure is in place, both types of users can often do their jobs simultaneously.”

QTP Help File on Keyword Driven Testing.

The QTP keyword-driven testing approach actually requires a significant manpower to create a test suite and maintain it. Also if you are following this test framework approach you need to put a significant effort into creating the infrastructure and then start writing the test cases. So it takes up a considerable amount of time before you see the benefits of QTP.

On the other hand if you are using Twist, it’s like you download it, install it and start automating your tests right from the day-one. You see an immediate benefit from the tool unlike QTP (Where you will have to wait for quite some time, maybe a month, two months or 15 days till the basic infrastructure is built). Twist does provide a user some guidelines to use the tool but you don’t need such heavy infrastructure to write tests. All you need is the domain knowledge or the application under test.

“Writing your scenarios at the right level of abstraction is critical to building an effective test suite. Here are a few guidelines that can help you get there:

• Tests should be independent of the user interface of the application;

• Tests should strive to be at the highest level of abstraction possible, without sacrificing readability;

• Don’t repeat yourself: Repeating a series of steps at any level — at a scenario, or in the implementation can increase the maintenance overhead;

• Use Extract Concept and Push to Implementation to achieve granularity that reflects your domain and also promotes reuse;

• Your test should ideally be an illustration of the functionality as opposed to some UI mechanics. For instance, if a test reads:

Shopping cart – enter ‘5’ in units and ‘10.0’ in price and select ‘air shipping’ in shipping dropdown.

The description of the UI clutters the intent here. Instead the test would be better as:

Shopping cart – create order for ‘5’ units at price ‘10.0’ for ‘air shipping’.

This helps to clarify the intent.”

Twist Help File on ‘Guidelines for your Test Suite’.

Following a page object pattern is a nice way to group all the actions that are possible on a given screen. Twist provides you more benefits than QTP when it comes to test suite creation and maintenance.

Nishant Verma

Senior consultant

www.nishantverma.com

The idea of a living lab has become fashionable in some quarters. It is a place where users or potential users can interact with software designers, ICT service providers and managers to explore new ideas and new solutions. It is often a place where public sector, private sector and the ‘people’ can meet.

It is not a user testing facility or a focus group but a significant development of those ideas in a move towards the empowerment of users in influencing the systems that affect them. These living labs are often focused on a particular town or city and many of the examples involve local authorities, university researchers and the private sector. They may have a specific focus, eg e-services for citizens, healthcare etc.

There are some in the UK – Coventry, Manchester, Newcastle, Sheffield etc. The European Network of Living Labs (www.openlivinglabs.eu) has more details.

Testing in the living lab

What is the connection with testing? Clearly, having access to users of systems in realistic environments and with the appropriate facilities for observing their behaviour when using prototype software, is one benefit. But it is more fundamental than that. The real benefit is to allow users to contribute more the identifications of needs, the outline of potential solutions and the integration of new services into a coherent ICT environment.

So far so good but the concept of user-driven innovation is only just beginning to percolate through to the industry and to those who commission software. It raises a number of issues for developers and testers. The first, and one I have discussed before, is that testers should be involved at the beginning of all projects in order to help focus the project on outcomes and performance rather than on abstract design. Not all software developers and managers are comfortable with that. What the Living Labs does is to bring the users into the equation as a driver and innovator of new systems. The emphasis is again the introduction of a stakeholder into the start of the process.

It might be difficult to adjust to but this is surely a positive development.

I love buffets! I can pick what I want in the combinations I want. Sometimes, those combinations only make sense to me. Today’s plate will most likely be completely different than last week’s plate. Next week’s will probably be different from this week’s. Oh some of the same things may be there, the ‘staples’, but others will probably vary. Test tools are a lot like buffets.

I like to maintain a smorgasbord of test tools. I’m constantly adding new ones to the fare along with some favourites, which I tend to pick often. They’re like the cheesecake of test tools. I have to have it. Everything else usually depends on the situation. What sounds good today? Maybe I’ll try something new? Maybe I’ll try the apple pie this time? It’s good to have choices.

The latest addition to my test tool buffet is an open source Graphical User Interface (GUI) test tool. Of course, using open source tools has its challenges. It doesn’t do everything I want it to do, but it should allow me to modify it by writing the code needed to support my unique testing situation. I’m not sure if I like it yet, but I’m keeping an open mind. I’m frustrated by some basic things that it won’t do. Automatically launching a browser has become a bit of a challenge. Whether I end up selecting and using the tool on my current project or not has yet to be determined. Much will depend on my frustration level and I’m getting pretty frustrated. One of the major frustrations with using any open source tool is the lack of documentation.

In the case of this tool, a new version has been released but the updated user documentation is not yet available. There is also a very small user community. I can typically find a local consultant to help me out but not this time. My next stop – surely someone has written a book on the subject? Nope. The jury is still out on the tool. So far, I’m not real happy with it. The quest continues.

The majority of my current test efforts are still in the backend. There is no user interface. I don’t have much of a need for a GUI tool right now, but I will soon. Our goal is 100 percent automation so I need to find something. Now as far as the backend testing goes, I’m pretty happy. Like the GUI tool, we had a variety of open source tools to choose from. This time we selected a tool that I knew rather than trying something new.

This one is much more popular so there is a large and vocal user community. There are even a couple of books available; but they’re terrible. We have been able to review posts from other users for solutions to most of our problems. As we suspected, others have had similar issues to ones we have uncovered and we have been able to work around most of them thanks to their help. But we are constantly breaking new ground.

Thankfully, I have a really supportive development team at my disposal. I merely have to mention that I need a little help and they are willing to write any code I need. Sure, I could do it myself, but who has the time? There are some blocks of code that are there just for me – the customer will never know they are there. It’s a very team-oriented approach – teamwork at its best – I absolutely love it!

Back to the buffet to find a performance testing tool, this time a tool that I had never used but I had heard of,  a popular tool with a wide user base. We tried it and liked it. It isn’t the easiest tool to use and the user documentation is somewhat sparse, but the user pool is deep. We have been able to solve most problems with a simple Internet search. Other problems were solved by putting our heads together and trying something new. Most of our solutions come from me asking “I wonder what would happen if….”

Even with known tools, we are using most of them in unique and different ways. We are always finding new and better ways to do things. Many of the approaches we take may be new ways to do things. We are more than happy to share what we are doing with the user community. But we all constantly complain about the lack of any real source of documentation.

I mentioned a few days ago that since there are no good books available, that perhaps we should write one. It was a joke but the team has taken it very seriously. They think we should really do it. It may not be a bad idea. But what to call it? I know, Smorgasbord – The Definitive Guide to the Test Tool Buffet. Maybe not. I’ll get back to you.

Dave Whalen

President and senior software entomologist

Whalen technologies

http://softwareentomologist.wordpress.com/

The idea that a practice which has been part of the IT industry for decades could now be seen as one of its most important new growth areas may raise a few eyebrows. Yet the growing importance of the software testing market has not gone unnoticed among executives and investors alike. Chris Livesey presents his ‘state of the testing nation’.

In the fast-moving information technology industry, barely a year goes by without a new ‘next big thing’; a radical new discovery which will transform the way businesses conduct their everyday operations, while saving them a small fortune in the bargain. Although the truth is rarely so clear-cut, in recent years we have seen innovative technologies such as SOA, virtualisation and cloud computing generate significant hype before their business benefits were fully realised.

When compared to these innovations, the idea that a practice which has been part of the IT industry for decades could now be seen as one of its most important new growth areas may raise a few eyebrows. Yet the growing importance of the software testing market has not gone unnoticed among executives and investors alike. The past eighteen months has seen software testing buck the downward trend in M&A activity, with vendors looking to gain a foothold in fast-growing areas of an industry which, ten to fifteen years ago, would barely have been recognised as a discipline in its own right.

While testing as a practice dates back as far as software development itself, it is only recently that it has become recognised as a distinct expertise and an area which can provide true competitive advantage to a development team and the business as a whole. This change in perception, allied to a growing need for software testing in an increasingly applications-reliant world, has made testing, and the broader process of software quality, a growing concern.

And there is no reason why it shouldn’t be. The testing process is believed to consume between one third and one half of all software development budgets, so it is crucial that its efforts are a success. In an industry in which only 32 percent of projects are successful and half of all development efforts are wasted , there is little margin for error.

Most importantly of all however, the growing importance of applications to modern society is driving the growth of testing. Increased levels of demand for applications, particularly Web 2.0 and online applications, has made their value to the business even greater. Yet it has also made the cost of their failure all the more tangible.

Every software application requires testing in one form or another, from a simple Web 2.0 widget through to a major enterprise system. With both consumers and businesses demanding smarter, faster and ever more sophisticated applications, and the cost of application failure becoming ever greater, the need for testing solutions and services which can ensure this is achieved is more pressing.

Testing is now big business. Testing software tools alone are worth an estimated $2 billion worldwide each year. These tools have, in turn, enabled the rapid growth of the testing services market, which one analyst has estimated to be worth in the region of $30 billion per year, rising to $46 billion in the next four years . This underlines why testing can legitimately be termed ‘IT’s invisible giant’.

Why test?

For a growing number of organisations today, their software is the first or only means of interaction with their customers. It is, therefore, essential that it works.

Most organisations have now realised that the effect of poor or insufficient testing can be disastrous. A 2002 study conducted by the US-based National Institute of Standards and Technology (NIST) calculated that software errors account for $59.5 billion worth of loss each year in the US alone. This, despite any improvements in software quality practices which may have taken place in the meantime, is a shocking statistic, and one which illustrates exactly why this is an issue. Indeed if the cost of software errors has increased in line with economic growth over the past eight years then we can estimate that its current cost to the US alone is somewhere in the region of $90 billion.

But the cost of defective software is not only financial. The effects of poor testing now stretch way beyond the back office, to the boardroom and even to the brand.

Toyota’s recent recall of hundreds of thousands of its hybrid vehicles (such as the successful Prius model) due to a problem with braking is one such example. What was believed to be little more than a small software glitch not only left the car giant millions of dollars out of pocket, but also risked the brand’s reputation on a global scale. The effect this may have had on the brand’s long-term standing will not be truly understood for at least another decade.

Senior executives, too, are not immune to the side-effects of software failure. In 2010, the CFO of one of Britain’s most prominent travel companies felt compelled to offer his resignation, following an accounting error (caused by faulty software) which forced the company to write down over £100 million worth of sales. This problem, like Toyota’s, would likely have been detected had rigorous software testing procedures been in place.

The potential cost of IT failure is simply too high to be ignored. In the Internet age, applications are the primary way in which individuals, both within the organisation and outside it, interact with a business. Put in these terms, rolling out an application which has not been through adequate testing procedures would be equivalent to sending out an untrained salesperson to meet customers, or instantly promoting a newly-recruited graduate to the role of CEO. For a business to function, so must its software, which is why testing is growing in stature.

Who tests?

Software testing is no longer the domain purely of large organisations running major enterprise systems. From independent software vendors through to one-man-band developers, testing is now an essential element of the IT function.

With organisations of all sizes now looking to move their business critical systems onto cloud platforms, this will create new challenges for IT professionals, requiring testing at every stage of the development process. The adoption of Agile practices, which allow testing to occur alongside development, will also serve to increase the volume of tests being conducted. And with testing taking place earlier and more frequently within the development process, there is a greater need for test automation.

With so much testing taking place, it is little surprise that a number of the world’s largest technology companies are major players in the sector. The key players in testing services, which accounts for the lion’s share of testing revenues, include some of the world’s most established IT brands, but the market is increasingly being dominated by large outsourcing providers. India is the new home of testing services, and both this and the increased strategic importance of testing have meant that the typical profile of a tester is changing.

In the past, the common misconception was often that software testers were lower-skilled than other IT professionals such as developers. However, professional qualifications in testing are now commonplace and the majority of practitioners are now degree-educated. The emergence of a new generation of highly-skilled and educated testers across the globe is proof of the industry’s growing importance. With testing now being incorporated into agile development practices, quality is becoming a critical element of any software development effort.

Test where?

It is hard to begin any conversation about the geography of software testing without first noting two of its centres: Silicon Valley, California, and Bangalore, India. Yet, when discussing today’s testing market, one would be equally justified in mentioning emerging markets such as Poland, Egypt or Brasil.

Silicon Valley, home to the majority of the world’s largest technology companies, is the spiritual home of software, and as such has also nurtured the testing methods and tools which software required as it rose to the prominence it today enjoys. Bangalore, by contrast, is the IT outsourcing capital of the world – a five million-strong technology hub within a nation which currently produces over 400,000 technology graduates each year.

While estimates of India’s overall share of the testing market vary greatly, most analysts would now agree that it is now higher than that of the US. And the traditional balance of power in this relationship may also be shifting.

Since the turn of the century, testing vendors in developed nations have traditionally focused on strategic, value-added services and IP-rich testing tools, while labour-intensive manual testing has been outsourced to areas such as Bangalore. Now, however, with a number of years of experience and an increasingly skilled workforce behind them, India’s testing suppliers are beginning to climb the value chain. The subcontinent’s outsourcing giants are increasingly capable of offering sophisticated testing services which rival those of their American and European counterparts.

However, an increasingly diverse range of competitors is emerging, challenging the established leaders at both end of the value chain. Perhaps inspired by the track record of India, China has made concerted efforts to establish itself as a major player within the technology industry over the past decade. A recent report estimated that the Chinese software industry already accounts for revenues in excess of $50 billion each year, with testing among the fastest-growing areas of this. As Indian vendors expand to challenge European and American rivals at the higher end of the value chain (and as average wages for professionals increase accordingly), China (and, to a lesser extent, Malaysia) may well emerge as a new hub for labour-intensive, outsourced testing services.

In addition, emerging nations across Central and Eastern Europe, North Africa and Latin America are successfully taking advantage of the trend towards ‘nearsourcing’ among businesses in Western Europe and North America. The Czech Republic and Poland both now attract BPO investment in excess of £500 million each year, while the likes of Mexico and Venezuela are becoming increasingly attractive outsourcing destinations for companies in the USA. Egypt, the Philippines, Tunisia and Bulgaria are also among a group of 14 ‘non-BRIC’ (Brasil, Russia, India, China) nations identified by the London School of Economics as potential hotbeds for BPO growth over the coming decades, and as a result are all areas where we would expect investment in testing services to increase.

Testing times ahead?

If there is one certainty to be gleaned from current trends in software, it is that testing and other areas of quality assurance will continue to grow in importance for the foreseeable future. As the potential impact of software failure increases, so too do the resources available to prevent such problems from occurring. This, in turn, is increasing levels of professionalism within the industry, making testing a more strategically-important and lucrative practice.

However, to suggest that testing will simply continue its current trajectory without any major changes to the landscape of the market would be foolhardy. The market as it now stands is virtually unrecognisable from that of ten or fifteen years ago, and it is reasonable to assume that the rate of progress over the coming decade will be even more rapid.

Here are what we believe will be five major trends in software testing between now and 2020:

1. Testing’s rightful place in the cloud

Cloud computing will be the single greatest influence on IT practices in the years ahead, and testing will, like every other facet of technology, be affected.

In addition to the obvious benefit of flexible pricing, the cloud model has a great deal to offer to the testing industry due to what is likely to be an exponential increase in demand for load testing. Cloud is a compelling option for companies conducting load testing, due to its ability to conduct short bursts of tests without requiring significant outlay on hardware or maintenance.

In a world where more and more organisations rely on applications (whether web-based or private), businesses require reassurance that these systems are durable enough to handle thousands of individuals using the application simultaneously and from multiple points of access. Companies who can offer the products and services required to do this look well-placed to prosper from the cloud age.

Testing-as-a-service is estimated to grow by over 33 percent each year between now and 2013, meaning that the market for cloud-based testing tools will reach over $700 million by 2013. This represents a huge opportunity for outsourcers, ISVs and software vendors alike, and shows just why many analysts have described cloud computing as the biggest step change in IT since the adoption of the Internet itself.

2. Testing skills shortage a likelihood

A 2010 survey found that almost three quarters of testing professionals in the UK felt that there was a skills shortage within the industry.

Gartner has estimated that, within non-software companies, the highest ratios of testers to developers is around 1:3, meaning that many companies may have a ratio of four or five to one, or even more. When one considers that between a third and a half of the total cost of application development is accounted for by the testing process, this seems ominously low. Such discrepancies between demand and supply show why the testing stage often becomes a bottleneck in the software development process.

While automation tools are capable of reducing much of the tester’s workload, it is clear that software testing, as a growing area of the IT industry, will require more skilled professionals in the years ahead. While the growing status of the industry will no doubt help in attracting new graduates and school leavers into the profession, as with all skills shortages, this will not be solved overnight. Rather, it will require the co-operation of government, business and academia to identify the areas in which shortfalls are the highest and to then tailor curricula to meet these needs. While this shortfall is being addressed, responsibility for ensuring software quality will fall upon the shoulders of every stakeholder involved in the project, from analysts through to developers.

Nevertheless, working to improve the ratio of testers to developers, and identifying and tackling skills shortages in testing will be deciding factors in whether the sector, on the one hand, flourishes over the coming decade, or on the other, becomes stifled by a lack of available talent and an ongoing reputation as the major bottleneck in the development cycle.

3. An automated and continuous approach to testing

One trend which will undoubtedly shape the testing tools market over the coming years is that of test automation. While test automation tools have been available for a number of years, it is only comparatively recently that businesses have truly begun to appreciate the value which they can add to the development process.

The increased automation of testing supports a more ‘continuous’ approach to software quality. Previous modes of assuring quality tended to focus on the last mile of the development process, where testing would only commence once development was complete, leading to frequent delays and re-work. By automating testing processes, quality can be emphasised at the start of development and problems can be addressed before they become too difficult or costly to remove.

Automation tools eliminate much of the laborious nature of testing and also remove exposure to human error within the process. This has led some to suggest that increased adoption of such tools could eventually replace the majority of manual tests. This in turn would have a huge effect on the testing services market, much of which relies on an ongoing requirement for labour-intensive, manual tasks. However, in a survey conducted by Micro Focus, manual testing still ranked as a far higher priority among testers than automation, showing that there is still a long way to go before automation becomes the norm.

In reality, there will always be a need for both manual and automated testing, and that will be as true in 2020 as it is today. Agile development practices require higher levels of automation, meaning increased adoption of Agile will lead to a growth in demand for automation tools. However, increased demand for testing services as a whole will more than compensate for any reduction in manual testing required as a result of increased adoption of automation techniques.

4. Increased agility

Agile methods are becoming increasingly important in software development. Companies value their flexible and extremely effective procedures, which work even for large projects, enabling products to be completed early and subsequent adaptations to be made as well. Agile testing plays an important part in this: testing at an early stage, and in parallel with software development, ensures that the quality of the software satisfies requirements more closely.

Relevant test procedures can now be carried out earlier in the course of the project, meaning problems can be identified in good time and rectified accordingly. Combining prompt testing with automation will also lead to greater efficiency: the inaccuracies of manual processes can be eliminated and tests can be repeated.

The ability to test earlier in the development process also means that more testing can now be ‘requirements driven’. Aligning the testing and requirements processes more closely is yet another way of ensuring that software quality is built into the development process from the start, rather than being undertaken only once an application nears completion.

As Agile continues to grow as a practice, the ability to test throughout the development process becomes ever more essential, and testing tools will need to change in order to meet this demand.

5. Applications economy

While it is by no means a new trend, one process which will continue rapidly over the coming decade is the growing importance of applications to the businesses they serve.

Even more so than today, applications, be they web-based widgets or back office batch processing systems, will be the lifeblood of the business. As Internet adoption becomes more prevalent throughout the developing world, greater strain will be placed on online applications, and with this increased demand will come greater business value and risk of failure. Further, as applications become more complex they consume more resources and can lead to increased loads. They also become more difficult to test due to the complexity of simulating interactions like those via Web 2.0 applications.

The financial and reputational cost of application failures will continue to skyrocket. The need to stay available and functional will drive a greater understanding of peak demand times and software quality processes, meaning that website outages or system failures, while not altogether becoming a thing of the past, should not occur as frequently as today. Organisations simply must test to ensure that applications perform, even under massive peak loads.

The invisible giant

With much of the IT industry still struggling to overcome the economic malaise of recent years, testing is one of the few areas in enterprise technology which is bucking the trend and showing strong growth prospects. With a market size which will soon exceed $50 billion, and prevailing trends showing that demand in the sector shows no signs of slowing, the potential rewards for companies and individuals which can succeed in this space are huge. While it may never make front-page headlines or appear at the top of many board agendas, software testing is now, and will continue to be, IT’s invisible giant.

www.microfocus.com