What does a quote from a 1940s movie starring Charlie Chaplin have to do with testing and well performing testing centres of excellence (TCE)? More than one would think according to Archie Roboostoff. In fact, some would say it forms the core of some of the best TCE implementations around the world.

A testing centre of excellence (TCE) provides a framework and methodology for bringing specialists and toolsets together that better leverage knowledge, technology and resources. A basic TCE structure brings together test managers, architects, automation engineers, subject matter experts, quality assurance and testers. While TCEs vary across verticals and companies, the basic premise is the same, create an organisation that delivers the highest quality software in the most efficient manner.

Given that mainstream, commercial test automation tools have been around since the late 1980s, organisations have had plenty of time to fine tune and adjust a wide variety of TCEs. From outsourced offshore to localised TCEs, there have been many derivatives that have both failed and succeeded. Every TCE methodology puts a strong emphasis on testing (ie, Feature-Driven Development, Test-Driven Development, Behaviour Driven Development, Insert you methodology here-Driven Development, etc) yet, most development organisations still cannot seem to get it right.

Product deliveries across the globe are still delayed and those that aren’t sometimes have quality issues. When a quality issue is discovered after a product has gone to market, the cost to recover is exponential. Having to pull development resources off future products to fix existing quality issues can severely impact not only the ongoing cost of the development team but will adversely affect the revenue generating capabilities of that organisation.

Now, back to that Charlie Chaplin quote and what it means to productive TCEs. The bottom line, it’s more important to focus on collaboration than it is to focus on the technology within a TCE. Collaboration leads to two benefits within a TCE, predictability and productivity. The impetus of TCEs was to foster collaboration yet as market and product cycles shrink, silos within TCEs are starting to emerge. It is human nature after all, that when under stress, most people shut down to try to recover.

In the past, this lack of collaboration remained hidden given the long delivery cycles of most software applications. Those TCEs that do not collaborate well will not have the luxury of hidden recovery given that the voracious technical appetite for social media driven applications, mobile devices, on demand content, wireless services and connected appliances is rapidly shrinking product lifecycles. Now, more than ever, TCEs are being pushed to the limits to get products out to market that work, perform, and function.

Building a collaborative TCE step 1: Requirements

Evolving a TCE into a collaborative ‘well-oiled machine’ that brings instant productivity to the people within the TCE and constant predictability to the management layers above the TCE is actually easier to accomplish than most organisations make it out to be. The tip of the spear, so to speak, is getting everyone on the same page with requirements. It is notoriously difficult for business users to communicate needs to development teams. As a matter of fact, up to 70 percent of all production defects stem from this stage when requirements are poorly defined.

Requirements come from numerous stakeholders that may be globally distributed and in a variety of formats. This is further complicated by the fact that users often don’t know what they want until they actually see it and the fact that the only constant with requirements is change. This leads to wasteful cycles where development teams try to implement what they understood and the TCE is trying to verify quality on misaligned goals. This results in missing capabilities, unnecessary features, and – down the road – a lot of rework. And that rework quickly adds up, especially when issues are discovered late in the development process.

Visual business scenarios are an effective means of communicating business process flows and concepts but are usually only propagated to the various business and development stakeholders. By sharing process visualisations coupled with simulations with all functional groups within the TCE, this helps reduce ambiguity and presents an instantaneous way to recognise sequencing issues, broken decision points, missing process steps, and opportunities for innovation. The overall benefit to the TCE is that the TCE now has the ability to create test cases while products are still in the planning phase.

Effort put into defining the paths that users can take within an application is a wise investment because this information is used to create test cases for each path. With the ability to calculate the magnitude of requirements risk and complexity, QA staff can identify high risk areas in an application and in doing so help set testing priorities. This in turn ensures that effort is effectively spent and not wasted on over- or under-engineered functionality. Generating tests cases directly from requirements also means that QA staff becomes active participants in the development process from inception and not as an afterthought. By doing this, an environment where test planning and effort estimation can become much more realistic. And thus, they become more predictable and productive because they have been validated early in the process.

By bringing the TCE into the loop during the requirements phase, and linking that information with testing assets, any change to the requirement and any affected policy changes are instantly communicated across the entire SDLC and TCE. This aspect of technology integration is a prime example of focusing on collaboration rather than pure technology that will ultimately drive efficient testing practices and an overall better product.

Building a collaborative TCE step 2:  Collaborative testing framework

Building a successful test automation practice within a TCE is a challenging task for many development organisations. Mature development organisations realise that simple UI test automation techniques like record/playback do not provide the collaboration required for the speed of today’s product delivery cycles.

There are many frameworks for test automation that have evolved over the years. Keyword-driven Testing (KDT) is a widely used test automation technique that many mature TCEs rely on to overcome the disadvantages of simple record/playback test automation. There are, however, some major disadvantages with managing the complexity inherent in KDT. Many applications require that thousands of automation keywords be developed to make use of KDT. Navigating and constructing test cases based on these keywords can be cumbersome and unpractical.

Acceptance Testing Frameworks (ATF) use a similar approach to structuring test cases (acceptance tests) using keywords that are implemented using coded fixtures. These frameworks do not support users in navigating all available actions (keywords). As with KDT frameworks, ATFs do not support collaboration since the structured fixtures (the implementation of the actions) cannot be easily reused and maintained.

Keeping in mind the end goal for a productive and predictable TCE organisation is to bring collaboration to the forefront; advanced testing frameworks should be considered. Again, focusing on people vs. technology, frameworks should be considered that allows testers to create automated tests as early (and change as often) as possible. The ideal state is to allow a test automation engineer to create test cases based on a product that still may be on the whiteboard.

State-driven Testing (SDT) is a framework that drives collaboration and addresses the maintenance and complexity issues of KDT by providing a UI-state transition model. By defining the given state at any moment of the user interface, the test automation engineer instantly knows what steps the application can take from there. SDT reduces the test case possibilities from thousands of allowable actions down to a manageable list of between 10 and 20 allowable actions. SDT uses a domain specific language (DSL) to define the test automation framework which provides a highly maintainable and simple productive approach to structuring a test framework.

SDT is the next evolution of collaborative frameworks with an approach that provides the ease-of-use for creating tests, enabling non-technical subject matter experts to write tests while eliminating the complexities and maintenance issues inherent in previous frameworks that created test silos.

Choosing a highly flexible framework that brings people together from the inception of an application all the way through to the testing endpoint overcomes many issues that are present in today’s frameworks. There are numerous reasons why many TCEs still struggle with test automation projects:

• Lack of required skills (test automation requires technical staff);

• Maintenance effort (test automation is sensitive to application change and volatility);

• Inconsistency of execution (test automation tools can deliver inconsistent results due to unresolved synchronisation points between test scripts and AUT);

• Lack of relevance/test documentation (without associating tests with specific requirements and/or code and without clearly documented test cases, it is not clear what actually has been tested);

• Lack of situational awareness (if a requirement changed, who informs the test automation engineer?).

While no framework is going to be the ‘holy grail’ of test automation, SDT does provide answers to many test automation challenges that remain within TCEs. SDT is an approach that suits organisations that want to deliver productivity through collaboration to their test automation practices without the pain and effort and limitations of relying on less capable commercial and open source tools.

Building a collaborative TCE step 3: Increase agility

As product lifecycles continue to shrink, software development practices are evolving to keep pace. It has become increasingly important that TCEs test and modify software from the perspective of the group that will ultimately decide the fate of an organisation’s business – their customers. With the customer perspective in mind, Agile methods are becoming a more common development approach. Companies value the flexibility, transparency and speed of these Agile development procedures, which are getting adopted more and more even for large development and testing projects.

Agile development enables organisations to deliver products early while making any necessary changes for future roll-outs along the way. Agile testing plays a critical role in this. Collaborative testing at an early stage – and in parallel with always visible requirements – ensures that the quality of the software satisfies all necessary requirements regardless of change. Most importantly, it enables the TCE to find potentially costly errors earlier in the development process, saving time, money and resources.

As stated before, quality assurance teams have typically lacked visibility into what is being tested, the actual test results, the business requirements driving testing and the extent to which they are reflected in the testing requirements. The movement toward Agile adoption promotes environments where organisations can deliver and benefit from real-time visibility into testing activities and the quality status of a project. This is the driving reason to start fostering collaboration early so that old habits can be eliminated. In order to deliver quality software, TCE using Agile testing methods collaborate more effectively with the business to improve project delivery outcomes.

As organisations continue to turn to Agile methods to speed up time-to-market, reduce risk, and improve the overall quality of their software projects, we have pulled together some practical tips for TCE organisations on how they can optimise their software testing procedures:

Test early and often: Not just some, but all relevant test procedures for a given project must be carried out comprehensively as early as possible in the lifecycle of a project. This means that unit tests, functional tests and load tests must be built into a single visible portal so that management, developers, and TCE members are on the same page.

Integrate unit testing: Unit testing is a critical step in the development process that ensures code being used will actually do what it is designed to do and will behave in the way it is intended.

Load test daily: As more software is designed for massively scalable deployments, load testing has emerged as an important step in the development process. The best product in the world will be useless if it crashes under the weight of too many users.

Know the start points: Test starting points must be clearly defined throughout the testing process, so it is important to take starting points into consideration even in the development stage.

Be flexible: One of the benefits of Agile testing is the flexibility it delivers to the development teams. Software development projects are not static – priorities, goals and requirements can be fluid and often change throughout the development lifecycle. Having an infrastructure goal that fosters collaboration can ensure that the team is always testing against the most up to date set of requirements and living up to the flexible environment that Agile development was designed to deliver.

The trend in technology is to bring people together. The first iPhone was laughed at by resident technocrats for not having as much technology as its competitors at the time. By focusing on how people work and bringing them together, Apple has continued to keep the iPhone the market leader in smartphones. Testing practices and TCEs have not followed suit. Rather than focus on how people work together, TCEs have compensated by throwing more technology and tools at the problem. By taking a step back and understanding that open and flexible communication through the tools being used will make sure everyone is on the same page. This collaboration delivers productivity through shortened development and testing cycles while at the same time bringing predictability into the environment. The TCE will be able to see and track how subtle changes from the business that once created uncertainties, are now understood, corrected, and learned from.

Borland solutions are different than anything else in the market. Where competitors might focus on throwing complicated technology at a problem, Borland focuses on fostering collaboration between the people developing the technology. Borland’s first priority is providing customers with tools that turn whiteboard ideas into customer facing products faster than anyone in the industry. From cost savings to faster product delivery cycles, organisations using our tools will continue to build better software faster, regardless of framework, platform or system.

Archie Roboostoff

Director of product management, Borland Solutions

Micro Focus

http://www.microfocus.com/

www.prweb.com//releases/2011/10/prweb8888216.htm

“By combining Spirent’s industry leading real world performance testing capabilities with iTrinegy’s accurate network emulation,  enterprises can assess the readiness of any new application, system upgrade or configuration change without putting the production network at risk,” said David Hill, vice president of EMEA at Spirent.  “This is a win-win partnership for our enterprises and cloud service provider customers.”

The Interxion Testlab deployment is a collaboration between Interxion, Flexiant, Boston Limited and GTT and acts as an online proof-of-concept (PoC) environment for cloud service providers. It can be used for testing scenarios before deploying a cloud infrastructure with the proof-of-concept being run on a reference architecture. This Testlab offers a framework that supports multiple hypervisors and allows service providers to build both private and public clouds.

The new test environment is located in Interxion’s City of London data centre and will deploy Flexiant’s Extility platform solution, have network connectivity via GTT and be housed on Supermicro hardware provisioned by Boston Limited.

“The rapid adoption of cloud computing has seen cloud-based services become a central part of business strategy,” comments Greg McCulloch, UK managing director, Interxion. “The launch of this UK Cloud Testlab deployment further reflects our commitment to the cloud and offers both customers and prospects the opportunity to test and develop cloud services at high speed and with best-in-class performance guarantees.”

“Information technology is becoming more and more sophisticated, complicated and as a result, more vulnerable,” says Anton von Troyer, vice president DACH from Codenomicon Deutchland. “We want to help training the future information security professionals to meet the challenge they are up against.”

Codenomicon’s Defensics test tool relies on fuzz testing, a technique where invalid or unexpected input is fed to the system under test to reveal potentially exploitable vulnerabilities. The technique is particularly effective in finding previously unknown, so called zero-day vulnerabilities. Unknown vulnerabilities are a challenge to product security, since there are no patches available, no way to protect the system against attacks targeted to unknown vulnerabilities.

“It is a fertile cooperation for both sides”, says Prof. Sachar Paulus from FH Brandenburg. “Beside the use of the software in teaching, possible innovations in the context of product use are examined together.”

Calyx Software, Gyrosoft, Trinity Computer Services, and divisions of MentecPlus, Touchstone Group, Sys-Care, and Avant-Garde have been integrated through acquisition into a single entity called m-hance. The announcement follows a series of recent acquisitions that has seen m-hance grow to 150 staff serving almost 1,000 direct customers across the UK and Ireland and a further 1000 through channel partners. The company operates from offices in Manchester, London, Loughborough, Dublin and Glasgow.

Mark Thompson, CEO of m-hance, comments, “The creation of m-hance allows us to go to our customers with a single, clear identity and message. The company benefits from the stability, long trading history and deep client relationships of each of the component entities. We’ve combined the strengths of seven respected specialists into one financially strong leader with an experienced management team backed-up by a solid balance sheet containing no external debt”.

The new structure allows the company to further develop its product set along vertical market specific lines. For example, in the distribution and manufacturing sectors it had been providing a best of breed industry solution including finance, CRM and other business critical functionality to more than 750 organisations for over 25 years and can now expand this model further into not-for-profit, education, financial services and hospitality and leisure, where it has a strong customer footprint.

So far Coverity’s Static Analysis testing tool has allegedly found over 40,000 defects in CERN’s software, which could have severely compromised the project had they not been detected. This case aptly showcases the importance of rigorous testing in a scientific experiment of the magnitude of the LHC project.

Director of software testing company BugFinders, Martin Mudge commented on the safety precautions involved in software testing: “Software testing is vital when the project involves safety critical systems. Sure, it’s bad when you come across a glitch in one of your iPhone apps, but if a computerised plane system gives the wrong information, it could end with serious loss of life. I wouldn’t dream what the consequences would be if a critical bug went unnoticed on a project involving nuclear energy.”

The fervent media controversy surrounding the Large Hadron Collider may have been overhyped, but there have been however past cases where a lack of testing has proven to be costly, sometimes even fatal.

The new Android testing capabilities complement the popular FoneMonkey for iOS (iPhone and iPad) automated testing tool and extends Gorilla Logic’s comprehensive open source automated testing solution set by bringing Android developers and QA testers the same recording/playback and test script generation already enjoyed by iOS developers. It benefits developers by providing rapid creation of functional testing scripts; allows testers without in-depth Android Software Developer Kit (SDK) knowledge to easily and quickly create testing scripts; and, offers increased application quality and faster time-to-market.

“FoneMonkey for Android addresses a major gap in the Android development tool chain,” said Stu Stern, president and CEO of Gorilla Logic and co-creator of FoneMonkey. “The FoneMonkey family of tools is the only record/playback functional testing tool for iOS and Android. With the addition of FoneMonkey for Android, Gorilla Logic has established its position as the pre-eminent provider of next generation functional testing tools for the mobile market.”

According to Gorilla, FoneMonkey is the only test tool for iOS and now Android that records all actions with the iPhone, iPad and Android phone while in use and plays them back as a test script at any time. It enables the interactive creation, editing and playback of automation scripts that exercise an application’s user interface. Using FoneMonkey, developers and quality assurance team members can create suites of tests that automate and perform user operation sequences, and then verify results. FoneMonkey supports development as well as QA testing, and tests can easily be incorporated into continuous integration environments.

FoneMonkey for Android is available now. Because it is an open source tool, it is free to download at www.gorillalogic.com/fonemonkey.

A survey asking 78 London-based influential software testers and QA professionals for their views on the key drivers and sources of innovation and skills that will influence their industry over the next three years reveals that 83 percent back the continued popularity of automated testing over manual testing. However, respondents were considerably less optimistic about relationships between management and IT. When asked what won’t change over the next three years, more than two-thirds of testing professionals (69 percent) believed that software testing will still not be fully understood or supported by managers in that time-frame. In addition, 45 percent were resigned to the fact budgets for software testing and QA would not increase over the same period.

When asked what would define testing innovation over the next three years, almost two thirds of survey respondents (64 percent) pointed at demand for mobile applications. The second most popular response to the question was increased adoption of Agile testing procedures and principles (39 percent), while testing in virtualised environments was also identified as a significant driver of innovation for more than a third of participants (35 percent).

Despite 40% of respondents agreeing that testing in the Cloud to reduce costs and improve agility would become ubiquitous, it appears that few organisations are actually ready to fully commit to the technology. Just 17 percent of test professionals said that they expected more than half of applications in their business to be migrated to the cloud within three years, while 47 percent were adamant that this would not happen.

Testers were also asked what technical skill sets they believed would be in greatest demand for software testers and QA professionals over the next three years. In accordance with other responses, automation was voted the most likely to be in demand, followed by Agile, mobile and then security testing. When asked what non-technical skill sets might be in greatest demand, communications skills were regarded as the most sought-after, followed by business experience and collaborative working, with austerity project planning thought of as a less important quality.

Apart from our testing top 20, another thing you can depend upon is change. When I first interviewed Axel Naumann for the CERN feature on page 16 of this issue, details had just been released, literally in the same week, about how the astonishing results of the OPERA experiment carried out by CERN in which 15,000 beams of neutrinos were fired from Geneva to the Gran Sasso Laboratory in Italy seemed to show that the neutrinos had travelled faster than the speed of light.

CERN researchers advocated prudence in the face of the early results, although at the time they believed their observations – in which the neutrinos made the 730 km journey 60 nanoseconds, ie, faster than light – were accurate.

Since late September when the initial details were publicised, another OPERA experiment was run, further data were processed and the same results were observed, seemingly confirming the initial flabbergasting results that the neutrinos were indeed travelling faster than light – and thus undermining the whole idea of cause and effect by theoretically allowing scientists to send data back in time. The second set of results were publicised just as I was working on the feature.

In the latest response to the research however, published just yesterday as I write this, a team of scientists led by the Gran Sasso National Laboratory claims to have refuted the OPERA faster-than-light neutrino result. The latest experiment used much shorter pulses to try and remove a potential systematic error from the set-up. To make sure that they could prove the time correlation between the received neutrinos and the originating event, the latest test used pulses three nanoseconds long, with more than 500 nanoseconds between pulses. This experiment claims that the neutrinos arrived 57.8 ns (±7.8ns) too early, seemingly corroborating the original OPERA data.

The Gran Sasso group however says that in attaining faster than light velocities the neutrinos should lose energy by producing photons and electron/anti-electron pairs. This, they assert, has not been observed; their spectrum seems to demonstrate that they didn’t lose that energy, therefore they can’t be passing the speed of light…

If this is the case, then perhaps the guys at CERN need to run the data processing software through a bit more static analysis. Whatever the facts, it’s clear that life on the cutting edge of theoretical physics is never simple – and by the time you read this, there will probably be a whole new set of results to confound us.

Until next time…

Matt Bailey