As testers, we were fully occupied with dates a few years back at the turn of the Millennium, hunting down the Y2K bug. And seeing as technology now allows us to have all this fun with our computerised gadgets and gizmos, we must have done a good job, notwithstanding the odd glitch here and there.

The job of a tester is not about fun though of course, it is a very serious business. So if I can detract from my remit to ‘keep it light’ just for a sentence or two, we carry the weight of responsibility, of hope that we left no bugs un-squished. We were the last to see the system before go-live, and we will be the first in the line of defence on go-dead. No buts.

So, back to the fun; have you heard the one about the man who was lost in some world without mobile phone masts, with only his smartphone for company? How did he survive? Well, he ate the dates from the calendar, and then on Sundays – you know the ending, as does your five year old… 

And in the last year, we’ve had many tasty dates to behold. We started last year with 010110. We started this one with 010111. Last October we had 011010, then 101010; all these ones and zeros; yes or nos; true or falses.  As a tester, this is easy to check. The happiest times at work for me were the times when I was just a tester – just as Dave Whalen has said in this very column in past issues*. The times as a tester when I understood exactly what the system was supposed to do. I had a properly written spec, a fully working test environment, test data under my control (it was my test after all), and a test I had spent days designing. I could turn the specification this way and that, figuring out how to wake up the sleepy spider so I could catch him mid-crawl, and then I would catch him, Bliss! 

All this talk nowadays of budgets, timescales, writing this strategy and that plan are all necessary evils – to tell everyone that we know what we’re doing – but nothing is quite as satisfying for me as rolling my sleeves up and doing the job, or showing someone else how to do it. One poignant moment in my career springs to mind; I had completed a comprehensive one-to-one training programme that lasted three weeks. I did as I was told, and got through the tests. The next morning I was armed with all that knowledge and ready to prove my worth. The clue to my impending downfall was the smirk on my boss’s face, as I would recall later that day.

I started to run the tests, and then the equipment started bleeping at me, I had done something wrong, but what? In training, I was taught how to pass, not how to fail! I was unaware that they were running a book upstairs on how long it would be before the phone started ringing. Apparently they did this with every recruit – you have got to have a bit of fun after all, and fun they had! And what fun it was for me afterwards, when I was the trainer, well, after the panic attack, with my boss teaching me in the morning and me teaching my charge in the afternoon – nothing like flying (or sinking) by the seat of your pants.

So back to significant dates. On 3.14 (14th March) we now apparently celebrate Pi day. I confess, I don’t much like this one.  Pi= 3.142 etc etc and the etceteras can go on and on into trillions of figures, into infinity. I can’t imagine there’s much fun to be had in finding out what the digits should be. In mathematics there is undoubtedly great beauty in the absolute proof, but the journey to get there is an arduous one and not much fun for us, not on Tester HappyNess Day. Join our petition now to have this declared a national holiday – we’re of age and they should know it.

www.pintaed.com

* Dave Whalen returns next issue.

Training teams often have a role to play in supporting testing and providing feedback on the end user experience. Paul Deed, director of SkillSet, discusses the part that training teams play in the analysis, design, development and delivery of software implementations.

Most people will be familiar with the famous quote by Benjamin Franklin: “By failing to prepare, you are preparing to fail”, however, this becomes particularly pertinent for anyone who has ever found themselves in a teaching role. In preparing to deliver training, professional trainers need to pre-empt any potential glitches or pitfalls likely to befall the users before the users encounter them in their training session. They are also uniquely positioned to provide feedback to project teams following training sessions with the users. In many cases, each training session can be viewed as a mini-UAT session to generate ideas for improvement or uncover unknown defects, so it is important that this information is tapped into.

Formal testing follows the test strategy plans and scripts developed by the project.  Training design and development often uses the test or a dedicated training environment.  As trainers work within this environment, they are working mimicking potential end-users, not following a test script or plan but building learning content based on business scenarios.  Inevitably, this will uncover anything from usability issues to defects within the system; how many project testing teams proactively engage the training team rather than relying on informal feedback.

Several training development tools have the ability to generate test scripts and even integrate with tools such as HP’s Quality Centre. Putting these tools in the hands of the project team can save testing time in creating scripts as well as giving the training team a jump-start to their work.

Training teams will seek to understand how a software package supports the business and its processes, creating materials that convey this in an instructionally sound manner to end users. In the delivery phase of a training project, the training team is in constant contact with most, if not all, end users of the system.  This provides them with a unique insight on the user impressions and concerns ahead of go-live, and they can provide hugely valuable input and a user perspective when working directly with the project team.

www.SkillSet.co.uk

I recently decided to take a break from managerial test roles and return to life as a simple tester. I assumed the stress would be less, I’d attend fewer meetings, I could just concentrate on my test assignments and go home at the end of the day and leave work at work.

Yeah right! The truth was – I really missed getting my hands dirty and doing the actual testing. My role as a test manager was much more ‘conceptual’. Providing test estimates, writing test plans and strategies, giving my ‘expert opinion’. And attending seemingly endless meetings! When I wasn’t in a meeting, I was in an office all by myself, rarely interacting with anyone. It killed me. I absolutely hated it! I know – I’ll be a consultant!

As a consultant it’s easy to shift gears a little. All I had to do was modify my resume, remove all the managerial stuff and replace it with hands-on testing stuff, and cast it into the water and see what bites. I got a lot of bites. But in spite of my best efforts, I was typically found to be over-qualified. A bit more tweaking was necessary. So I reworked my resume again and took a look at some of my responses to interview questions and revised them. It worked – I’ve been able to land a couple of pure testing roles. This is usually where the “and he lived happily ever after” line comes in. Not so much.

I found it really hard to take off my test manager’s hat. I was pretty successful as a test manager. I’ve written about test management, spoken about test management and taught test management, I know everything there is to possibly know about test management. I thought to myself – these people don’t know how lucky they are to get me. I could assume my ‘tester’ role and at the same time help them improve their testing skills, test planning, defect management etc. It didn’t quite work out that way.

I learned a few things about myself though. First, that I can be a little over-bearing. OK – a lot. I’ve had to learn to just ‘shut up and test’. After all, that’s what they hired me to do. Not improve their test management. But it’s hard. After 20 years of doing this I’ve seen what works and what doesn’t. Whenever I saw something about to be implemented that had failed for me in the past I spent more time raising the ‘this ain’t gonna work’ alarm rather than helping the team move forward. I admit it; I was somewhat of a whiner. That’s OK though – I’d just save up a big ‘I told you so’ to use at the first opportunity. I rarely got that far.

Second – my opinions were rarely wanted. Just test! But you don’t understand – I am after all… me! People pay money to listen to me (sort of) and I’m giving it to you for free. What’s wrong with you people?

So what I’ve learned is this. Just because they don’t do things my way, it doesn’t mean they are doing it wrong. They’re just different. Give ‘em a chance. If I don’t like something or disagree – discuss it as a team member rather that crashing down on them with my ‘vast testing knowledge and experience’. People appreciate listening to ideas rather than having them shoved down their throats by ‘the consultant’. Consult – don’t dictate! Tell them your experience; offer suggestions; be there to help. I now tell test managers that I may disagree with something (I probably will). Take it for what it is – an idea. It’s me. It’s what I do. You don’t have to agree with me, but at least give me the opportunity to stick my two cents-worth in. Let me share my ideas and opinions – in private! You don’t have to agree with me – just listen. At the end of the day, we may disagree behind closed doors, but once they make a decision, I’m their biggest cheerleader.

Is it easy? No. I still struggle with it everyday. Much of it is just my personality. Most of my previous consulting gigs were aimed at process improvement. Analyse, and make recommendations to improve. That’s not my role anymore. I’ll be honest, maybe it’s time for me to return to a test management role or a problem solving consultant role. But until then – with the exception of this article – I’ll just “shut up and test.”

http://softwareentomologist.wordpress.com/

One requirement, based on US federal regulations, was that a customer could make no more than five online transfers between accounts in a month. So late one Friday afternoon, which coincidentally was also the last day of the month, I made five transfers from my test user’s checking account to her savings account. Since I wasn’t playing with real money, I made five transfers of $100. I attempted a sixth transfer and as expected, received an appropriate error message. Since Monday was a new month I figured I would try another transfer first thing Monday morning. If the transfer was allowed, the test would pass and I could move on.

On Monday I launched the application successfully and was presented with the log-on screen. I entered my test user’s log-on but received an error message that my username and/or password were invalid. No worries, probably just a typo. I tried again – same error. It was Monday and the coffee hadn’t kicked in yet so I figured I had probably typed something wrong. Third time, I made sure every character was absolutely correct. I clicked the log-on button and once again it failed. Only this time, with three consecutive invalid attempts, the account was locked.

 I sauntered, head hung low, into my boss’s office to relate the embarrassing tale of being locked out of my test account. What he told me knocked me to the floor. Apparently, the woman who owned the account noticed some strange activity and changed her password. What!!! We were testing with live data and live accounts! Seriously! This can’t be true!

Sadly, it was true. All of the testers had been testing using accounts that they had created. It was due to sheer laziness on my part that I had decided to use an existing account. I assumed we had a copy of the production database to use for testing.

I quickly raised an issue with the test project manager and the CIO, anyone who would listen, that this was completely unacceptable. They didn’t get it. They told me to just be sure to reverse any changes I made during testing. I ended the contract on the spot!

There are a number of key concepts related to any successful test effort. Repeatability and consistency are important, they are actually somewhat related. Tests should be ‘repeatable’ in that we always run the same test, under the same conditions, with the same data, each and every test cycle or when validating a defect fix. Repeatability drives Consistency. When we run the same tests, under the same conditions, with the same data, we should always get the same result. Bottom line, we have to do the exact same steps, with the exact same data, and the exact same actions, each and every time we run a test. If not the test becomes unreliable. But if the tests alter the data, or the data is constantly changing, how can we do that?

We lose repeatability and consistency if the data changes with every new test cycle. Sadly, this is way too common. A number of organisations that I’ve worked with will update the test database with a current copy of the production database at the beginning of each new test cycle. The result is that tests are typically vaguely written because the data is unknown or pieces may be missed if data doesn’t exist to support the test.

We really should define and build test data as part of the test authoring process. With every new test cycle, we rebuild the data with a clean, known set of test data. Use the same data each and every time.

How do we create test data? We have a couple of options depending on the time you have: 1. Take a copy of production, modify it to fit your tests, and then save it! 2. Create it from scratch.

Option 2 is rarely a viable option; however option 1 usually works well. As a compromise, you may be able to write test data on top of a ‘copy’ of production data. In other words, run a script that adds test data to a copy of production data. It’s going to take time to create the initial test data set and as time goes by, it may need to be modified. But it saves a lot of time in the long run and increases the reliability and consistency of your test results!

Can we always do this? I’d hate to say always, but I’m willing to venture about 90 percent of the time we can. And, although this should be a blinding glimpse of the obvious, never, under any circumstances, test using live data!

http://softwareentomologist.wordpress.com/

Early and frequent testing plays a key role in Agile software development. Joachim Herschmann, product director for Test Automation at Micro Focus, offers tips on how software developers and testers can use Agile principles to make their test procedures more efficient.

Agile methods are becoming increasingly important in software development and Agile has become one of the key concepts in quality engineering both within and outside the software testing arena. Companies value their flexible and extremely effective procedures, which work even for large projects, enabling products to be completed early and subsequent adaptations to be made as well. Agile testing plays an important part in this: testing at an early stage and in parallel with software development ensures that the quality of the software satisfies requirements. It has never been more important to test early and test often.

The following are six practical tips on how testers can optimise their software testing procedures:

1. Not just some, but all relevant test procedures for a project  – ie, unit, functional and load tests – must be carried out comprehensively as early as possible in the course of the project. By doing this, problems can be identified in good time and rectified accordingly.

2. Unit tests must be integrated into the test management environment. This increases test coverage, particularly in cases where several configurations have to undergo testing.

3. Simple load tests should be incorporated into the daily software build in order to identify performance problems as early as possible.

4. Early test automation leads to greater test efficiency: the inaccuracies of manual processes can be eliminated and tests can be repeated. In addition, test automation makes sprint to sprint regression testing possible.

5. Test starting points should be taken into consideration even at the development stage, for example by attaching ‘testability hooks’ to existing interfaces or adding attributes which can be used for the test procedure.

6. Regular reassessment of test exercise priorities during the course of the project can ensure the flexibility of the process.

Basic test principles also apply in the world of Agile methodology but there are some new additional features too. The ‘Test-Often-and-Early’ method is a particularly compelling one and will sooner or later become part of mainstream test philosophy.

http://microfocus.com

As a profession, we have had to fight for recognition as equals amongst equals. When I started my career as a test engineer, I was told there was a possibility that I may be a bright girl, and so I should aspire to be a programmer. Because how much skill can reasonably be attributed to a person who just checks someone else’s work?

As you know, there is a big difference between possibility and probability. I learnt how to write programs, and even wrote a few deep and meaningful ones. But I never aspired to becoming a programmer. In all probability then, I was never going to be equal. But, still, my Mum said she’d love me regardless, so how bad could it really get?

 The good old days

So I figured, with nothing to lose now, already a ‘failure’, and only just turned 21, I may as well just enjoy myself, at work and at play. So I carried on testing, regardless. And what fun it was, before and after hours. We did the job – by day, we tested; we wrote about it; we fought for fix. The software quality increased. There were fewer defects in production. There seemed to be something in this testing lark. By night – we tested, who could eat the hottest nuclear hot pizza, who could drink the most foul of concoctions, by the pint.

Then people began to leave the company – for the city, big money, bigger nights! Our company was quick on the uptake, they saw the danger, and dealt with it – comprehensively. Our wages went up, and up, until we had pulled away from our peers– yes, there was a good side to Y2K.

Fast-forward to today

Now I’m all grown up. No more those heady nights, I’ve really got to set an example to my niece and nephew. They’re about to head off to university – I need them to work hard so they can pay my pension. As an adult I’ve managed to ruin the planet and the economy for them – not by myself of course, but I’m not sure that’s an excuse.

As professionals though, they will face similar choices to mine. Which job, which country? When I graduated, the advice was to choose the company with the best training programme on offer, not the one with the highest salary. Train first, gain later. Nowadays, like many things, companies are outsourcing their training programmes. In-house trainers still rattle around, but the hot courses seem to be the ones with the external certificates attached. These have their cachet, but they don’t come cheap.

Also, the recession is ignorant of its victims. It will hit anyone, including training and exam providers. They too will dive for cover. Training providers will try to fill up their classes by offering price reductions, exam providers will attempt to cut costs. This may be a reason for the increasing number of multiple-choice-based exams on offer. These certainly have their value, but not at all levels, in all topics.

Testing has its prescriptive elements – that all testing should follow a process from planning through to closure; that a risk-based approach to testing requires identification, analysis and mitigation of risks; that boundary value analysis requires equivalent partitions to be identified first, amongst many others. The rule-book however, does not and cannot claim to inform on what the decision should be when things do not go to plan. Effective decision making requires full understanding of all of the facts. It also requires that know-how – of the people, of the politics, of the budget, of past experiences. The rule book might say that if at first you don’t succeed, try try again. The rule book does not know that each exam costs time and money.

Buyer beware

I know that the external certificates have a certain ‘sugar hit’ – a few days in the classroom, adrenalin pumping exam, certificate through the door. However, we now need to consider – those bucks, that bang – will they match?

I am getting feedback that is not always the case – the hype did not turn into reality. The market likes papers – they make it easier to filter prospective new recruits. Although I do like the option of just throwing away half the CVs – you don’t want unlucky people working for you! (I took this from The Office – a comedy programme in the UK and US). And even if you are lucky, remember that at first review time, it will be you that will be assessed, not your CV. Those papers – where will they be?

So, if you decide that you must have a piece of paper, download the syllabuses and read them. Don’t rely just on the course outlines created by the training providers – they have a purpose – sales. You have a responsibility – budget. Know what you’re buying.

The syllabuses are freely available, read them! And yes, I know you’ll end up reading well over 100 pages for some – if you can’t stay the distance to read them all then put away that purchase order until you have. Buy the sample exam paper and check if the test is applicable to your needs. If there are no full sample papers, then keep that PO in its box. Remember the fundamental rule of trade – caveat emptor – let the buyer beware. Trade is trade, even if it is qualifications being traded. You are a professional – trust that you know what you want, check that you’re likely to get it, and evaluate from end to end. Everyone in the chain is accountable, from delegate to course provider to exam provider. This is the ‘Age of Austerity’ – definitely not the age of blind trust.

Back to the real world

In 2010, our main worries are: Micro – what price our daily bread? Macro – what price our daily wheat? Intergalactic – wonder if we can get wheat to grow on Mars? OK, a bit off the wall, but you’ve got to admit, our own planet just isn’t playing ball these days, it just keeps on spouting stuff at us – clouds, oil, water, disease, fat cat bonuses.

At work, in order for us testers to earn our crust and pass those reviews, our testing skills must be complemented by other skills. As a test analyst, we would typically need to analyse specifications; liaise with business analysts; write test scripts; work with developers to get defects fixed; and write test reports.

Usually we would be part of a team, so we would need to be able to interact with people effectively. As a test manager we would typically be able to participate in risk workshops; write test strategies and plans; assess extent of testing carried out; write reports and present findings. As consultants we would be expected to be all things to all men, and an expert from day one, so pretty fast talking is essential. We also of course have to manage the bods back at base – really fast talking needed here!

To be effective in any of these roles we need to know not just which boxes to tick, we sometimes need to create the boxes first. We need to gather requirements; write reports; deal with people; manage conflict; give presentations; manage budgets; amongst many others.

In addition, we need to keep up with new technologies in order to stay ahead of the game. The lingo of the last decade won’t in itself set you apart from the rest. Knowing what a mash-up is and what it has to do with you just might. Knowing what other healthcare providers around the world are doing, would be invaluable to those in healthcare. Knowing how other countries are dealing with the supply of IT in the public sector might prove useful.

So perhaps we should seek out recognition of not just our testing skills, but our soft skills, our db skills, our IT knowledge. Recognition that our vocabulary also includes words like Agile; SOA; SCRUM; computing in the cloud; QC; accessibility; facebook; iphone; no one’s listening; the developers are over there and I’m over here; the requirements keep changing; timescales are way too short; the customer sets my test window; none of those answers match my thinking.

Big society thinking

In other words, to be a tester in this decade and beyond requires that ‘Big Society’ thinking. But how new is this Big Society idea that we should be aware of the world around us and not hide in our boxes? Albert Einstein said “Try not to become a man of success but rather a man of value.” He said this rather a long time ago, and of course by man he clearly meant tester – he was way ahead of his day after all.

To attempt an answer to this, in good old-fashioned speak, tune in next time, where we will revisit those good old days, to see what they (and Einstein) can do to make us equals amongst equals.

Angelina Samaroo

Managing director

Pinta Education

http://www.pintaed.com/

 So you just finished taking an HTML course at your local college and you’ve hung out your banner announcing your newly acquired skills as a Web Master. As a token of good will, you volunteer your services to your local club or organisation. Not so fast!

After almost 20 years in web design and development – including teaching it for four years – and web testing, I thought I had seen it all. But I’m still amazed at some of the things I see, from both novice Web site builders and professionals. Here’s a tongue-in-cheek list of the some of the things that users despise and many web designers and developers love to do, and they are all things that I have seen…

Multiple scroll bars: Vertical scrolling is typically unavoidable and most users will tolerate it. But horizontal scrolling should be avoided. You know what’s really cool?  Combine the two so a user has to scroll both ways or, even better, build your site using frames and make the user scroll in every frame. Multiple scroll bars rock! Here’s a special tip – require scrolling, but turn off the scroll bars!

Bizarre colour combinations: Make the background lime green and use hot pink text. Even better, use a black background with navy blue text, who needs to read it anyway? Why limit yourself to only four or five colours, most monitors support over 16 million. Try to use as many as you can. If you don’t burn my retinas, you’re not really trying.

Sound: I’m sitting at my computer, with my new Jimmy Buffett CD in the player and I go to your site and I’m immediately hit with the electronic version of YMCA you have playing on your web site… And don’t give me a way to turn it off.

Animated GIFS: When I taught web development, I used to tell my students that I never gave Fs – if they wanted them, they would have to earn them! One sure way to earn one was to put anything on the page that moved. Animated GIFs and marquees were always sure bets. Even better – use lots of them!

Large graphic files: I have nothing better to do than wait for that eleventy-jillion-pixel picture of your cat to load. Use lots of them; then I’ll have time to go get another Landshark Lager… from Las Vegas.

Text, text, and more text: There is nothing I like better than an entire screen of nothing but text. The web is just an electronic book. Just to make sure it all fits on a single screen, use a really small font.

Changing layouts: Make sure every page has a different layout! Keep me on my toes and keep moving stuff around, like buttons and links so I really have to look for them. No two pages should ever look alike.

Background images: If the text on your site isn’t hard enough to read, you can always display it on top of a picture.

Fade-ins/outs: Who would want pages to be displayed immediately when you can slowly fade them in. And you should chain them together so not only do I have to wait for the next page to be displayed, I also have to wait for the current page to disappear.

HTML leaks: Everyone loves broken HTML. No page is complete unless “font>” is displayed somewhere. Never view or test your pages!

Last updated: Always include a ‘last updated’ notice on your site, but never change it. Who wants current information?

Counters: Everyone cares how many visitors come to your site, so be sure to waste plenty of space telling them.

Flash: Not only will it require you to buy expensive software to create it, it also requires the user to download a plug-in to see it.

Awards: Most don’t mean anything, and nobody really cares. If you really want one, I can design one for you!

Empty drop-down boxes/selection lists: Options, who needs ‘em? Also, be sure to include some type of instructions as one of the options. Something like: ‘Please select an item‘ is good. Don’t forget to allow me to actually select it. Make the boxes change width with each selection and blow out the entire page. Or make the width so small I can’t read the options.

Ads: Everyone loves ads and pop-ups. I always wondered what those pop-up blockers did. The more browser windows I can have open the better!

Cursor tails: Sometimes I can’t find my cursor on the page. Please help me by attaching some nonsense text or a graphic to my cursor that will follow it all over the screen. Or change the cursor to something else so it’s really hard to lose!

Broken links/graphics: That red X icon is much more appealing than any picture you could possibly have – and it downloads so much faster. Give me a valid link and I may leave your site and that would be bad! Oh, and of course, never, ever use the ALT attribute. Keep me guessing.

Dave Whalen

President and senior software entomologist

Whalen technologies

http://softwareentomologist.wordpress.com/

Predicting the future for the volatile world of technology services is hardly a straightforward thing to do. But for both suppliers and customers it is an essential consideration. For suppliers it obviously gives them a strategic view and forms the basis of their future business strategy, but for customers it is just as essential. Considering the strategic direction and commercial motivation of a supplier in a selection process is an important stage and one that will often raise concerns or reservations that must be weighed carefully against other criteria. When we consider the competitive landscape in the testing market this has become particularly true, due to the broad area that testing covers; from technology to business and from a specialist level to a management level.

When considering the current market situation, the Tier 1 suppliers such as IBM, Accenture and Tata have already globalised their offering and at the same time created specialist service lines of which Testing is a specific ‘centre of excellence’. The second Tier of suppliers are typically sub $250m companies and they aspire to attain the kind of growth and market share that Tier 1 companies have. Of course achieving this growth requires these Tier 2 businesses to emulate their big brothers, aligning themselves with their customer’s objectives and widening and deepening their service portfolios to avoid reaching a growth plateau with certain customers.

This pattern of ‘widening’ and ‘deepening’ means that as Tier 2 companies get bigger they look for bigger opportunities and the specialist niches become a less attractive market area to them, due in part to their lower overall revenue returns, but also due to their drive to outsource. Hence, there is always opportunity for the new upcoming companies (let’s call them Tier 3) to look for a new niche that the Tier 2 companies are overlooking, or gradually becoming less specialised in.

Traditionally, some of the technical areas such non-functional testing are excellent examples of these specialist niches. But, things are even changing on this front and the ‘niche areas’ are not being confined to the deeper technical tasks such as performance and security. In reality, offshoring and outsourcing are driving different behaviours by the bigger suppliers and causing different demands from the marketplace as a whole.

In the current issue of T.E.S.T I look at how the Tier 1 and Tier 2 companies service the market and the kind of business that they are best suited for. I will highlight the changes in the supplier landscape and how the ‘onshore’ suppliers are growing their services to fill the gaps left. Lastly, I will consider how these dynamics might influence the development of the ‘onshore’ testing services market.

Adam Ripley, MD, Certeco.

Software tests can be divided into two categories: positive tests and negative tests. A positive test is used primarily, if not solely, to validate that a given system, function, operation, etc works as designed when a user enters the right data, in the right place, at the right time, clicks the right buttons, etc. Negative tests try to purposely break the system to verify that the system responds as expected and fails gracefully when it gets the wrong data, in the wrong place, at the wrong time. It’s with negative tests that we really earn our stripes as testers. A negative test should cause an error. It’s expected to cause an error. If it causes an error, and the error is handled correctly, the test passes. A positive test should not cause an error. Does that make it an invalid test – absolutely not!

The ultimate goal

The ultimate goal of positive and negative tests is completely different. We need to verify that something works correctly before we try to break it. If we don’t know that it works correctly, then how can we know when it doesn’t? Positive tests answer that question. If the system doesn’t work as it’s supposed to when everything is correct, all other tests, especially negative tests, are really irrelevant.

Let me state right up front – I’m not a professional software developer – not even close. I never claimed to be one. From my limited experience I like to make sure something works first, and then I focus on what happens when it doesn’t. It’s also much easier to build.

Imagine my surprise when I was recently told that some negative functionality would be delivered before the positive functionality. I was perplexed! The results were predictable. I couldn’t create and save a basic record but the error handling was really nice. In a nutshell, the system could not do what it was designed to do but it looked really good.

Car trouble

It reminds me of a car I bought in high school. It looked really nice in the driveway. It had to sit in the driveway – it rarely ran. I remember a friend telling me that the car was over-rated and didn’t perform the way it was rumoured to. I just wanted it to start and take me to the store. Apparently it also got really bad gas mileage. Of course, sitting in my driveway it got great gas mileage!

Automotive magazines would advertise all kinds of devices to boost miles per gallon. My ‘friends’ would encourage me to purchase these gadgets. “They will pay for themselves after a couple of tanks”. I just wanted the stupid thing to run, and then I’d worry about how much gas it used.

I view positive tests in much the same way. Show me it works like it’s supposed to and then we’ll worry about what happens when it doesn’t or how well it performs (or doesn’t).

Fun time

I always write and run positive tests first. Once the system can pass the positive tests the fun starts. Now I get to be creative and break it. Equivalence Class testing, Boundary Value testing, etc are all great test techniques, but they are effectively useless if the system isn’t functioning correctly to begin with. If the system is failing with valid data, it doesn’t really make sense to test with invalid data – yet. Unless, of course, the system accepts the invalid data – that would be bad. But that’s why we need to test both positive and negative scenarios. Test the positive first to make sure the system responds correctly to good data, correct sequences of operation, correct user actions, etc. Then we can validate what happens when entering invalid data, incorrect sequences, or incorrect user actions.

An additional benefit of positive testing – smoke tests! When you receive a new code drop or build, what better way to validate the core system functionality than to run through your suite of positive tests? Positive tests are my first automation candidates. They are typically quick and easy to run. My smoke tests will usually consist of the entire library of positive tests, or a large subset of them: the critical ones.

I like to target no more than 30 minutes to run a valid, end-to-end smoke-test. With a good test automation tool you can achieve a lot in 30 minutes. I like to run an automated smoke test with every new build, on every environment. If we’re doing daily builds, I run a daily smoke test. When the smoke test passes I can be reasonably sure I have a good system to begin more in-depth testing. I can accept the build, and start my test clock. If it fails – I can kick it back.

For a bit of extra incentive – consider the doughnut factor. If the smoke test passes, I buy doughnuts for the team. If it fails, the development team buys the doughnuts. I hear bagels work too.

Growth will return to the IT industry in 2010. Most industry commentators are in agreement on this, and so, while this is very welcome news, it is not actually tremendously newsworthy. Perhaps of more interest is the fact that such recovery is expected to take place alongside zero growth in IT budgets or employee numbers, placing yet greater pressure on IT to deliver what the business needs. So, if 2009 was about cutting costs and stripping out non-essential projects, 2010 has its emphasis placed firmly on the ‘more’ part of ‘doing more with less’.

To make matters worse, IT’s ability to deliver successful projects currently has its “highest failure rate in over a decade”, according to the Standish Group. And while Gartner figures suggest something less bleak, there continues to be enough high profile examples of failure and waste in IT to suggest that many will struggle to meet the needs of companies focussed on growth and profitability through the course of the next twelve months. In January 2010, one UK national newspaper reported that a “series of botched [Government] IT projects has left taxpayers with a bill of more than £26bn for computer systems that have suffered severe delays, run millions of pounds over budget or have been cancelled altogether.”

Failure to communicate

In many cases, such as the UK’s £12.7bn National Project for IT, users were simply never consulted on “what they wanted the new system to achieve” or kept informed as projects rolled on and challenges arose. This is surely completely unacceptable and, what’s more, totally inexplicable. Especially when 70 percent of production defects are considered to have been created during requirements and design – not to mention that the costs associated with fixing these issues rise exponentially the longer they remain undiscovered. In a March 2009 paper on software quality, Gartner analyst Thomas Murphy quite understandably (and with some degree of under-statement) observes that this is a cause of “IT versus business friction.”

It is little wonder, therefore, that the tight linkages established in 2009 between IT spending and business performance metrics are considered to be here to stay.

With scrutiny and concern the order of the day, many of the industry analyst predictions that have appeared at the start of this new decade are focused on what IT should be doing to improve its success in the process of delivering quality software on time, on budget, and, perhaps more importantly, in line with the needs of the user.

A beautiful friendship

Despite being cash-strapped and under-resourced, there are several factors that would suggest IT has reasons to be optimistic in 2010. Necessity has shown itself to be the mother, if not of invention, then of adoption. Organisations of all sizes are embracing new ways of doing things as they seek to break through yet another glass ceiling of productivity and cost-efficiency.

One view, from Forrester Research, is that they need to start thinking like the “underfunded start-up that is always in the throes of a one-company recession.” It is not about working even harder than they do already. It is about working smarter. It is about working efficiently on the projects that matter, that drive revenue and growth. And staying close enough to the business along the way to know when priorities change and new goals arise.

It’s also about knowing when enough is enough, so that software no longer arrives bloated with redundant features that seemed like a good idea on the drawing board. Many companies find, as requirements are delivered incrementally and in line with business priority, that projects not only complete sooner, but the final 20 percent is often perceived as non-critical for deployment – and often never asked for again.

Such collaboration between all members of the software delivery team yields benefits for everyone. It creates what Thomas Murphy refers to as a “hive” mind set instead of being “adversarial”, where the different disciplines (such as business analysts, testers and developers) bring their respective strengths and viewpoints to bear on the problems being solved, helping to deliver software earlier in the project lifecycle and reducing rework.

You had me at hello

In Gartner’s December 2009 research note, ‘Predicts 2010: Agile and Cloud Impact Application Development Directions’, the analyst firms draws attention to a number of ways in which IT can raise its game and deliver yet greater value to the business, with closer collaboration and the need for a broader definition of software quality very much at the heart of this.

Perhaps one of the most fundamental statements the research note makes is that “software quality can’t be tested at the end”. The IT horror stories mentioned earlier are living proof of this. Companies must look to drive quality throughout the development lifecycle and make use of facilities and processes that support this.

Agile development methods are already helping. They are driving (if not demanding) greater levels of collaboration. And the fact that these methods are now starting to take root in mainstream development shops is great news for lovers of quality software. Gartner believes that by 2012, “agile development methods will be utilized in 80 percent of all software development projects”, and, furthermore, that companies who embrace it, and introduce the cultural and behavioural changes to support it, are already seeing “four times the improvement in overall productivity”.

For IT to succeed in 2010, Vendors have a responsibility to provide not only the tooling, but also the process support to help companies drive quality from start to finish on a project, including helping them shore up weak requirement practices. By moving ‘quality’ upstream within the development process and linking it more closely with the beginning rather than the end of the lifecycle (for example, testing against user stories rather than function points), business and IT will understand each other more fully and improve their chances to share a common goal. Only then will the growth that everyone is predicting, the growth that everyone needs, come to the industry on the back of fundamental, grass roots improvement, rather than through increasing the stress levels of an already stretched group of people. As an industry, it is time to grow up once and for all. It is time to talk.