The intake test

Before we want to start testing a product, it needs to be testable for the level of testing we want to perform. To ensure the subject under test is testable, we could require a whole set of measurable requirements as for example:

• The product (part) is delivered with all it's functional dependencies
• All interfacing products are in place, issues are known
• Amount of defects of certain severities closed in previous test levels VS total are met
• Amount of elements planned to test in previous test levels VS tested are met
• Test environment is setup and tested
• Test data is prepared and consistently scrambled over the whole integrated platform
• Batches are running
• Configuration is setup and tested
• Test preparation is done
• Testers are trained
• Licenses for tools are handed out and confirmed
• ...

When facing reality, these entry criteria are formulated at the very start of the project, but often barely maintained and applied as the delivery pressure becomes higher.

But how often do we end up in situations where at least one of those entry criteria are not met, which blocks us from proceeding with testing? And how many times did we assess entry critria as passed but in fact they were not? A defect could be wrongly allocated, users have access to the tools, but have the incorrect user rights, the environment is build but it can't be accessed... Releasing untestable chunks of code to testers or releasing in an environment that is not fit for testing, is despite many project managers opinion a waste of time and resources.
Environments need to be maintained longer, defects are discussed with more stakeholders and have longer turn-around times.

In short: Bad entry criteria management costs the project time and money.

A pragmatic and easy way to keep a decent check on the status of your entry criteria, is to define an intake test for every substantial module under test.
This intake test should be executable in a very limited amount of time and describe the positive flow(s) trough your modules under test. Based on the result of this intake test, delivery issues can be found instantly. And which statistical report can compare with "showing that it works". From the moment of ordering the environment, the teams focus should be on making this test work and all reasons why it doesn't will return into critical issues or defects that require high priority to solve.

When the intake test for a module is successful, it can be recorded and stored in a library as a proof and testing for this module can start.

An example of a simple defect life cycle

This blog post is providing a practical example of what I think is a minimal defect flow, accompanied with the roles and responsibilities

In general the flow has 3 end statuses.

• Rejected - Closed: This will tell something about the quality of defects that are raised. Rejects occur most often when registering duplicates or when the desired functionality is not well understood.
• Tolerated - Closed: Defects that end in this status are defects that are known. When going to production, these are the known issues that the product will be shipped with. We need to keep them separated from the rejected and fixed defects.
• Fixed - Closed: Defects that end in this status are unambiguously fixed defects that have been found resolved.

 Some roles have been defined, regarding the defect flow

• Issuer: Anyone who registers a defect. This is not necesarily a tester.
• Tester: Someone who is trained in testing, knows the functionality of the product to be delivered and has a brief technical insight into development, infrastructure and architecture. The tester can challenge the not accepted defects and is able to retest defects in their context.
• Defect manager: Someone who has development insight and decision power to accept, not accept and assign defects. This can role often maps on a development management role. 
• Developer: A wide role for anyone who can fix defects in the code, infrastructure, parametrization or even in the design documentation. A defect does not necessarily reside only in code. A defect can also be an inconsistency in the design documentation.

On top of this it is advised to agree on SLA's on defect resolution times and agree on defect severity levels and/or priority levels. They can be combined and interrelated.
I mostly use the following basic levels:

• Urgent -  Prio 1: Defect needs a patch - testers or users are blocked and no workaround can be put in place - Patch required within 2 working days
• High - Prio 2: Defect might need a patch but a workaround can be put in place or a less substantial amount of functionality is blocked. - Update required in a week
• Medium - Prio 3: Defect needs to be resolved, but there is a workaround possible. - All defects need to be resolved before go-live
• Low - Prio 4: Cosmetic defects. - 70 percent of defects need to be resolved before go-live

Sometimes cosmetic defects increase in ranking, when for example they occur on documents that are sent out to customers.

The status flow and its description below depicts the minimum requirements for a defect flow to be implemented.

From status	Responsible	Description	To status
-	Anyone / Issuer	Anyone who finds a defect, can register it in the system. A defect has an unambiguous title, a clear description of what is expected and what is observed and is evidenced.	Open
Open	Defect manager	Assesses the opened defect and decides that the defect is valid. The defect is not yet in the pipeline of being resolved but it has been agreed with a developer to start working on, taking priorities and timelines into consideration	Accepted
Open	Defect manager	Assesses the opened defect and decides that the defect is not valid. The defect is updated with the rationale behind the disagreement.	Not accepted
Not accepted	Tester	The Tester assesses the non accepted defects and where they do not agree, they discuss with the defect manager and project manager. TOGETHER, they agree that the defect is valid and move the defect to the accepted status and directly assign it to a developer.	Accepted
Not accepted	Tester	The tester assesses the  non accepted defects and where they agree, they consult with the issuer of the defect that it can be rejected.	Closed Rejected
Accepted	Developer	The developer indicates that they start working on the defect assigned to them.	In progress
In progress	Developer	The developer agrees with the project manager that other defects get priority over this particular defect under fix. The defect fix is put on hold.	On hold
In progress	Developer	The developer indicates that the fix has been made and updates the defect with information on the fix that has been applied	Fixed
On hold	Defect manager	The defect manager or development manager propose defects that can be tolerated in the release of the software to production. The defects that are accepted by all stakeholders, move to the status closed-tolerated so that they can be addressed in a later release.	Closed Tolerated
Fixed	Defect manager	When a build or fix is released in a test environment, the defects that were fixed, will be re-testable. The defect manager updates those defects to the retest status.	Retest
Retest	Tester/issuer	Any defect to be retested is taken up by a tester to verify if the defect has been really fixed in all instances. When the defect has been partially fixed or not fixed (not when another regression defect occurs) the defect is set to the status Open with the according evidence and comments.	Open
Retest	Issuer	Any defect to be retested needs ultimately retesting of the issuer of the defect. When the fix is confirmed, the issuer brings the defect to the final status of Closed-Fixed	Closed-Fixed

Introducing proper defect management

A defect process, a process where project related defects (bugs) are addressed and resolved, can be run efficient or can become frustrating and very time consuming. Certain people don't take up their responsibilities, certain procedures are not clear and therefore not followed. Defects start flying up and down between parties and start revealing some people's deepest emotional frustrations or  interesting dialogues, sometimes they even carry the only analysis of a certain topic available in the whole program... 
A defect is not an information exchange process, nor is it an analysis addendum or personal ventilation tool. A defect should be clear and unambiguous. The turn-around time of a defect should be quick. Therefore a defect needs to be light and to the point.

Process

The defect status flow is most often discussed. You might be using the companies existing process flow. However, often even if the process is defined, it is not always unambiguously explained. When your company is less mature in testing and test process implementation, it is advised to keep status flows as simple as possible. More statuses require more overhead and result in more confusion.
The 3 main questions you need to clarify in order to describe a defect process are:

1. What are the circumstances for a defect to arrive in a certain status?
2. Which role is responsible to treat a defect when it arrives in a certain status?  
3. What are the next possible statuses a defect can arrive in?

Defect procedures

Defect procedures look into your particular instance of the maybe generic defect flow. Who will take up certain roles? What are the steps to be undertaken in certain situations? Agree on clear responsibilities.
These are, which I found vital defect procedures 

• Defect coordination board with as main purpose treating defects that are not agreed upon. 
• Defect retest procedures. Who retests where and how do we know a defect fix is deployed in which environment.

Defect guidelines or instructions 

Defect guidelines or instructions are the good practices you share with all testers and developers when registering a defect. A developer needs to know how to reproduce a defect and a manager needs to see what a defect is exactly about at a first glance.

These are defect guidelines that I find most important:

• Summary - One line issue description, following a convention of giving the location where the defect occurred and a short and clear description of the defect that occurs: e.g. Billing - Field 'Street' should also accept numeric values 
• Comments - Comments are added to a defect after them being discussed. No changes are made to a defect without a complete and concise explanation.
• Prioritization - Define severity, impact and priority parameters unambiguously. The more statuses you invent, the more complex it will become to keep an overview. Keep it simple.

Defect coordination is mainly communication

You can introduce a flawless defect process, defect procedures and guidelines, if they are not understood and carried by all involved parties, your defect coordination is doomed.  When you are to become a defect coordinator, make sure you involve your main stakeholders. The easiest way to do so is to involve them in a workshop and start from a decent proposal. In environments where defect management is new, it often gets confused with production issue management. If defect management is a completely new item in the company, it is recommended to do a dry run with your stakeholders. 

How software testing can contribute to a decrease in software quality

Time after time, project, program and line managers tried to impress me, telling they will increase the budget for testing, hoping to boost my interest and commitment as a test manager.
Hallelujah!, I thought, when this happened to me for the first time.
I felt the pressure as well as my growing set of responsibilities. Doubling a test team in the heat of the moment with only a couple of months left to deliver a massively buggy program, is not easy.

Especially when at the same time, analysts are being fired, development delays, entry criteria are used to mop the vermin floor and the deadline resides, carved in stone.

As we all know  project delivery is always in fragile balance between cost, time and quality.
While delivering within time and budget constraints trying to aim at quality, in reality quality is often decreasing despite of the increased focus on testing.

How can this possibly happen?
 
1. Start testing too early

It's the project managers first responsibility to deliver in time and budget. To do so the test manager easily gets overruled when entry criteria are not met. Starting to test when a product is in fact not yet testable, means that you start registering defects that are probably already known and are being taken care of. Looking into known defects is a waste of development time and budget which will without exception be rewarded with decreased quality.

2. Replace analysts with testers when analysis is done
The analysts are too expensive to keep them on the project but are also most aware of the difficulties and issues. Replacing them by testers is not only cheaper, but also increases the team freshness. The real knowledge about the product and the issues walked away together with the analysts. The quality of the product inevitably decreases.

3. Make testers and client responsible for product quality
Test cases are the full responsibility of the testers. Testers get demotivated to receive answers to questions as the analysts have disappeared. As an extra security, it is often practiced that the client validates the test cases, as they are responsible to accept. This altogether easily turns into ass-covering exercise.
The issues have smaller chances to be found during testing and when they are found, they are more easily accepted.

4. CMMI Levels and TPI levels are used as silver bullets
The process is sacred. Making sure that everything is done by the book makes a project easy to manage. Following predefined processes seems to add quality. When complexity increases, it is more tempting to fall back on the known processes and procedures, avoiding failure.  However, following processes without identification of anomalies, decreases product quality.



5. Use test management tools to control testing activities
Test management tools can easily contribute to 'report-driven testing'. Tests for example need to be created before execution and can not be increased or run twice because we need to be able to plan how many test cases there will be run per day per person and report against this plan. Test management tools make management give the illusion that testing is defined and measurable and tends to get micromanaged easily. This can incapacitate testers in doing their actual jobs (finding defects as soon as possible) and turn them into mindless test script executors.

Quality Insurance

How does a change in the software affect your business process?
What's the risk of having an issue in production? What's the occurrence of -  and damage on failure?


Most business fear lies in signing analysis documentation to send it off to development. They suddenly become responsible for knowing what they want and how they want it, before ever trying it out. Making mistakes is human, not being able to see the future is human as well. So why would you sign a document, knowing mistakes are most probably inside?
What else could we do, except for agile product delivery to deliver the clients explicit and implicit changing needs? And even there, how can we prevent a business going broke as a result of a software defect that was not found during development?

For example:
http://www.reuters.com/article/2012/08/02/knightcapital-loss-idUSL2E8J27QE20120802

The answer might be.. QUALITY INSURANCE (or QI)

Why the term QI?

Quality insurance is a term used to oppose the often wrongly used term 'Quality Assurance'

How would that conceptually work?

We would make a difference between soft- and hardware insurance.
By default, this insurance system would not focus on the underlying infrastructure. Neither is it explicitly excluded from the insured scope.

A software insurance policy could be taken after the Go/No-go decision was positive and would start at the end of supplier warranty.
For starters, the business processes in scope of the insurance policy would be described by a team of experienced analysts and testers.
An intake is done on the delivered product in a technical go live in a production-like environment.
This intake would consist of an exploratory test.

Possible product issues and risks are reported. Per identified risk, an insurance fee is mapped to an insured amount.
The interesting risks to insure are the ones with a very low probability of occurrence, but having a very high impact on failure.

The company can then decide to reduce risks (and insurance costs) by fixing defects if the fix cost is lower than the fee or decide to insure themselves for this risk.

   
What would be the Strengths?

• Increase the business attention on product risks. Issues can be resolved before go-live.
• The business can go live with a product they are not sure about resulting an overall decreased project failure.
• Direct linking of risks and issues with costs. This confronting exercise increases awareness.

 What would be the Weaknesses?

• Resolving issues late in the development cycle is still more expensive. The direct cost of software delivery is not reduced but rather increased due to the additional insurance policy.

What would be the Opportunities?

• Level out the current overrated importance of certification  - a tester will need to prove what they're worth. Also development quality will stand more in the picture. 
• The real value of a good tester becomes visible. The tester's function would get more appreciation and direct value. 
• The value of the Exploratory Testing method can be proven. Only by exploring, focusing and de-focusing, the dangerous underlying issues, inconsistencies and process gaps can be found... and insured.
• Increase software quality by increasing the awareness of the cost on failure.

What would be the Threats?

• Decrease of software quality and user satisfaction due to competitive insurance policies. 
• Decrease of insurance quality due to competitive insurance policies

10 Signs that you're not cut to be a tester

I had quite some fun reading trough signs indicating why you would be not cut for IT, not cut to be a developer or an IT consultant on the Tech republic blog:
http://www.techrepublic.com/blog/10things/10-signs-that-you-arent-cut-out-for-it/3072

I don't mean to say that it made a whole bunch of sense to me, but I did notice that one particular blog post was still outstanding... so I decided to suggest this one myself:

These are my top 10 signs that you're not cut to be a tester:

1. You get nervous from buggy software

The software you're facing on a day to day base is going to be full of bugs and will look like a poor Picasso imitation. You'll need to find workarounds. And when a bug has been resolved, you'll be facing the next problem. You'll have to go trough again.

2. You're unaware of business expectations

If you're not aware of business expectation, you'll look over half of the defects that are in front of you. You'll read over inconsistencies and you won't question the rationale behind an architectural solution that has been proposed. An overview of what business wants and why they need an application, gives you structure and focus on finding the defects that potentially are of high impact.

3. You get tired from explaining defects occurrences

Not everyone knows the system like you do. Not everyone understands the reason why a bug is a bug. You'll need to explain it... again and again. Even when it works on their machine...

4. You don't read blogs, books or attend conferences about software testing

A tester keeps up to date on the latest evolutions on tools, techniques, methods and can learn from practical experience of other testers. It's no use to invent what already has been invented for you. It is  even less useful not to learn from others mistakes.

5. You're ashamed of your role in the project

A tester could also be an analyst, a developer, a designer or an architect, but chooses to be a tester. A tester is not a developer or analyst, coming close to retirement. A tester is proud of their job and aims on improving software quality. They aim on challenging development, analysis and architecture on their correctness, completeness and coherence. A tester is proud of and dedicated to find the human flaws in a program before it is released to production.

6. You know how to check but you don't know how to explore

Checking against expectations is a given. This needs to be done. After this, the software needs exploration. How else are we going to find those undefined features and prove that if you click on the left mouse button 6 times, then do a page refresh, while keeping your right mouse button clicked and then releasing the right mouse button after which you disconnect the keyboard, your money has gone from your account, but has not been transfered?...

7. You are not keeping up to date on the technical aspects of IT

IT terms are not just hype words for a tester. A tester has to have an understanding of OO, has to know what a web service is or a message queue, what the function is of an isolation layer or why server and client side validations can be implemented while building a website. A tester needs to constantly improve and understand what they are technically dealing with as every technicality has it's weakness. That's where most probably the defects will be.

8. You don't like to communicate

Communication, communication, communication. Testing is about finding bugs and inconsistencies and communicating about them.

9. You are not aware of the application development life cycle

A tester needs to know about architecture, design, analysis, development, infrastructure, release management, ... and also how they fit together. A testers could even carefully address flaws in the application life cycle, since they might result in software bugs later on.

10. You can't get over defects you found and don't get fixed

Face it. You'll find those defects that are obvious and you would never want to see in your software. However, you're not paying for every single defect to be fixed. Decisions will be made that might not make you happy. Your job is to point to the risks and issues of not fixing a defect. Your job is not to get them resolved.

Becoming an Inspiring Test Coordinator

Many starting testers would like to become test coordinators in stead of better testers. However, many good test coordinators are good all-around testers with extra coordination skills. I don't know for myself, how good of a tester I would make but I know I always liked testing and I keep improving my testing skills, even though officially, I'm not testing anymore for the last 3 years.

Seeing Test Coordinators, and working with and for them, I realized there are only few inspiring leaders among them. The biggest pitfalls a test coordinator has in front of them, that keep them from becoming inspiring leaders are the following:

"The common enemy" or "The common goal"?

As a coordinator, you'll need to run test activities like clockwork. The best way to do so is by making your individual testers work as a team.  Bringing people together can be done the easy way, by finding the common enemy, or the hard way, by identifying the common goal. The common goal sounds a bit naive maybe. Facing a goal that you might not reach also requires some sort of bravery. Defining achievable targets towards the common goal, is the key to move on and increase the team spirit on success and even on failure.

"Driving the heard" or "Leading by example"?

Telling people what to do, often results in the opposite effect. The best way to get respect,  is to take part in the test activities. So you better aim to be a good tester, if you ever want to become a good coordinator. Coaching is part of the job. If a tester doesn't know how to get started, you will have to come up with something that works. If your testers miss out on something, it's your job to find out about it and make them aware. 

"Responsibilization" or "Taking the problems away"?

Your testers will face issues constantly. Unstable environments, data restore issues, changing user expectations, defect discussions... Issues will overcome them on a daily base. You have 2 options. Try to let your testers solve the issues by making them responsible or taking their problems away. Taking problems away is often a hard job and might not be very rewarding. You'll need to make your team aware of how you manage their problems, you can even ask them for advice.

"Push pressure further" or "Push pressure back"?

Project managers like to hear that all is going well and tend to become a pain in the ass when testing is not proceeding as planned. Mostly that is because all the buffers they placed in the project are used and your poor test activities are getting squeezed in timelines. Your job is to make sure that testing continues like clockwork and keep management changing ideas, controls and unnecessary reports as far away from your testers as possible. 

"Mushroom coordination" or "Transaparant coordination"?

Knowledge is power. As a test coordinator, you know more than your team members. You take part in steering and coordination meetings. You can use this knowledge either to outsmart your team members and stay one step ahead of them, or you can use it to inform your team members for which they will reward you with trust.