Why testing is NOT a "standard process"

In the beginning there was consumption...

It all starts with the consumption society in which we, software testers, live.
An average consumer, willing to spend some money, asks for a certain custom product.
Being a consumer for quite a while, and having the experience of buying standard products, we know a price and an exact scope of  what we're going to get for our money.

When I for example go and buy a car, I know exactly what I'm going to pay, how my car will look like and when I'm going to receive it. The same goes by the way for a book, cd or even a software package.

So our problem, of testing being a process, begins exactly here. We are all very familiar with the trade of standard products. But there is a big difference between standard products and custom made products. Many clients and suppliers are not very competent in buying or selling such a non-standard, custom product.
And even though you think your car is customized when you have carefully chosen your options and colour - you just configured your car with the standard options and colour a supplier provides for this car. If you disagree, go to the garage and try to have an integrated GPS with bluetooth module, traffic board recognition and rear-driving camera, ESP and ABS system in a 1992 Opel Corsa and you'll see what I'm talking about.

When testing is required, we're talking mostly about new product development. We're going to build or integrate something that we never built or integrate before. So this is NOT a standard product that we can just take off a shelf.

What we seem to like to omit, is the fact that in custom software production, we're going to make something for the first time. We might integrate with components that aren't there yet, using technologies we haven't been using yet, working with people we have never seen before. There are a lot of unknown factors here.


So how do we need to buy or sell a custom product?

When we sell or buy something that does not yet exist, we don't want to find out that what we wanted is not really what we needed on the moment we spent all our money. We want to verify step by step and have the possibility to learn and steer and even more, get a good understanding about the real cost of the complete product we're asking for.

What we need to do is learn quickly, as we will be making false assumptions on cost or complexity, forgot about important things and maybe had some misunderstandings on all different levels. We have to keep our eyes open, and our focus sharp. Is this product that we're building fit for purpose? Are we on the right track? We better find out that we made a mistake before we spent all those millions.

So ideally we plan, build and act in small iterations. Each iteration gives us time to act and learn so that we can adopt in order to do better during the next iteration. Testing is a part of acting and is taking place in close cooperation with all stakeholders and is therefore a important driver of the learning and improvement of the team.

We buy and sell by building a solid trust relation between buyer and seller that gives room for continuity and results after a couple of iterations in a very clear view on budget, scope, planning and their  respective interferences.


Process VS Learning

Now what is a process?
A process is a series of predefined actions that produce something or that lead to a particular result.
Processes tend to always start from the same starting points, with known input and the output of a process is also known.

Processes are used to streamline factories and service companies that offer standard products or services. We don't want those people, building cars, delivering lettres or producing cd's to learn and improve. They have to follow the process. We want them to work effectively and at a minimal cost.

When building custom products, it is close to impossible to have a concrete view on any standard process to follow right from the start as there is no routine yet, that could be the base for a process to be put in place. On the contrary, there are many unknown factors in input, actions to be taken and even outputs.

This does not mean that no agreements can be made about communication and cooperation.
By learning and improving, processes and procedures might be put in place during software development. But those processes  and procedures will be very specific and tailored to the people and the context of this piece of software that is being built.


Conclusion

I would like to claim that testing can not be performed successfully by only following any standard process. Quality of any custom made software product can not be guaranteed by following any process. Good and effective testing requires expertise, knowledge, experience and skill.

During testing, it is the learning curve that can lead to creation of specific processes that can be applied within projects, programs or cooperating teams. But it is the learning curve and close stakeholder cooperation that leed to less issues in production, not any process.

If you read this and you agree with me, please go and sign here to support the testing community.

The role of QA in the Agile organization

What is quality not (anymore)?

Some companies (still) think of quality as a standard set of KPI's that should be met for all projects in the company and reported on.

Those KPI's consist in the worst case for example of amount of analysis documents finished in time VS finished late, amount of test cases prepared per designed function, quotes on how well templates are being followed, bugs per test case, reopened bugs, bug fix times etc...

One role in the company is ultimately responsible for implementing quality in systems.

This role is the QA Lead or QA Manager who represent the project "head of police". They take their best-practice sheet out of the magical best practice box and start implementing it in the company.

The result of this way of implementing quality in a company producing software is a product that meets KPI's, was more costly to build and the people building it getting frustrated from working in a framework they don't support.

People on the floor don't seem to see the added value of the QA Manager and their KPI's, but those KPI's are sent way up, so they need to shine... in the worst case at the cost of product quality.

Maybe this was an acceptable way to approach quality when building products using the waterfall and V-model, and as a step to take in order to learn how to do better.
Now there are better ways to do this.

What is quality and how to pursue it?

"Quality" is like any software requirement. The idea of what an application needs to and may not do, changes over time and not all quality aspects can or should be fulfilled in order to deliver a successful project.

Each and every project member needs to understand the goal of what they are building and why they are building it. That's the first step in delivering software quality - a clear common goal.

The team with all their individual members are the owners of quality, the QA lead is their coach. 

The definition of what quality is, is formulated by it's stakeholders and can change over time.

What it means in reality is that good software quality (how stakeholders perceive quality) can only be reached when the common understanding of quality is carried by the team.

Stakeholders in this case are not only the business, but also the developers, testers, end users, release team, support teams etc...

The QA job in a project starts by making sure all stakeholders are identified in the first place and making sure they are aware they need to provide (continuous) input on what they need from a software product in order to reach the level of quality they want to achieve.

The next thing QA needs to do, is coach the product owner (or project sponsor - who owns the budget) in deciding which of all identified aspects are more or less important considering the budget and timelines. 

Then QA talks with the team on how these quality aspects will be put in place. The team will decide, together with QA on the definition of done (What are the measurable aspects of the software product that need to be fulfilled before we can ever say any component of this software has been "released".)

By doing so, the team defines how they will reach quality standards to ensure development speed, release efficiency, product maintenance, product aesthetics, acceptance criteria etc...

Based on the team's "definition of done"  that evolves over time, the individual project will gain more maturity in understanding what quality means for their project and will start reporting on measurable project related KPI's.

The QA Lead can assist in setting up and maintaining those reports, but needs to take care not to become the owner of those reports.

Projects are further monitored by testers (or test engineers, QA engineers or whatever you call the people who look for flaws in products and processes). They assess process and product, and contribute to the definition of done by testing the product, but also by checking with the team and stakeholders on the completeness and correctness of the definition of done  - that can be adapted after every iteration.

At the end of every iteration, it is important to look back on what went well and what went wrong in this project.

Lessons learned and incentives are the most valuable aspects a project can receive. Making sure those sessions take place and result in action points, can also be a QA role.

Results of implementing quality are not statistics in the means of the traditional overall KPI's anymore, but are project specific KPI's and even more importantly - they result in more successful projects, more accurate cost estimations, faster release cycles, faster delivery cycles, faster defect fix-rates, closer business cooperation and even higher retention levels of employees.