The role of QA in the Agile organization

What is quality not (anymore)?

Some companies (still) think of quality as a standard set of KPI's that should be met for all projects in the company and reported on.

Those KPI's consist in the worst case for example of amount of analysis documents finished in time VS finished late, amount of test cases prepared per designed function, quotes on how well templates are being followed, bugs per test case, reopened bugs, bug fix times etc...

One role in the company is ultimately responsible for implementing quality in systems.

This role is the QA Lead or QA Manager who represent the project "head of police". They take their best-practice sheet out of the magical best practice box and start implementing it in the company.

The result of this way of implementing quality in a company producing software is a product that meets KPI's, was more costly to build and the people building it getting frustrated from working in a framework they don't support.

People on the floor don't seem to see the added value of the QA Manager and their KPI's, but those KPI's are sent way up, so they need to shine... in the worst case at the cost of product quality.

Maybe this was an acceptable way to approach quality when building products using the waterfall and V-model, and as a step to take in order to learn how to do better.
Now there are better ways to do this.

What is quality and how to pursue it?

"Quality" is like any software requirement. The idea of what an application needs to and may not do, changes over time and not all quality aspects can or should be fulfilled in order to deliver a successful project.

Each and every project member needs to understand the goal of what they are building and why they are building it. That's the first step in delivering software quality - a clear common goal.

The team with all their individual members are the owners of quality, the QA lead is their coach. 

The definition of what quality is, is formulated by it's stakeholders and can change over time.

What it means in reality is that good software quality (how stakeholders perceive quality) can only be reached when the common understanding of quality is carried by the team.

Stakeholders in this case are not only the business, but also the developers, testers, end users, release team, support teams etc...

The QA job in a project starts by making sure all stakeholders are identified in the first place and making sure they are aware they need to provide (continuous) input on what they need from a software product in order to reach the level of quality they want to achieve.

The next thing QA needs to do, is coach the product owner (or project sponsor - who owns the budget) in deciding which of all identified aspects are more or less important considering the budget and timelines. 

Then QA talks with the team on how these quality aspects will be put in place. The team will decide, together with QA on the definition of done (What are the measurable aspects of the software product that need to be fulfilled before we can ever say any component of this software has been "released".)

By doing so, the team defines how they will reach quality standards to ensure development speed, release efficiency, product maintenance, product aesthetics, acceptance criteria etc...

Based on the team's "definition of done"  that evolves over time, the individual project will gain more maturity in understanding what quality means for their project and will start reporting on measurable project related KPI's.

The QA Lead can assist in setting up and maintaining those reports, but needs to take care not to become the owner of those reports.

Projects are further monitored by testers (or test engineers, QA engineers or whatever you call the people who look for flaws in products and processes). They assess process and product, and contribute to the definition of done by testing the product, but also by checking with the team and stakeholders on the completeness and correctness of the definition of done  - that can be adapted after every iteration.

At the end of every iteration, it is important to look back on what went well and what went wrong in this project.

Lessons learned and incentives are the most valuable aspects a project can receive. Making sure those sessions take place and result in action points, can also be a QA role.

Results of implementing quality are not statistics in the means of the traditional overall KPI's anymore, but are project specific KPI's and even more importantly - they result in more successful projects, more accurate cost estimations, faster release cycles, faster delivery cycles, faster defect fix-rates, closer business cooperation and even higher retention levels of employees.

How software testing can contribute to a decrease in software quality

Time after time, project, program and line managers tried to impress me, telling they will increase the budget for testing, hoping to boost my interest and commitment as a test manager.
Hallelujah!, I thought, when this happened to me for the first time.
I felt the pressure as well as my growing set of responsibilities. Doubling a test team in the heat of the moment with only a couple of months left to deliver a massively buggy program, is not easy.

Especially when at the same time, analysts are being fired, development delays, entry criteria are used to mop the vermin floor and the deadline resides, carved in stone.

As we all know  project delivery is always in fragile balance between cost, time and quality.
While delivering within time and budget constraints trying to aim at quality, in reality quality is often decreasing despite of the increased focus on testing.

How can this possibly happen?
 
1. Start testing too early

It's the project managers first responsibility to deliver in time and budget. To do so the test manager easily gets overruled when entry criteria are not met. Starting to test when a product is in fact not yet testable, means that you start registering defects that are probably already known and are being taken care of. Looking into known defects is a waste of development time and budget which will without exception be rewarded with decreased quality.

2. Replace analysts with testers when analysis is done
The analysts are too expensive to keep them on the project but are also most aware of the difficulties and issues. Replacing them by testers is not only cheaper, but also increases the team freshness. The real knowledge about the product and the issues walked away together with the analysts. The quality of the product inevitably decreases.

3. Make testers and client responsible for product quality
Test cases are the full responsibility of the testers. Testers get demotivated to receive answers to questions as the analysts have disappeared. As an extra security, it is often practiced that the client validates the test cases, as they are responsible to accept. This altogether easily turns into ass-covering exercise.
The issues have smaller chances to be found during testing and when they are found, they are more easily accepted.

4. CMMI Levels and TPI levels are used as silver bullets
The process is sacred. Making sure that everything is done by the book makes a project easy to manage. Following predefined processes seems to add quality. When complexity increases, it is more tempting to fall back on the known processes and procedures, avoiding failure.  However, following processes without identification of anomalies, decreases product quality.



5. Use test management tools to control testing activities
Test management tools can easily contribute to 'report-driven testing'. Tests for example need to be created before execution and can not be increased or run twice because we need to be able to plan how many test cases there will be run per day per person and report against this plan. Test management tools make management give the illusion that testing is defined and measurable and tends to get micromanaged easily. This can incapacitate testers in doing their actual jobs (finding defects as soon as possible) and turn them into mindless test script executors.

How to deliver a qualitative product

The mind-map I'm posting, has been created by looking back on different projects and consulting different colleagues, including my wife, the best Business Analyst I've ever seen. It is most probably not complete. But I think it is ready to start sharing it.

The idea behind this mind-map is to focus on key aspects required to deliver a qualitative software product of considerable size and how testers can contribute to obtain a higher level of quality of the product to be delivered.

If a tester wants to contribute to quality, they can do more than testing a product when it is delivered. Testers have many other duties during software development. They know the product and project in- and outputs and are aware of their risks. They understand the changing business needs and their explicit and implicit requirements. They challenge the architecture and analysis, coach developers and acceptants to check their delivered product effectively on standards. They facilitate in environment and data preparation. A tester understands and contributes to the release and deployment mechanisms. 

Planning & Control

It is the testers job to:

• Take part in the project planning and approach; challenge the planning
• Look into the processes and challenge them in function of quality delivery
• Find dependencies and flag them up to project/programme management
• Understand the test requirements and build a log of them
• Advise on test tool selection in function of the budget, timeline, experience and test requirements.

 Design

It is the testers job to:

• Understand the product needs and rationale behind the project
• Test the designs and architectural models and point to their weaknesses
• Find and report flaws in communication patterns and stakeholder involvement
• Coach acceptants in the definition of E2E prototyping test cases; lead them and help them in the  E2E test preparation. 

Delivery

It is the testers job to:

• Test, test, test, and automate tests
• Coach developers in checking against standards
• Contribute to and cooperate closely with deployment and release management
• Coordinate proper and clean bug reporting and the bug life cycle 
• Provide and manage the required test infrastructure.

Acceptance

It is the testers job to:

• Coach the acceptants in their verification and testing process
• Analyze and report test results in simple and understandable language
• Provide release advise
• Coordinate E2E test execution.

Communication

A tester is a great communicator. They understand the business needs, reason with analysts, challenge architects and managers and coach users and developers.