How software testing can contribute to a decrease in software quality

Time after time, project, program and line managers tried to impress me, telling they will increase the budget for testing, hoping to boost my interest and commitment as a test manager.
Hallelujah!, I thought, when this happened to me for the first time.
I felt the pressure as well as my growing set of responsibilities. Doubling a test team in the heat of the moment with only a couple of months left to deliver a massively buggy program, is not easy.

Especially when at the same time, analysts are being fired, development delays, entry criteria are used to mop the vermin floor and the deadline resides, carved in stone.

As we all know  project delivery is always in fragile balance between cost, time and quality.
While delivering within time and budget constraints trying to aim at quality, in reality quality is often decreasing despite of the increased focus on testing.

How can this possibly happen?
 
1. Start testing too early

It's the project managers first responsibility to deliver in time and budget. To do so the test manager easily gets overruled when entry criteria are not met. Starting to test when a product is in fact not yet testable, means that you start registering defects that are probably already known and are being taken care of. Looking into known defects is a waste of development time and budget which will without exception be rewarded with decreased quality.

2. Replace analysts with testers when analysis is done
The analysts are too expensive to keep them on the project but are also most aware of the difficulties and issues. Replacing them by testers is not only cheaper, but also increases the team freshness. The real knowledge about the product and the issues walked away together with the analysts. The quality of the product inevitably decreases.

3. Make testers and client responsible for product quality
Test cases are the full responsibility of the testers. Testers get demotivated to receive answers to questions as the analysts have disappeared. As an extra security, it is often practiced that the client validates the test cases, as they are responsible to accept. This altogether easily turns into ass-covering exercise.
The issues have smaller chances to be found during testing and when they are found, they are more easily accepted.

4. CMMI Levels and TPI levels are used as silver bullets
The process is sacred. Making sure that everything is done by the book makes a project easy to manage. Following predefined processes seems to add quality. When complexity increases, it is more tempting to fall back on the known processes and procedures, avoiding failure.  However, following processes without identification of anomalies, decreases product quality.



5. Use test management tools to control testing activities
Test management tools can easily contribute to 'report-driven testing'. Tests for example need to be created before execution and can not be increased or run twice because we need to be able to plan how many test cases there will be run per day per person and report against this plan. Test management tools make management give the illusion that testing is defined and measurable and tends to get micromanaged easily. This can incapacitate testers in doing their actual jobs (finding defects as soon as possible) and turn them into mindless test script executors.

Quality Insurance

How does a change in the software affect your business process?
What's the risk of having an issue in production? What's the occurrence of -  and damage on failure?


Most business fear lies in signing analysis documentation to send it off to development. They suddenly become responsible for knowing what they want and how they want it, before ever trying it out. Making mistakes is human, not being able to see the future is human as well. So why would you sign a document, knowing mistakes are most probably inside?
What else could we do, except for agile product delivery to deliver the clients explicit and implicit changing needs? And even there, how can we prevent a business going broke as a result of a software defect that was not found during development?

For example:
http://www.reuters.com/article/2012/08/02/knightcapital-loss-idUSL2E8J27QE20120802

The answer might be.. QUALITY INSURANCE (or QI)

Why the term QI?

Quality insurance is a term used to oppose the often wrongly used term 'Quality Assurance'

How would that conceptually work?

We would make a difference between soft- and hardware insurance.
By default, this insurance system would not focus on the underlying infrastructure. Neither is it explicitly excluded from the insured scope.

A software insurance policy could be taken after the Go/No-go decision was positive and would start at the end of supplier warranty.
For starters, the business processes in scope of the insurance policy would be described by a team of experienced analysts and testers.
An intake is done on the delivered product in a technical go live in a production-like environment.
This intake would consist of an exploratory test.

Possible product issues and risks are reported. Per identified risk, an insurance fee is mapped to an insured amount.
The interesting risks to insure are the ones with a very low probability of occurrence, but having a very high impact on failure.

The company can then decide to reduce risks (and insurance costs) by fixing defects if the fix cost is lower than the fee or decide to insure themselves for this risk.

   
What would be the Strengths?

• Increase the business attention on product risks. Issues can be resolved before go-live.
• The business can go live with a product they are not sure about resulting an overall decreased project failure.
• Direct linking of risks and issues with costs. This confronting exercise increases awareness.

 What would be the Weaknesses?

• Resolving issues late in the development cycle is still more expensive. The direct cost of software delivery is not reduced but rather increased due to the additional insurance policy.

What would be the Opportunities?

• Level out the current overrated importance of certification  - a tester will need to prove what they're worth. Also development quality will stand more in the picture. 
• The real value of a good tester becomes visible. The tester's function would get more appreciation and direct value. 
• The value of the Exploratory Testing method can be proven. Only by exploring, focusing and de-focusing, the dangerous underlying issues, inconsistencies and process gaps can be found... and insured.
• Increase software quality by increasing the awareness of the cost on failure.

What would be the Threats?

• Decrease of software quality and user satisfaction due to competitive insurance policies. 
• Decrease of insurance quality due to competitive insurance policies

10 Signs that you're not cut to be a tester

I had quite some fun reading trough signs indicating why you would be not cut for IT, not cut to be a developer or an IT consultant on the Tech republic blog:
http://www.techrepublic.com/blog/10things/10-signs-that-you-arent-cut-out-for-it/3072

I don't mean to say that it made a whole bunch of sense to me, but I did notice that one particular blog post was still outstanding... so I decided to suggest this one myself:

These are my top 10 signs that you're not cut to be a tester:

1. You get nervous from buggy software

The software you're facing on a day to day base is going to be full of bugs and will look like a poor Picasso imitation. You'll need to find workarounds. And when a bug has been resolved, you'll be facing the next problem. You'll have to go trough again.

2. You're unaware of business expectations

If you're not aware of business expectation, you'll look over half of the defects that are in front of you. You'll read over inconsistencies and you won't question the rationale behind an architectural solution that has been proposed. An overview of what business wants and why they need an application, gives you structure and focus on finding the defects that potentially are of high impact.

3. You get tired from explaining defects occurrences

Not everyone knows the system like you do. Not everyone understands the reason why a bug is a bug. You'll need to explain it... again and again. Even when it works on their machine...

4. You don't read blogs, books or attend conferences about software testing

A tester keeps up to date on the latest evolutions on tools, techniques, methods and can learn from practical experience of other testers. It's no use to invent what already has been invented for you. It is  even less useful not to learn from others mistakes.

5. You're ashamed of your role in the project

A tester could also be an analyst, a developer, a designer or an architect, but chooses to be a tester. A tester is not a developer or analyst, coming close to retirement. A tester is proud of their job and aims on improving software quality. They aim on challenging development, analysis and architecture on their correctness, completeness and coherence. A tester is proud of and dedicated to find the human flaws in a program before it is released to production.

6. You know how to check but you don't know how to explore

Checking against expectations is a given. This needs to be done. After this, the software needs exploration. How else are we going to find those undefined features and prove that if you click on the left mouse button 6 times, then do a page refresh, while keeping your right mouse button clicked and then releasing the right mouse button after which you disconnect the keyboard, your money has gone from your account, but has not been transfered?...

7. You are not keeping up to date on the technical aspects of IT

IT terms are not just hype words for a tester. A tester has to have an understanding of OO, has to know what a web service is or a message queue, what the function is of an isolation layer or why server and client side validations can be implemented while building a website. A tester needs to constantly improve and understand what they are technically dealing with as every technicality has it's weakness. That's where most probably the defects will be.

8. You don't like to communicate

Communication, communication, communication. Testing is about finding bugs and inconsistencies and communicating about them.

9. You are not aware of the application development life cycle

A tester needs to know about architecture, design, analysis, development, infrastructure, release management, ... and also how they fit together. A testers could even carefully address flaws in the application life cycle, since they might result in software bugs later on.

10. You can't get over defects you found and don't get fixed

Face it. You'll find those defects that are obvious and you would never want to see in your software. However, you're not paying for every single defect to be fixed. Decisions will be made that might not make you happy. Your job is to point to the risks and issues of not fixing a defect. Your job is not to get them resolved.

Becoming an Inspiring Test Coordinator

Many starting testers would like to become test coordinators in stead of better testers. However, many good test coordinators are good all-around testers with extra coordination skills. I don't know for myself, how good of a tester I would make but I know I always liked testing and I keep improving my testing skills, even though officially, I'm not testing anymore for the last 3 years.

Seeing Test Coordinators, and working with and for them, I realized there are only few inspiring leaders among them. The biggest pitfalls a test coordinator has in front of them, that keep them from becoming inspiring leaders are the following:

"The common enemy" or "The common goal"?

As a coordinator, you'll need to run test activities like clockwork. The best way to do so is by making your individual testers work as a team.  Bringing people together can be done the easy way, by finding the common enemy, or the hard way, by identifying the common goal. The common goal sounds a bit naive maybe. Facing a goal that you might not reach also requires some sort of bravery. Defining achievable targets towards the common goal, is the key to move on and increase the team spirit on success and even on failure.

"Driving the heard" or "Leading by example"?

Telling people what to do, often results in the opposite effect. The best way to get respect,  is to take part in the test activities. So you better aim to be a good tester, if you ever want to become a good coordinator. Coaching is part of the job. If a tester doesn't know how to get started, you will have to come up with something that works. If your testers miss out on something, it's your job to find out about it and make them aware. 

"Responsibilization" or "Taking the problems away"?

Your testers will face issues constantly. Unstable environments, data restore issues, changing user expectations, defect discussions... Issues will overcome them on a daily base. You have 2 options. Try to let your testers solve the issues by making them responsible or taking their problems away. Taking problems away is often a hard job and might not be very rewarding. You'll need to make your team aware of how you manage their problems, you can even ask them for advice.

"Push pressure further" or "Push pressure back"?

Project managers like to hear that all is going well and tend to become a pain in the ass when testing is not proceeding as planned. Mostly that is because all the buffers they placed in the project are used and your poor test activities are getting squeezed in timelines. Your job is to make sure that testing continues like clockwork and keep management changing ideas, controls and unnecessary reports as far away from your testers as possible. 

"Mushroom coordination" or "Transaparant coordination"?

Knowledge is power. As a test coordinator, you know more than your team members. You take part in steering and coordination meetings. You can use this knowledge either to outsmart your team members and stay one step ahead of them, or you can use it to inform your team members for which they will reward you with trust.

How to start a new mission as a Test Manager

Starting to work in a new company or starting a new mission that sometimes even looked pretty similar to the previous one, I always find every mission to turn out quite different in it's objectives and challenges.

In this blog post I will focus on how I learnt to orient in a new job as a test coordinator or test manager.

Who will I be working for? 

This might be a Project or Programme manager or Line manager - but there might be other important stakeholders out there. Find out who they are. Who will need/want information from you? They can be persons trusted by higher management, key business users, release managers, operation teams...These are your stakeholders. Know who they are and what they want, as you will be working for them and reporting to them (this can be directly or indirectly).


Who will I be working with?

• Who are your team members that will be reporting to you? What are their skills? What do you expect and what can they deliver? Find out what you can learn from them and what you will need to teach them. Also here, involve your stakeholders so that you can align the stakeholders' wishes with the budget they need to set aside for it.
  Your team can be on shore or off shore, external or internal, and work under different contractual situations. You need to get a hold of each of them and find a way to make all of them work together as ONE team.
• Who are  your direct colleagues that you need to share information with? Release management or deployment management, delivery management, analysts, designers, architects, project managers, development, infrastructure... Get to know them and find out for each of them how you will exchange information and what you will require or what might be required from you. You'll need a project planning, a methodology, an architectural blueprint, a data model, designs, analysis or a test environment in a specific way so that you can use it for testing. You'll need to address questions, issues, defects and in this way that they accept them and understand your team's role.

Watch and observe. Treat your colleagues and team members with respect. Understand what they need from you. Make them understand what you need from them. Communicate with them.

How does the project team structure look like?

You can be working in many different kinds of structures using horizontal divisions or vertical divisions, flat structures, resource pools, very hierarchical structures or even complete chaos. A project manager might in reality be an issue manager or a project management officer. A business analyst might be an architect and a designer might be an analyst. Get to an understanding of the project structure and your responsibilities. Find out how the informal and formal roles and responsibilities are defined and where you find yourself in this structure. You can even test the project structure, identify the gaps and bring them up to your management. It's always your job to consult in favor of qualitative delivery.
 

How is testing implemented and what are my responsibilities? 

• Look at test levels - Are you responsible for unit, system testing, system integration testing, acceptance testing? Who will be performing the tasks? Define your scope and check with your stakeholders.
• Look at reference material for testing - What will you need to perform testing and to prepare for testing? Assess the quality of the reference material and inform about possible risks or issues.
• Look at entry and exit criteria - Are you responsible to report on entry criteria or to even monitor them up front?
• Look at the testing approach - How is testing performed currently? Can it be improved? Point to gaps in the used methods and to their risks to your stakeholders and find out if those risks are accepted or if you suddenly increased your job scope.  
• Look at test environments and infrastructure - Are you responsible for defining, ordering, maintaining test environments or test infrastructure, defining or improving release processes? 
• Look at test types. Are you responsible for functional testing only or also for usability testing, security testing, regression testing, performance testing... ? Do you have the required skills in your team to address the required test types to be performed?
• Look at test tools. Which test tools will you require or are standard in the company? Do you need to define or adhere to procedures? Make sure all your team members are mastering and if required contributing to the tool selection and setup.
• Look at communication and reporting patterns - What is discussed in which structural meetings? Which deliverables, reports and metrics are expected at which times? What would you think would be required? Come forward with proposals.

Refer to existing, representative products and processes in the company. Find out how testing has been performed in the past. What was expected, what was delivered and how it was the result perceived by stakeholders.

Don't go in and re-invent the wheel. Gather the existing knowledge.  If there is one, adopt the existing way of working. Bring your colleagues, team members and stakeholders towards improvement step by step.


Some tips and tricks that help me

• Make a list of your colleagues' names. You can make a drawing of the desks in the office and put their names on it.
• Document what you learn as you learn it (if not yet documented) and store it in a central place. Let other team members contribute to that.  (special phone numbers, creation of test users,...)
• Hear people out. Ask for their opinions. Show that you can listen - and learn from their skills and experience.
• Wear clean, washed clothes. Don't go meeting people directly after smoking.
• Say good morning and good evening to everyone on the floor.
• Don't say directly YES or NO if you don't understand the request or question completely, but say you will look into it and come back on the next day.
• Set-up workshops with your business users and identify the important end to end scenario's with them as soon as possible. This will keep you focused during project delivery.
• Always get to an agreement with at least your key stakeholders separately before proposing new elements to them together in a meeting.
• Estimate and plan, using techniques and common sense. Make plans with input from your team members. Make them all agree on and commit to the plans before presenting and committing a plan to your management.
• As a people manager, try to make a positive difference for your people. 

List of requirements for a new test environment

It was the first time we needed an integration test environment that we didn't have yet.
There was no infrastructure, no procedures, no version control, no data restore process,...
What we needed was a controlled test environment where we could test integration for a project of considerable size.

As a tester, I know that knowing what you want, is not obvious and putting it all together to make it work is even more difficult. So, a long brainstorm session with all stakeholders, some Google sessions and blog-scans came afterwards.
It lead to the following list of requirements that we used as a base, to start building our integration test environment.

Testability requirements

• All applications that exist or are to be delivered in the production environment have to exist or simulated in the test environment. 
• List of settings and configurations is available together with the comparison of the settings and configurations of the production environment. 
• Required tools and licenses are available for testers (If for any reason tools can not be provided, a work-around is in place to facilitate testing)
• Front end devices to access software through different channels are available (card readers, GPS Units, desk tops, laptops, pads, phones,...)
• Batch procedures are internally controllable and automated as in a production environment (manual trigger, start, pause, stop batch procedures)
• Measuring tools are in place to enable monitoring the communication between applications and application layers.
• External Connectivity to test environments for support, deployment and testing is required (as different vendors need to deploy and test their software and integration on the environment)
• Time traveling is possible in at least one of the following means: "time goes by faster (eg: 1 week = 1 hour)", "move forward and reset from a defined time in the future", "coherent data creation/update to a  time stamp in the past"
• Input and/or output simulators are in place and documented for every interface type
• All configurations of the different interfacing systems within the test environment have to be centrally managed 
• At least 1 test user exists for every defined role in the application under test
• Database read access is required for all testers
• The test infrastructure has an availability of 95% (planned deployments are not included)
• Defects are centrally managed, using 1 agreed defect managing system and flow 

Test data requirements

• Any production data loaded into the test environment is scrambled in a way that it still fits logically together but is impossible to link back to actual clients in the production database
• A representative set of  production(-like) test data can be unloaded in the test environment
• Back-up and restore procedures exist to back-up and release a partial or full data extract
• Back-up and restore procedure does not take longer than 4 working hours
• Back-up and restore procedure should be possible at least on a weekly basis with a lead time of 1 working day
• Full Data refresh with scrambled production data is possible on request  with a lead time of at least 1 working week. 
• A procedure to request for a data is implemented

 

 Deployment/Release management requirements

• Version control is applied to all development products and system documentation
• Deployments are always done from 1 agreed central data store
• Roll-back scenario's are in place for every deployment cycle
• Users are trained and informed about deployment management agreements
• A patch procedure is known, documented and implemented
• Deployment tasks follow formalized checklists - verified by the test team
• Builds or data refreshes in a test environment take only place after approval of the impacted test teams
• A deployment/release management role is assigned
• Every defect fixed for a new deployment in the test environment is at least communicated within an agreed defect management tool 
• Every deployed, fixed component is at least communicated trough a build note
• Builds are subject to formal and/or informal entry criteria

Maintainability requirements

• Procedures to add/delete or update hardware components are existing and followed
• The test environment components are sized in relation to the production environment
• Log files are available for the different applications deployed on the test environment
• Procedure exists to request and add test users for any application in the test environment
• Disaster recovery plan is known, documented and implemented
• Unexpected downtime of the (partial) test environment is communicated to all stakeholders (preferably using a central communication channel - for instance an intranet website)
• Physical locations of different infrastructural components are identical to the production environment and documented. Any deviation is clearly identified and approved by the project team 

I definitely forgot some requirements and not all of them are actually in place. But but based on these we set up an environment we can control with a handful of people and test in properly. We are agile enough to patch and refresh data properly which improved the test experience drastically.
Any suggestions to improve this list are welcome.

How to deliver a qualitative product

The mind-map I'm posting, has been created by looking back on different projects and consulting different colleagues, including my wife, the best Business Analyst I've ever seen. It is most probably not complete. But I think it is ready to start sharing it.

The idea behind this mind-map is to focus on key aspects required to deliver a qualitative software product of considerable size and how testers can contribute to obtain a higher level of quality of the product to be delivered.

If a tester wants to contribute to quality, they can do more than testing a product when it is delivered. Testers have many other duties during software development. They know the product and project in- and outputs and are aware of their risks. They understand the changing business needs and their explicit and implicit requirements. They challenge the architecture and analysis, coach developers and acceptants to check their delivered product effectively on standards. They facilitate in environment and data preparation. A tester understands and contributes to the release and deployment mechanisms. 

Planning & Control

It is the testers job to:

• Take part in the project planning and approach; challenge the planning
• Look into the processes and challenge them in function of quality delivery
• Find dependencies and flag them up to project/programme management
• Understand the test requirements and build a log of them
• Advise on test tool selection in function of the budget, timeline, experience and test requirements.

 Design

It is the testers job to:

• Understand the product needs and rationale behind the project
• Test the designs and architectural models and point to their weaknesses
• Find and report flaws in communication patterns and stakeholder involvement
• Coach acceptants in the definition of E2E prototyping test cases; lead them and help them in the  E2E test preparation. 

Delivery

It is the testers job to:

• Test, test, test, and automate tests
• Coach developers in checking against standards
• Contribute to and cooperate closely with deployment and release management
• Coordinate proper and clean bug reporting and the bug life cycle 
• Provide and manage the required test infrastructure.

Acceptance

It is the testers job to:

• Coach the acceptants in their verification and testing process
• Analyze and report test results in simple and understandable language
• Provide release advise
• Coordinate E2E test execution.

Communication

A tester is a great communicator. They understand the business needs, reason with analysts, challenge architects and managers and coach users and developers.