There was no infrastructure, no procedures, no version control, no data restore process,...
What we needed was a controlled test environment where we could test integration for a project of considerable size.
As a tester, I know that knowing what you want, is not obvious and putting it all together to make it work is even more difficult. So, a long brainstorm session with all stakeholders, some Google sessions and blog-scans came afterwards.
It lead to the following list of requirements that we used as a base, to start building our integration test environment.
Testability requirements
- All applications that exist or are to be delivered in the production environment have to exist or simulated in the test environment.
- List of settings and configurations is available together with the comparison of the settings and configurations of the production environment.
- Required tools and licenses are available for testers (If for any reason tools can not be provided, a work-around is in place to facilitate testing)
- Front end devices to access software through different channels are available (card readers, GPS Units, desk tops, laptops, pads, phones,...)
- Batch procedures are internally controllable and automated as in a production environment (manual trigger, start, pause, stop batch procedures)
- Measuring tools are in place to enable monitoring the communication between applications and application layers.
- External Connectivity to test environments for support, deployment and testing is required (as different vendors need to deploy and test their software and integration on the environment)
- Time traveling is possible in at least one of the following means: "time goes by faster (eg: 1 week = 1 hour)", "move forward and reset from a defined time in the future", "coherent data creation/update to a time stamp in the past"
- Input and/or output simulators are in place and documented for every interface type
- All configurations of the different interfacing systems within the test environment have to be centrally managed
- At least 1 test user exists for every defined role in the application under test
- Database read access is required for all testers
- The test infrastructure has an availability of 95% (planned deployments are not included)
- Defects are centrally managed, using 1 agreed defect managing system and flow
Test data requirements
- Any production data loaded into the test environment is scrambled in a way that it still fits logically together but is impossible to link back to actual clients in the production database
- A representative set of production(-like) test data can be unloaded in the test environment
- Back-up and restore procedures exist to back-up and release a partial or full data extract
- Back-up and restore procedure does not take longer than 4 working hours
- Back-up and restore procedure should be possible at least on a weekly basis with a lead time of 1 working day
- Full Data refresh with scrambled production data is possible on request with a lead time of at least 1 working week.
- A procedure to request for a data is implemented
Deployment/Release management requirements
- Version control is applied to all development products and system documentation
- Deployments are always done from 1 agreed central data store
- Roll-back scenario's are in place for every deployment cycle
- Users are trained and informed about deployment management agreements
- A patch procedure is known, documented and implemented
- Deployment tasks follow formalized checklists - verified by the test team
- Builds or data refreshes in a test environment take only place after approval of the impacted test teams
- A deployment/release management role is assigned
- Every defect fixed for a new deployment in the test environment is at least communicated within an agreed defect management tool
- Every deployed, fixed component is at least communicated trough a build note
- Builds are subject to formal and/or informal entry criteria
Maintainability requirements
- Procedures to add/delete or update hardware components are existing and followed
- The test environment components are sized in relation to the production environment
- Log files are available for the different applications deployed on the test environment
- Procedure exists to request and add test users for any application in the test environment
- Disaster recovery plan is known, documented and implemented
- Unexpected downtime of the (partial) test environment is communicated to all stakeholders (preferably using a central communication channel - for instance an intranet website)
- Physical locations of different infrastructural components are identical to the production environment and documented. Any deviation is clearly identified and approved by the project team
I definitely forgot some requirements and not all of them are actually in place. But but based on these we set up an environment we can control with a handful of people and test in properly. We are agile enough to patch and refresh data properly which improved the test experience drastically.
Any suggestions to improve this list are welcome.
