Improving the Test Process: Implementing Improvement and Change - A Study Guide for the ISTQB Expert Level Module (2014)

Chapter 6. Process for Improvement

The implementation of test process improvements is likely to be more effective if the improvement process itself is supported by a standard description. One such description, the IDEAL process improvement framework, is described in this chapter. We describe the individual IDEAL phases, explain the activities to be performed, and provide an overview of the principal output documents: the assessment plan and the test improvement plan.

6.1 Introduction

Syllabus Learning Objectives

6.1.1 IDEAL Process Improvement Framework

Primarily, the models described in chapter 3 provide a list of best practices and/or a description of a mature test process. They typically do not offer a standard approach to a deployment and improvement program in an organization. To support the implementation of these types of improvement models, the Software Engineering Institute (SEI) has developed a framework model for process improvement: IDEAL [McFeeley 1996].

The IDEAL model has proven to be very useful when implementing both model-based and analytical-based test improvements. IDEAL offers an extensive and practical reference standard for improvement processes and also shows what needs to be done when implementing test process improvements in an organization. The model contains a five-phase improvement cycle as shown in table 6–1.

IDEAL An organizational improvement model that serves as a road map for initiating, planning, and implementing improvement actions. The IDEAL model is named for the five phases it describes: initiating, diagnosing, establishing, acting, and learning.

Table 6–1 The IDEAL five-phase improvement cycle

In addition to IDEAL, there are several other models for the implementation of process improvement. In general, all of these models are based on Edward Deming’s Plan, Do, Check, Act (PDCA) cycle (see section 2.4.1). The Deming cycle starts with making a plan that determines the improvement goals and how they will be achieved (plan). Then the improvements are implemented (do), and it is determined whether the planned advantages have been achieved (check). Based on the results of this assessment, further actions are taken as needed (act).

This chapter is about the phases and activities of IDEAL. The phases of an improvement program in accordance with IDEAL are shown in figure 6–1.

Before we get to each phase of the IDEAL model, the next section describes the need to first establish an overall test policy that also addresses test process improvement. The test policy is basically the driver and starting point for all test improvement activities. The test policy is also a means to create management commitment. Before starting with the test process improvement, you should know why you are doing this, what (business) problems will be addressed, and what the main principles for testing are in your organization.

Figure 6–1 Phases of an improvement program according to IDEAL

6.1.2 Test Policy

When an organization wants to improve its test process, it should first clearly define a test policy [van Veenendaal and Wells 2012]. There are several questions that should be answered up front by means of a test policy that are relevant to test process improvement:

What is the problem?

What are we trying to achieve?

Why are we doing test process improvement?

The test policy describes the organization’s philosophy toward testing (and possibly quality assurance). It is set down either in writing or by management direction and lays out the overall objectives regarding testing that the organization wants to achieve. This policy should be developed in cooperation with stakeholders, such as information technology, research and development, and product development departments. It should reflect the organizational values and goals as they relate to testing.

The test policy is preferably complementary to or a component of a broader quality policy or even business policy. This quality policy describes the overall values and goals of management with regard to quality.

As stated, the test policy defines the organization’s overall objectives, goals, and strategic views regarding testing. A test policy is necessary to attain a common view of testing and its objectives between all stakeholders within an organization. This common view is required to align test activities (including process improvement) throughout the organization. The test policy should address testing activities for both new development and maintenance projects.

Objectives for test improvement should be stated within the test policy. These objectives should subsequently be translated into a set of key test performance indicators. The test policy and the accompanying performance indicators provide a clear direction and a means to communicate expected and achieved levels of testing performance. The performance indicators must show the value of testing and test process improvement to the stakeholders and provide quantitative indications of whether the organization is improving and achieving the defined set of test (improvement) goals. Within the TMMi model, the definition of a test policy is even part of the model itself, within the level 2 process area Test Policy and Strategy. By comparison, the TPI NEXT model includes a single checkpoint for whether the test policy is followed.

Modification of the test policy and test strategy is usually required as an organization’s test process evolves and moves to a higher maturity level. The test policy may also reference a standard for testing terminology to be used throughout the organization.

Where a written test policy exists, it may be a short, high-level document that accomplishes the following objectives:

Provides a definition of testing, such as building confidence that the system works as intended and detecting defects

Lays out a fundamental test process, such as, for example, test planning and control, test analysis and design, test implementation and execution, evaluation of test exit criteria and test reporting, and test closure activities

Describes how to evaluate the effectiveness and efficiency of testing—for example, by the percentage of defects to be detected (Defect Detection Percentage or DDP) and the relative cost of defects detected in testing as opposed to after release

Defines desired quality targets, such as reliability (e.g., measured in terms of failure rate) or usability

Specifies activities for test process improvement, such as, for example, application of the TMMi or TPI NEXT model or implementation of recommendations from project retrospectives

A test policy might typcially include the following elements [van Veenendaal and Wells 2012]:

A definition of testing

A definition of debugging (fault localization and repair)

Basic views regarding testing and the testing profession

The objectives and added value of testing

The quality levels to be achieved

The level of independence of the test organization

A high-level test process definition

The key responsibilities of testing

The organizational approach to and objectives of test process improvement

Test policy A high-level document describing the principles, approach, and major objectives of the organization regarding testing.

An anonymized real-life example of a short one-page test policy is provided in figure 6–2.

Figure 6–2 Example of a test policy

6.2 Initiating the Improvement Process

Syllabus Learning Objectives

The IDEAL model starts with the phase called Initiating. In the initiating phase, the infrastructure for a successful change program is established. Actions taken at this initial stage directly influence the final results of the improvement process. Poorly initialized improvement processes may deliver unsatisfactory results and significantly reduce the chances of being able to take any future improvement initiatives.

The goals and expected results with respect to the change program and what needs to be contributed by the different parties concerned are defined. The goals of a test improvement implementation need to be in line with the quality goals and the organizational goals. In this phase, the goals cannot always be formulated according to the SMART model (specific, measurable, achievable, realistic, and time-bound), which is why the goals are specified in more detail in the establishing phase.

In this phase, management commitment requires explicit attention; management commitment is needed from test management, IT management, and business management.

At the initiating phase, a number of people-related critical success factors must be considered; these can have an influence on achieving improvement objectives:

Level of knowledge—for example, people who have a lack of knowledge, for instance of testing issues, may as a consequence be adverse to changes.

Organizational culture—for example, the organization may not be process-oriented and yet you are trying to define mature test processes.

Initiating (IDEAL) The phase within the IDEAL model where the groundwork is laid for a successful improvement effort. The initiating phase consists of the following activities: set context, build sponsorship, and charter infrastructure.

People culture—for example, there is culture where people get blamed if a failures occurs, and as a consequence, there may be strong resistance to defining a test measurement program.

Level of acceptance—for example, an initiative for software process improvement was started a few years ago and failed. As a consequence, people look at this new initiative very skeptically.

Recommendations for test improvement should be sensitive to these people issues and be able to suggest alternative improvement strategies depending on the styles, culture, and needs of the people in the organization.

The initiating phase has the following main activities:

Identify stimulus for change

Set objectives for test improvement

Set context

Build sponsorship

Charter infrastructure

Each of these activities is discussed in the following sections.

6.2.1 Identify Stimulus for Change

As a first step in establishing the need for improvement, IDEAL refers to the identification of a “stimulus for change” that generates awareness about the need for process improvement. The stimulus can come from a number of sources:

Dissatisfaction with the results of current testing (e.g., poor testing efficiency)

Unexpected incidents (e.g., major failures in production)

Changing circumstances (e.g., organizational change or changes to quality or test policy)

Customer demand (e.g., to achieve a formal certification for test maturity)

Market trends or information taken from an internal measurement repository (e.g., a benchmark result that indicates that achieved levels of quality are too low)

Areas for improvement may be captured and defined based on preliminary analysis and interviews with stakeholders. As part of the analysis it may be necessary to determine current indicators such as the total cost of quality, based on the total cost of failures in production and the total cost of testing.

The proposed change needs to contribute to the success of the organization and needs to complement any existing quality and organizational goals. The extent to which the process change is in accordance with the organization’s goals largely determines the success of the change.

6.2.2 Set Objectives for Test Improvement

Clear objectives for test improvement are needed to provide a clear focus and to enable the activities to be reviewed against the objectives. These are a key to success. The objectives of test process improvement must always be established in relation to business values and goals. There are typically three steps for setting objectives for test improvement:

Establish a general vision for the (improved) future

Set specific objectives

Align testing improvement to business goals

The general vision and the objectives for test improvement are typically documented in the organization’s test policy (see section 6.1.2).

Establishing a General Vision for the Future

Test process improvement must focus on the (business) benefits required by a sponsoring stakeholder and a vision of the overall objectives to be achieved. Improvement initiatives need such a vision of the future for the following reasons (for example):

Sponsors need to be convinced of the return on investment before committing resources.

The management of any necessary changes must relate to agreed-upon objectives. Any improvement action should contribute directly or indirectly to achieving the agreed-upon objectives.

Inability to define a common vision may result in failure of the proposed test improvements (see chapter 9):

Poorly defined objectives may hide unresolved conflicts of interest.

We may focus on inappropriate or unachievable objectives that may be a waste of resources and/or fail to deliver improvements.

Although all of this may sound very obvious to the reader, we have come across many organizations that have been attempting to improve the testing process for some time unsuccessfully. Asking the questions “Why?” and “What are the business objectives behind the improvement process?” resulted in total silence. No common objectives were established and agreed upon up front. Only with real management commitment and clear (business) objectives can test process improvement be successful.

Setting Specific Objectives

As stated, specific, well-defined objectives are needed for any test process improvement for the following reasons:

They enable appropriate actions to be taken.

The success (or failure) of the improvement efforts can be defined and reported back to management and other stakeholders.

Commitment can be obtained from management and other stakeholders.

A number of possibilities are available to enable objectives to be represented:

Quantitative objectives with metrics. For example, the Goal-Question-Metric (GQM) method (see section 4.3.) enables metrics to be defined and can be used to link objectives (goals) with measurable results.

Objectives expressed as maturity levels. If test process improvement is to be conducted using a process model, objectives may also be represented in a form appropriate for the model in question. This typically involves defining specific levels of maturity to be achieved either for the test process as a whole (e.g., managed) or for individual aspects of the test process (e.g., test design). This type of objective is often applied as a marketing vehicle by (test) organizations providing testing services to the market (for example, organizations that perform outsourced testing).

It is also possible to have qualitative objectives, perhaps supported by appropriate scales (e.g., from “very bad” to “very good” or “getting better” to “getting worse”). However, it is strongly recommended to combine qualitative objectives with quantitative objectives.

Aligning Testing Improvements to the Organization

Test process improvements must never be done in isolation. They must be aligned to the business goals and objectives of any software process improvements being performed and any organizational improvements taking place (see chapter 2).

Corporate dashboards and balanced scorecards can be used by the organization to allow test improvements to be aligned to organizational improvement targets.

Corporate Dashboard

A corporate dashboard is an easy-to-read, often single-page source of information showing graphical presentations of the current status (snapshot) and historical trends of an organization’s key performance indicators. They allow managers to monitor the contribution of the various processes and departments in their organization and enable instantaneous and informed decisions to be made at a glance.

Using corporate dashboards provides the following benefits:

Visual presentation of performance indicators

Ability to identify and correct negative trends

Ability to make more informed decisions based on collected data

Alignment of strategies and organizational goals

Ability to save time compared to studying multiple reports

Ability to gain total overall visibility instantly

Figure 6–3 shows an example of a corporate dashboard. Note that it is not intended for the reader to be able to read every detail of this figure but to get a feeling of what a corporate dashboard can look like. Dashboards are rapidly becoming more popular within testing and are used by some test managers to show the status of testing; for example, a one-page dashboard might show status regarding requirements coverage, test progress, defect status, and product quality.

Corporate dashboard A dashboard-style representation of the status of corporate performance data.

Figure 6–3 Example of a corporate dashboard

Balanced Scorecard

The balanced scorecard is a strategic planning and management instrument that is used extensively to align business activities to the vision and strategy of the organization, improve internal and external communications, and monitor organization performance against strategic goals. It was originated by Drs. Robert Kaplan and David Norton [Kaplan and Norton 1996] as a performance measurement framework that added strategic nonfinancial performance measures to traditional financial metrics to give managers and executives a more “balanced” view of organizational performance. The balanced scorecard has evolved from its early use as a simple performance measurement framework to a full strategic planning and management system.

Recognizing some of the weaknesses and vagueness of most management approaches, the balanced scorecard approach provides a clear indication as to what companies should measure in order to “balance” the financial perspective. The balanced scorecard is a management system (not only a measurement system) that enables organizations to clarify their vision and strategy and translate them into actions. It provides feedback regarding both the internal business processes and external outcomes to continuously improve strategic performance and results.

Balanced scorecard A strategic tool for measuring whether the operational activities of a company are aligned with its objectives in terms of business vision and strategy.

The balanced scorecard suggests that we view an organization from four perspectives and that we develop metrics, collect data, and analyze it relative to each of these perspectives (see figure 6–4).

Figure 6–4 Four perspectives of a balanced scorecard

The learning and growth perspective includes employee training and corporate cultural attitudes related to both individual and corporate self-improvement.

The business process perspective refers to internal business processes. Metrics based on this perspective allow the managers to know how well their business is running and whether its products and services conform to customer requirements (the mission).

The customer perspective relates to the importance of customer focus and customer satisfaction in any business.

The financial perspective reflects the traditional need for financial data.

Some examples of aligning business value to test improvements are shown in the following sections. This list provides test and quality topics that can be used as a source of inspiration to integrate testing into a corporate dashboard or balanced scorecard. Note how some of the targets almost explicitly relate to the views of quality introduced in chapter 2.

Learning and Growth

People targets—for example, job satisfaction, staff turnover, and sickness, which may align to any of the quality views but will also affect the transcendent view of quality (trust, reputation)

Innovation and improvement targets—increased number of new products to market, speed to market, and process/framework/standards accreditation (e.g., TMMi or industry standard) aligned to the “value” view of quality

Customer

Customer targets—for example, improved market share, improved customer satisfaction, improved risk management process; also aligned to the user view of quality

Improved product quality—aligned to the product, manufacturing, and user views of quality

Social involvement/political targets—for example, environmental impact of the organization, reputation, and publicity, which may align to any of the quality views but will also affect the transcendent view of quality (trust, reputation)

Internal Business Processes

Internal targets—for example, greater predictability of project outcomes, reduced faults/failures during the software development, reduced project elapsed time, and reduced effort/costs aligned to the manufacturing view of quality

Financial

Financial targets—for example, productivity improvements, improved revenue, improved profit, improved financial turnover; also aligned to the value view of quality, test productivity, and/or costs of quality

Note how the preceding list is explicitly linked to the four perspectives of the balanced scorecard.

6.2.3 Set Context

The management needs to determine how the change effort fits the quality and business strategy. Which specific organizational goals will be realized or supported by the test improvements? How are current projects influenced by the improvement? Which proceeds need to be yielded—for example, in terms of fewer issues and incidents or the shortening of the lead time for test execution? During the project, the context and effects will become more concrete, but it is important to be as clear as possible early in the project.

Part of setting the context is related to defining the scope of the test process improvement project. This includes obtaining answers to the following questions:

General process scope: Which processes other than the test process are in scope?

Test process scope: Which parts of the test process need to be addressed?

Test levels: Which test levels are within the scope of the improvement program?

Project scope: Are project(s) or the organization in scope?

6.2.4 Build Sponsorship

Gaining support from the responsible managers, or building sponsorship, is extremely important in improvement projects. This involves test management, IT management, and business management sponsorship because all these stakeholders will be influenced by the change.

Sponsorship is important during the entire project, but because of the insecurity caused by changes, active support at the beginning of the project is especially important. Providing support for the improvement program is a major part of sponsorship, but sponsorship also includes, for example, providing active participation or promoting the project when there is resistance. During this activity, stakeholders are presented with the results from the previous step of the improvement process; for example, the objectives for test improvement and test policy and their commitment to these results needs to be obtained. Without their commitment, the improvement project will fail in the short or medium term.

6.2.5 Charter Infrastructure

As a final activity in the initiating phase, the way in which an improvement project is executed is determined. An organizational infrastructure such as a Test Process Group is put in place, and it must be described explicitly, including responsibilities and qualifications.

Usually the infrastructure consists of a project board guiding several improvement teams. In the project board are the sponsor, possibly the other sponsors of other improvement projects, the manager of the improvement program, and possibly an external consultant. In addition, there is often an (external) test expert. The project board is ultimately responsible for the improvement program and agrees on plans, milestones, and final results. It has the ultimate power to decide and is the highest level of escalation.

Organizational issues are described in further detail in section 7.2..

6.3 Diagnosing the Current Situation

Syllabus Learning Objectives

Diagnosing (IDEAL): The phase within the IDEAL model where it is determined where one is, relative to where one wants to be. The diagnosing phase consists of the following activities: characterize current and desired states and develop recommendations.

High Level Overview

In the Diagnosing phase it is determined where the organization stands as opposed to what it wants to achieve. The current state of the organization is determined and its desired future state is clearly formulated.

The diagnosing phase consists of the following main activities:

Characterize the current and desired states

Develop recommendations

At the end of the diagnosing phase, a test assessment report is available.

Characterize Current and Desired States

The activities performed depend largely on whether the approach to be taken to test process improvement is model-based (chapter 3) or analytical (chapter 4).

If a model-based approach is to be used, then an assessment will be planned and performed in which measurements are compared to a reference standard such as, for example, TMMi level 2. A test process improvement model can be used to define the desired state.

An assessment, either formal or informal, is conducted to determine the current state. The proposed practices of a test improvement model may be used as a checklist to determine the maturity of the test processes. The desired state must align with the stimulus for change as determined in the initiating phase and it must be within the realm of possibility for the organization.

If an analytical-based approach is to be adopted, the various causal analysis techniques described in chapter 4 may be applied and metrics, measures, and indicators analyzed. Refer to section 6.3.5 for more information about analyzing results.

Process assessment A disciplined evaluation of an organization’s software processes against a reference model [After ISO/IEC 15504].

Develop Recommendations

The recommendations suggest a way of proceeding in subsequent activities. Which test processes are addressed first? Which part of a test process is to be addressed and in what way? The recommendations may be formulated under the guidance of (internal or external) test experts in the specific process area, and they are typically formulated in a test assessment report.

Detailed Tasks

The Expert Level syllabus breaks down the principal activities defined in IDEAL into the following tasks:

Planning the assessment

Preparing for the assessment

Performing interviews

Giving feedback

Analyzing results

Performing solution analysis

Recommending improvement actions

A detailed description of these individual tasks is given in the following sections.

6.3.1 Planning the Assessment

The purpose of the planning phase is to come to an agreement with the sponsor about scope, costs, activities to be carried out, timeline, planned feedback sessions, and reports. The planning is executed using the assessment request (the documented or undocumented request to conduct an assessment) and discussions between the client and the (intended) assessor.

Prior to its initiation, a plan for the assessment is formulated. This plan is distributed to all key stakeholders, among which are the assessment sponsor and the assessment team.

There are three main inputs to the assessment plan, as follows:

1. The purpose of the assessment—for example, determining if a particular TMMi level has been attained, determining the defects in certain process areas, or identifying recommendations for optimizing the test process

2. The scope of the assessment—defines how wide-ranging the assessment is, including organizational elements, process areas, and projects that will be covered

3. The constraints—for example, availability of key resources, the maximum duration of the assessment, specific organizational areas to be excluded from the assessment, and any confidentiality agreements that need to be taken into account

The assessment plan identifies the activities, resources, schedules, and responsibilities and includes a description of the intended assessment outputs. If the assessment is not intended to include all aspects of the test process, the planning identifies which aspects are to be covered and in which sequence. It should also explicitly state which aspects are not covered.

If the TMMi model is used, a decision needs to be made about whether a formal or an informal assessment is required. A formal assessment has enough depth to officially determine to what extent an organization meets the requirements as defined in the model.

An informal assessment does not lead to an official result about the process maturity; it provides only an indication. An informal assessment is often used to identify the major improvements that need to be made, and it can also be used to determine the progress of an improvement implementation. This form of assessment is often adequate as an initial survey.

The roles and responsibilities for performing assessments are described in section 7.2..

6.3.2 Assessment Preparation

The purpose of the preparation phase is to prepare everything needed to conduct the assessment and to make a detailed schedule for the (optional) kick-off meeting, interviews, progress reports, feedback sessions, presentation, and report. Depending on the scope and size of the assessment, the detailed schedule may already have been defined in the planning phase.

During this phase a document study is conducted, the purpose of which is as follows:

To gain insights into the current testing process prior to interviewing those involved.

To prepare particular questions for the interview.

To perform formal elements of the assessment that do not require discussion. For example, the documents may be checked to ensure completeness and conformity to standards.

The document study starts with providing the organization with a checklist, in which the required test documentation can be found. Typical documents for an assessment of the organization include test policy and test strategy. For project assessments, the test plans, test specifications, defect reports, and test reports are in focus.

When the assessor has been provided with this documentation, the document study can be started to gain initial information about the current situation within the organization. The findings are documented in the initial version of the assessment report. It is important that when the findings are documented, their source is also recorded, so the results of the assessment can be verified.

Part of the assessment preparation includes ensuring that a suitable environment exists for conducting the planned interviews. Interview participants will not be able to freely give information if they are not comfortable, if there are constant disturbances, or if confidentiality cannot be guaranteed. A self-contained, quiet room away from the interviewee’s own working area is generally acceptable.

6.3.3 Performing Interviews

The purpose of the interview phase is to collect information about the current test practices. The following activities need to be executed:

Conducting interviews as a means to gather and verify information

Performing a preliminary result analysis, which can lead to additional questions in successive interviews or even to the planning of additional interviews

Interviews are performed according to the assessment plan, and the content is generally guided by the model being used.

Interviews are mostly conducted on an individual basis to allow the interviewee freedom to express views and to ensure that a “secure” environment exists. The interviewer should always discuss the importance of confidentiality for interviewees and information providers prior to an assessment taking place, particularly if sensitive issues are likely to arise.

It may be tempting to conduct group discussions instead of individual interviews. Even though this might save time, the quality of the information received may suffer from group dynamics. Individuals react differently to group discussions depending on their personality types; some may dominate discussions with their own opinions and others may feel reluctant to contribute. These effects should not be allowed to influence information gathered in the assessment.

A wide range of skills are required for a successful interview. These are described in more detail in section 7.3.1.

6.3.4 Giving Feedback

At the end of the interviews an initial feedback may be given. This is particularly helpful in confirming initial assessment findings with the interviewee and for clarifying any misunderstandings.

The initial feedback may also take place based on several interviews and to a larger audience. It will then typically provide an overview of main points to stakeholders and may be an important opportunity to manage their expectations regarding final results and recommendations. Care should be taken to maintain the rules of confidentiality at all times and to avoid assigning blame for any problems identified.

6.3.5 Analyzing Results

The analysis of results is conducted according to the overall approach adopted to test process improvement.

If a model-based approach is adopted, the assessment results are compared to the model. When a process model such as TMMi or TPI NEXT is used, the assessment results are evaluated against the conditions (e.g., specific practices or checkpoints) for achieving a given process maturity level described in the model. When a content model such as CTP is used, results are compared against the practices recommended by the model. The latter will be expert-based because no clear set of conditions is defined by the content-based models.

If an analytical-based approach to test process improvement is adopted, the analysis of results is one of the principal activities performed and the techniques described in chapter 4 are applied:

Analysis of metrics.

For example, if Defect Detection Percentage (DDP) is being applied as an indicator and has fallen below the required level, a causal analysis of failures found in production should be performed.

Analysis of interactions between different process components using systems thinking [Weinberg 1992].

For example, the results of interviews with a test manager may show that there is frustration about stakeholders not reading the test status reports they create. The reports have therefore become increasingly poor quality with many items missing. Interviews with the stakeholders indicate that the reports started off much too lengthy and are not directed at their actual information needs. At meetings with the test manager, stakeholders showed little interest for issues raised in the reports by the test manager. Using systems thinking, the interactions between the stakeholders and test manager show a negative reinforcing loop (vicious circle), which results in a steady decline in the quality of the test status reports. The worse the reports are, the less they are read by the stakeholders and the less motivated the test manager becomes to write good reports.

Analysis of tipping points

Certain aspects of the Tipping Point theory [Gladwell 2000] can help identify specific points in a system where apparently insignificant events may result in the rapid worsening of a situation. In the context of test processes, the failure to maintain a master test plan, for example, may be considered a tipping point that signals to the entire test team that test documents in general are not usable and a waste of time. The neglect of test documents may spread rapidly throughout the team by word of mouth until no test documentation is created at all. This may be the result observed in assessment interviews. Performing analysis of these results using Tipping Point theory may help expose the cause.

Analysis of benchmarks

Some results may need to be compared with “normal” or “standard” benchmarks created by a project, an organization, or a branch of industry. Process models also include a form of benchmark. The analysis provides information about whether the benchmarks (perhaps defined as service-level agreements) are achieved and how far below or above the benchmark the results are.

6.3.6 Performing Solution Analysis

Solution analysis is used to identify potential solutions to problems and then to choose between those solutions. It helps to define concrete and feasible recommendations. The task of solution analysis typically involves a variety of individual activities, and different approaches can be adopted. The following activities are among those that may be included:

Performing cost-benefit analysis

Identifying the risks of implementing solutions

Identifying any constraints or requirements on proposing solutions

Identifying conflicts and dependencies between possible solutions

Prioritizing the possible solutions according to one or more of the preceding points

A number of approaches may be adopted to performing solution analysis:

If a model-based approach has been used, solutions may be obtained from the model. If the TPI NEXT model is used, the TPI matrix shows the earliest cluster not achieved, and the checkpoints not achieved in that cluster provide the “solution.” Care must be taken when following models in this manner; the “next practice to adopt” proposed by the model may not provide the necessary solution and could even make it worse. Adopting the recommendations suggested by models should not be a substitute for judgment exercised by experienced testing professionals. In general, any recommended solution should be evaluated against the test policy and the stated test improvement objectives. Does this recommendation contribute to achieving these objectives?

If systems thinking [Weinberg 1992] has been used for the analysis of results, the theory may now also be of assistance in helping to identify solutions. Taking the “vicious circle” example mentioned in section 6.3.5, an analysis of the system may reveal ways to break the negative reinforcing loop. In this case a relatively simple solution may be may proposed by, for example, agreeing with stakeholders about what kind of information they need in test status reports, how often they need the information, and how they would prefer the information to be presented (raw data, graphics, etc.). Using systems thinking is particularly useful if the interrelationships between different parts of a system (e.g., a test process) are complex. There may be different loops within a system, some of them balancing (i.e., stabilizing), some reinforcing (i.e., destabilizing). Either type may be identified as having a good (virtuous) or bad (vicious) impact on the system behavior. Systems thinking helps us to untangle the complexity and understand interactions more clearly. Once this is achieved, solutions can be analyzed more thoroughly and systematically. Performing solution analysis in this manner may result in well-targeted recommendations, but the analysis process also consumes more resources compared to, for example, adopting the solution(s) proposed by a model. This is also one of the benefits of using a model-based approach.

Tipping Point theory may be particularly useful if solutions are needed that will need to be adopted by a large number of people (e.g., a department or organization). Considering the “bad documentation” example mentioned in section 6.3.5, we try to identify a solution that may be relatively simple but can trigger wide-ranging improvements. For example, the solution may be to start regularly updating test plans and making this well known. With Tipping Point theory, part of the solution is not just the actual change but the way the message (“from now on, documents are updated and useful”) is delivered to the people and becomes rapidly adopted by them. This kind of “viral” adoption doesn’t just happen by itself; we need to identify key people who will spread the message to the team and motivate them to adopt the new measures. These are well respected, knowledgeable people who have good connections and like to share information and knowledge. The end result (solution) in this example may be that a whole department sees the benefits of keeping test documents up to date and adopts this not just for test plans but for all required documents. Time is needed to construct solutions developed using Tipping Point theory. Most of this time is devoted to strategic issues rather than the actual “impulse” change.

Sometimes finding a solution does not actually involve analysis as such. Consider the following two “approaches”:

Adoption of preconceived ideas is, unfortunately, commonly practiced. The underlying philosophy behind this approach is “Don’t bother me with the details; I’ve already made my mind up.” A typical preconceived solution might be the reorganization of a testing department in order to improve testing. The decision to reorganize is already made (perhaps motivated by factors other than to improve testing), and the results of the test process improvement activities are expected to confirm the preconceived solution. Since a solution analysis stage is not actually carried out, this can initially be a relatively cheap approach. Clearly, the disadvantage is that the preconceived solution may not improve testing and may even make it worse. Depending on the motivation for the preconceived solution, the success of future test process improvement initiatives may be endangered if they are perceived as biased or manipulative.

Scope for selection of a solution may be limited if specific requirements or constraints have been stated by a customer or stakeholder. If, for example, a tool is proposed as part of a solution, the actual tool selected may be constrained by commercial issues such as licenses or preferences for a particular vendor. Required standards may also impose constraints on the proposed solution. If company standards stipulate, for example, that a particular test environment may only be located onshore, then test process improvements involving testing infrastructure are constrained by these requirements. The advantage for the solution analysis task is that the scope for proposed solutions is very clearly defined. The disadvantage is that the required improvement could be suboptimal.

Clearly, when solution analysis is performed, a mix of different approaches may be applied to provide a tailored result for the specific project or organization context. The methods described in this section are often applied in the context of a workshop and/or brainstorming session.

6.3.7 Recommending Improvement Actions

At the end of the diagnosing phase, the final activity performed is to give the organization feedback about the assessment findings and recommendations. This is typically performed in two stages.

Preliminary Stage

The preliminary assessment report and/or presentation are created. The use of a standard template is recommended. (Typical contents are described in the section “The Assessment Report.”)

The assessment results and recommendations are presented to the stakeholders. Depending on the scope of the test improvement, this may be a series of presentations to individual stakeholder groups or a single presentation to all stakeholders.

Findings and recommendations are validated by the stakeholders. The presenter will require a wide range of skills (see chapter 7) to enable this validation to take place successfully. This is the stage where stakeholders (especially those that were not involved in interviews) are probably hearing findings and recommendations for the first time; these may not always be to their liking! The validation of results and recommendations is not a negotiation; the presenter must keep the focus on ensuring that there are no mistakes or misunderstandings in the preliminary report and on gaining overall recognition that the findings are valid.

Final Stage

Any agreed-upon corrections from the preliminary stage are implemented.

The final assessment report is created and issued.

Assessment report A document summarizing the assessment results; e.g., conclusions, recommendations, and findings.

The Assessment Report

An assessment report typically contains the following items:

A management summary, including scope and objectives

Planning details

The time period in which the assessment was conducted

Names and roles of people interviewed

Subjects covered in the interviews

Name of interviewers

Location of interviews

An overview of the method that was used to conduct the assessment.

A list of inputs (documents, interview reports) that were used for the assessment

The findings (“the evidence”) that were collected during the document study and the interviews

Results of the analysis (the assessment conclusions), which should address both positive aspects and aspects that need improving. The results should be related to the specified test improvement objectives.

A list of concrete recommendations, including, for each recommendation, an identifier, possible impact, objective it relates to, estimated costs, time scales, and risks of implementation

Tips

The following tips will help you write good assessment reports:

High-level, coarse-grained recommendations should be decomposed into smaller recommendations with specific goals where possible. This will make the task of planning and controlling the test improvements easier.

A frequent mistake is to pack the assessment report with large amounts of information by incorrectly assuming that lengthy reports are good reports. Avoid reporting relatively insignificant low-priority findings; stakeholders will probably not have time to read them all and they may even distract from the most important points.

It is always good to have an independent review on the report and presentation. The reviewer should preferably first study the findings and the assessment conclusions and ask the question, “Do the conclusions reflect the findings?” Finally, the reviewer should study the management summary and ask the question, “Does the management summary reflect the findings, conclusions, and recommendations stated in the report?” All of this may seem obvious to the reader, but we have learned throughout our experiences that these question reveal many mistakes in the report. Thus, reading an assessment report from the start to the end during the review is preferred to reading from back to front.

6.4 Establishing a Test Improvement Plan

Syllabus Learning Objectives

During the Establishing phase of the IDEAL improvement framework, a detailed plan is developed to implement the developed recommendations. The general goals as laid down in the initiating phase are further specified in SMART goals, and the recommendations made in the assessment report are prioritized. The following main activities are performed in the establishing phase:

Set priorities

Develop an approach

Plan the improvements

The end result of this phase is typically a test improvement plan.

Establishing (IDEAL) The phase within the IDEAL model where the specifics of how an organization will reach its destination are planned. The establishing phase consists of the following activities: set priorities, develop an approach, and plan actions.

6.4.1 Set Priorities

The recommendations from the assessment report are prioritized according to a list of criteria, each of which may be weighted according to the need for the improvement and the stakeholders involved. For example, it is futile to implement all five process areas of TMMi level 2 at once. When priorities are set, it is determined which process area(s), and which recommendations are implemented first. Several factors, such as available resources, visibility of the results, likely resistance, and contribution to organizational goals should be taken into account. At a minimum, the following factors should be considered:

Duration of improvement—A balance needs to be achieved between short-term and long-term improvements. Short-term improvements (“quick wins”) have the advantage of quickly showing return on investment and may have a strong motivational impact on the implementing team. Long-term improvements may address some of the fundamental improvements in the testing process, including cultural and organizational issues.

Implementation risk—Many improvements require a change to existing testing practices. There is a risk of failure associated with each of those improvements (see chapter 8 for more on managing change).

Link to objectives—A clear association must be made between the proposed improvement and the stated objectives of the business.

Leverage—How much impact will this improvement have on specific objectives (e.g., high, medium, low)?

6.4.2 Develop an Implementation Approach

Using the recommendations and priorities, a strategy is developed for achieving the desired situation and the desired process maturity level. The resources needed to achieve them are identified.

Technical factors to be considered include new methods, techniques, and resources. Attention must be paid to training, developing process descriptions, and possible tool selection.

Nontechnical factors to be considered include knowledge and experience, implementation approach, resistance, support, sense of urgency, and the organization’s culture, among other things.

The two principal strategies/approaches for the implementation can be distinguished as follows (see table 6–2).

Top-down

Bottom-up

Note that the results of the bottom-up strategy may be used for a subsequent top-down rollout of proven improvement measures.

Table 6–2 Implementation approaches

For the top-down approach, presentation and negotiation skills are particularly relevant in achieving consensus on objectives and recommendations (see section 7.3.). Critical success factors with this approach are managing people effectively, obtaining sponsorship, and managing expectations. The bottom-up approach may be adopted where funding for a test improvement program is limited and costs/benefits first need to be demonstrated.

6.4.3 Planning the Improvements

With the approach defined, detailed actions can be determined. Together with information taken from prior activities, these are combined into a test improvement plan including, among other things, actions, schedule, milestones, decision points, resources, responsibilities, measurements to be taken (including performance indicators), tracking mechanisms, risks, and implementation strategy.

Test improvement plan A plan for achieving organizational test process improvement objectives based on a thorough understanding of the current strengths and weaknesses of the organization’s test processes and test process assets [After Chrissis, Konrad, and Shrum 2004].

The primary planning activities are listed here:

Prioritizing and combining groups of related improvements into packages (step-by-step improvement)

Linking the improvement issues to the recommendations and required performance indicators for achievement

Deciding on an approach for process improvement

Scheduling the changes

Establishing groups or teams to implement the improvements

Assigning tasks

Documenting the above-mentioned points in a test improvement plan

The test improvement plan is typically an action plan with a timeframe of one year or less focused on detailed planning of test improvement activities.

If test process improvements are taking place across the entire organization, perhaps in parallel with other process improvement initiatives, a longer-term strategic action plan may be created. This brings together the organization’s process improvement strategy into a cohesive plan covering the next three to five years.

6.5 Acting to Implement Improvement

Syllabus Learning Objectives

This phase is about concrete activities. This is where the action is! The recommendations must be specified in detail and must be implemented according to the test improvement plan. Obviously this phase consumes the most effort, because while developing the solution takes up about 30 percent effort, implementing the solution takes up about 70 percent [Cannegieter 2003].

Acting (IDEAL) The phase within the IDEAL model where the improvements are developed, put into practice, and deployed across the organization. The acting phase consists of the following activities: create solutions, pilot/test solution, refine solution, and implement solution.

The following main activities are performed in the Acting phase:

Create solutions

Pilot/test solutions

Refine solutions

Implement solutions

Create Solution

The acting phase begins with developing solutions to address the broadly outlined problems. These solutions should contribute to achieving the desired situation. The solutions can include processes, templates, tools, knowledge, skills (training), information, and support. The solutions, which can be quite complex, are often developed by improvement teams (see section 7.1.1). An approach using improvement teams that has proven to be successful is the improvement team relay [Zandhuis 2009]. In an improvement team relay, a number of succeeding improvement teams develop and implement (parts of) the solution in a short time. Some advantages of the improvement team relay include reducing the lead time that would be required if only one overall improvement team was used, achieving results quickly, and allowing for more exact guidance. Every improvement team needs to have a clear goal and be given a clear assignment by the management. As many employees as possible need to be involved in actually working out the solutions; an external consultant can provide guidance and content input.

Tip: Process Documentation Structure

In an early phase of the improvement project a document overview should be established. In one test improvement project one of the authors was involved in, the process documents were initially delivered separately by different teams for their particular improvement action. When all individual documents came together, it was obvious that an overall architecture, tying all parts together into one bigger picture from the start to the end of the test process, was missing. Once this overall architecture had been developed, it became the starting point of a structure or hierarchy for all the other test process documents in the quality system. By thinking about a thorough structure for your process documents at the beginning of your project, you can save a lot of time on reworking and restructuring later.

6.5.1 Selecting and Executing a Pilot

Pilot/Test Solution

Following Tom Gilb’s advice, “If you don’t know what you are doing, don’t do it on a large scale,” the created solution first needs to be tested in one or more pilot projects. Sometimes only practical experience can show the exact impact of a solution. In pilots such as this, usually one or more projects are appointed in which the improvements are implemented and evaluated before they are adopted by additional projects.

Piloting a proposed improvement is an effective way of reducing the risk of implementation failure, gaining experience and building support. This is especially important where those improvements involve major changes to working practices or place a heavy demand on resources. Selection of a pilot should balance the following factors:

Realism. For example, is the pilot representative of the “real world”?

Scalability of solution. For example, can the results from the pilot be used in all contexts?

Impact on current projects. Particular care is required if existing practices are to be replaced by the improved practices during the pilot.

Risk of failure, that is, the risk that the pilot itself may be a failure.

Refine Solution

With the use of the results of the test or pilot, the solution can be optimized. Several iterations of the test-optimizing process may be necessary to reach a satisfactory solution that will work for all projects. A solution should be workable; waiting for a “perfect” solution may unnecessarily delay the implementation.

If the analysis of lessons learned from performing the pilot yields positive results, the decision may be made to roll out the improvements to other parts of the organization and/or other projects.

6.5.2 Manage and Control the Implementation

Once the solutions are deemed workable, they can be implemented throughout the (test) organization. This is usually the most radical activity, and it can provoke much resistance. Several implementation approaches can be used:

Big bang: All the organizational changes are implemented at the same time.

One project at a time. In every project, the change is implemented at a set moment in time.

Just in time: The change is implemented when the process is executed.

Basically the implementation of the test improvement plan is performed and monitored and progress toward achieving improvement goals is reported. The measures, metrics, and indicators specified in the test improvement plan are collected and compared to the established objectives.

No one implementation approach is better than the other; the approach should be chosen based on the nature of the improvement and organizational circumstances. For a major change, implementation may require substantial time, resources, effort, and attention from the management.

The rollout of improvements follows a defined process, especially where an entire organization is affected by the change. Chapter 8 is devoted entirely to this critical aspect of test process improvement.

6.6 Learning from the Improvement Program

Syllabus Learning Objectives

The Learning phase completes the improvement cycle. One of the goals of the IDEAL model is to continuously improve the ability to implement change. In the learning phase, the entire IDEAL experience is reviewed to determine what was accomplished, whether the intended goals were achieved, and how the organization can implement change more effectively and efficiently.

During and after implementation of the test improvement plan, project retrospectives are performed with the stakeholders. The organization or person that has implemented the improvement usually manages the retrospectives, which are typically performed as workshops.

The following activities are performed in the learning phase:

Analyze and validate

Propose future actions

Learning (IDEAL) The phase within the IDEAL model where one learns from experiences and improves one’s ability to adopt new processes and technologies in the future. The learning phase consists of the following activities: analyze and validate, and propose future actions.

Analyze and Validate

This activity answers several questions:

How did the improvement program go?

What has been accomplished; have the initial goals been achieved?

What worked well?

What could be done more efficiently or effectively?

Using these questions for guidance, lessons learned are collected, analyzed, summarized, and documented.

Propose Future Actions

Based upon the previous activity, recommendations are formulated that are intended to improve future improvement programs. These recommendations are provided to upper management for consideration, which may include performing a new improvement cycle.

6.7 Exercises

The following multiple-choice questions will give you some feedback on your understanding of this chapter. Appendix F lists the answers.

6-1 Which phase of the IDEAL framework relates to the following goal: “Determining the organization’s current state as opposed to what it wants to achieve”?

A: Initiating

B: Establishing

C: Acting

D: Diagnosing

6-2 Which is the correct order for the phases of a test improvement cycle?

A: Initiating, establishing, diagnosing, acting, learning

B: Initiating, diagnosing, establishing, acting, learning

C: Initiating, acting, establishing, diagnosing, learning

D: Initiating, diagnosing, acting, establishing, learning

6-3 Which is of the following is not a main purpose of the document study to be carried out during the assessment preparation phase?

A: Gain insights into the current testing process prior to interviewing those involved.

B: Gather and verify information.

C: Prepare particular questions for the interview.

D: Perform formal elements of the assessment that do not require discussion. For example, the documents may be checked to ensure completeness and conformity to standards.

6-4 During an assessment, when will the initial feedback be provided?

A: After analyzing the assessment results

B: When presenting the assessment report

C: At the end of the diagnosing phase

D: At the end of the interviews

6-5 When assigning priorities to the recommendation from an assessment, which of the following factors needs to be taken into account?

A: Implementation risk

B: Added value for objectives

C: Duration of the improvement action

D: All of the above

6-6 Which of the following is an activity that will typically be performed as part of the learning phase of a test improvement cycle?

A: Analyze and validate

B: Plan actions

C: Develop recommendations

D: Refine solutions

6-7 Which of the following strategies/approaches for an implementation best matches the following characteristics?

Less formal; often an analytical approach and/or prototyping in order to gain experience and build support can be distinguished

Typically covers no more than one or two projects

A: Big bang

B: Just in time

C: Bottom-up

D: Top-down

6-8 Which of the following activities will typically be performed as part of the acting phase of a test improvement cycle?

A: Selecting and executing a pilot

B: Developing an implementation approach

C: Planning the Improvement

D: Analyzing and validating the solution

6-9 Which of the following would typically be part of a test policy?

A: Acceptance criteria

B: Main tasks for each test level

C: Generic risks for the products to be developed

D: The objectives and added value of testing

6-10 Which factor should be taken into account when selecting a pilot?

A: Is the pilot project lightweight, therefore making the chances of a successful pilot better?

B: Can the results from the pilot be used in all contexts; that is, is the solution scalable?

C: If the project is already late, would it be OK to delay it further due to a pilot?

D: Is the project a high-profile project, thus already having the required level of management attention?