Static Test - Software Testing Foundations: A Study Guide for the Certified Tester Exam (2014)

Software Testing Foundations: A Study Guide for the Certified Tester Exam (2014)

Chapter 4. Static Test

Static investigations like reviews and tool-supported analysis of code and documents can be used very successfully for improving quality. This chapter presents the possibilities and techniques.

An often-underestimated examination method is the so-called static test, often named static analysis. Opposite to dynamic testing (see chapter 5), the test object is not provided with test data and executed but rather analyzed. This can be done using one or more persons for an intensive investigation or through the use of tools. Such an investigation can be used for all documents relevant for software development and maintenance. Tool-supported static analysis is only possible for documents with a formal structure.

The goal of examination is to find defects and deviations from the existing specifications, standards to comply with, or even the project plan. An additional benefit of the results of these examinations is optimizing the development process. The basic idea is defect prevention: defects and deviations should be recognized as early as possible before they have any effect in the further development process where they would result in expensive rework.

4.1 Structured Group Evaluations

4.1.1 Foundations

Systematic use of the human ability to think and analyze

Reviews apply the human analytical capabilities to check and evaluate complex issues. Intensive reading and trying to understand the examined documents is the key.

There are different techniques for checking documents. They differ regarding the intensity, formality, necessary resources (staff and time), and goals.

In the following sections, the different techniques are explained in more detail. Unfortunately, there is no uniform terminology concerning static analysis techniques. The terms used in this chapter are similar to the terms in the ISTQB syllabus and [IEEE 1028] (see the glossary in the appendix). Detailed descriptions can be found in [Freedman 90] and [Gilb 96].

4.1.2 Reviews

Review is a common generic term for all the different static analysis techniques people perform as well as the term for a specific document examination technique.

Another term, often used with the same meaning, is →inspection. However, inspection is usually defined as a special, formal review using data collection and special rules [Fagan 76], [IEEE 1028], [Gilb 96]. All documents can be subjected to a review or an inspection, such as, for example, contracts, requirements definitions, design specifications, program code, test plans, and manuals. Often, reviews provide the only possibility to check the semantics of a document. Reviews rely on the colleagues of the author to provide mutual feedback. Because of this, they are also called peer reviews.

A means for quality assurance

Reviews are an efficient means to assure the quality of the examined documents. Ideally, they should be performed as soon as possible after a document is completed to find mistakes and inconsistencies early. The verifying examinations at the end of a phase in the general V-model normally use reviews (so-called phase exit reviews). Eliminating defects and inconsistencies leads to improved document quality and positively influences the whole development process because development is continued with documents that have fewer or even no defects.

Positive effects

In addition to defect reduction, reviews have further positive effects:

§ Cheaper defect elimination. If defects are found and eliminated early, productivity in development increases because fewer resources are needed for defect identification and elimination later. These resources can instead be used for development.

§ Shortened development time.

§ If defects are recognized and corrected early, costs and time needed for executing the dynamic tests (see chapter 5) decrease because there are fewer defects in the test object.

§ Because of the smaller number of defects, cost reduction can be expected during the whole product life. For example, a review may detect and clarify inconsistent and imprecise customer requests in the requirements. Foreseeable change requests after installation of the software system can thus be avoided.

§ During operation of the system, a reduced failure rate can be expected.

§ As the examinations are done using a team of people, reviews lead to mutual learning. People improve their working methods, and reviews will thus lead to enhanced quality of later products.

§ Because several persons are involved in a review, a clear and understandable description of the facts is required. Often the necessity to formulate a clear document lets the author find forgotten issues.

§ The whole team feels responsible for the quality of the examined object. The group will gain a common understanding of it.

Potential problem

The following problem can arise: In a badly moderated review session, the author may get into a psychologically difficult situation, feeling that he as a person and not the document is subject to critical scrutiny. Motivation to subject documents to a review will thus be destroyed. Concretely expressing the review objective, which is improving the document, may be helpful. One book [Freedman 90] extensively discusses how to solve problems with reviews.

Reviews costs and savings

The costs caused by reviews are estimated to be 10–15% of the development budget [Gilb 96, pg. 27]. The costs include the activities of the review process itself, analyzing the review results, and the effort put toward implementing them for process improvement. Savings are estimated to be about 14–25% [Bush 90]. The extra effort for the reviews themselves is included in this calculation.

If reviews are systematically used and efficiently run, more than 70% of the defects in a document can be found and repaired before they are unknowingly inherited by the next work steps [Gilb 96]. Considering that the costs for defect removal substantially increase in later development steps, it is plausible that defect cost in development is reduced by 75% and more.

Hint

§ Documents with a formal structure should be analyzed using a (static analysis) tool that checks this structure before the review. The tool can examine many aspects and can detect defects or deviations that do not need to be checked in a review (see section 4.2)

Important success factors

The following factors are decisive for success when using reviews (as suggested by [IEEE 1028]):

§ Every review has a clear goal, which is formulated beforehand.

§ The “right” people are chosen as review participants based on the review objective as well as on their subject knowledge and skills.

4.1.3 The General Process

The term review describes a whole group of static examinations. Section 4.1.5 describes the different types of reviews. The process underlying all reviews is briefly described here in accordance with the IEEE Standard for Software Reviews [IEEE 1028].

A review requires six work steps: planning, kick-off, individual preparation, review meeting, rework, and follow-up.

Planning

Reviews need planning

Early, during overall planning, management must decide which documents in the software development process shall be subject to which review technique. The estimated effort must be included in the project plans. Several analyses show optimal checking time for reviewing documents and code [Gilb 96]. During planning of the individual review, the review leader selects technically competent staff and assembles a review team. In cooperation with the author of the document to be reviewed, she makes sure that the document is in a reviewable state, i.e., it is complete enough and reasonably finished. In formal reviews, entry criteria (and the corresponding exit criteria) may be set. A review should continue only after any available entry criteria has been checked.

Different perspectives increase the effect

A review is, in most cases, more successful when the examined document is read from different viewpoints or when each person checks only particular aspects. The viewpoints or aspects to be used should be determined during review planning. A review might not involve the whole document. Parts of the document in which defects constitute a high risk could be selected. A document may also be sampled only to make a conclusion about the general quality of the document.

If a kick-off meeting is necessary, the place and time must be agreed upon.

Kick-Off

The kick-off (or overview) serves to provide those involved in the review with all of the necessary information. This can happen through a written invitation or a meeting when the review team is organized. The purpose is sharing information about the document to be reviewed (the review object) and the significance and the objective of the planned review. If the people involved are not familiar with the domain or application area of the review object, then a short introduction to the material may be arranged, and a description of how it fits into the application or environment may be provided.

Higher-level documents are necessary

In addition to the review object, those involved must have access to other documents. These include the documents that help to decide if a particular statement is wrong or correct. The review is done against these documents (e.g., requirements specification, design, guidelines, or standards). Such documents are also called base documents or baselines. Furthermore, review criteria (for example, checklists) are very useful for supporting a structured process.

For more formal reviews, the entry criteria might be checked. If entry criteria are not met, the review should be canceled, saving the organization time that would otherwise be wasted reviewing material that may be “immature,” i.e., not good enough.

Individual Preparation

Intensive study of the review object

The members of the review team must prepare individually for the review meeting. A successful review meeting is only possible with adequate preparation.

The reviewers intensively study the review object and check it against the documents given as a basis for it as well as against their own experience. They note deficiencies (even any potential defects), questions, or comments.

Review Meeting

A review leader or →moderator leads the review meeting. Moderator and participants should behave diplomatically (not be aggressive with each other) and contribute to the review in the best possible way.

The review leader must ensure that all experts will be able to express their opinion knowing that the product will be evaluated and not the author. Conflicts should be prevented. If this is not possible, a solution for the situation should be found.

Usually, the review meeting has a fixed time limit. The objective is to decide if the review object has met the requirements and complies with the standards and to find defects. The result is a recommendation to accept, repair, or rewrite the document. All the reviewers should agree upon the findings and the overall result.

Rules for review meetings

Here are some general rules for a review meeting:1

1. The review meeting is limited to two hours. If necessary, another meeting is called, but it should not take place on the same day.

2. The moderator has the right to cancel or stop a meeting if one or more experts (reviewers) don’t appear or if they are not sufficiently prepared.

3. The document (the review object) is subject to discussion, not the author:

o The reviewers have to watch their expressions and their way of expressing themselves.

o The author should not defend himself or the document. (That means, the author should not be attacked or forced into a defensive position. However, justification or explanation of the author’s decisions is often seen as legitimate and helpful.)

4. The moderator should not also be a reviewer at the same time.

5. General style questions (outside the guidelines) shall not be discussed.

6. Solutions and discussing them isn’t a task of the review team.

7. Every reviewer must have the opportunity to adequately present his or her issues.

8. The protocol must describe the consensus of the reviewers.

9. Issues must not be written as commands to the author (additional concrete suggestions for improvement or correction are sometimes considered useful and sensible for quality improvement).

10.The issues should be weighted2 as follows:

o Critical defect (the review object is not suitable for its purpose, the defect must be corrected before the object is approved)

o Major defect (the usability of the review object is affected, the defect must be corrected before the approval)

o Minor defect (small deviation, for example, spelling error or bad expression, hardly affects the usage)

o Good (flawless, this area should not be changed during rework).

11.The review team shall make a recommendation about the acceptance of the review object (see follow-up):

o Accept (without changes)

o Accept (with changes, no further review)

o Do not accept (further review or other checking measures are necessary)

12.Finally, all the session participants should sign the protocol

Protocol and summary of results

The protocol contains a list of the issues/findings that were discussed during the meeting. An additional review summary report should collect all important data about the review itself, i.e., the review object, the people involved, their roles (see section 4.1.4), a short summary of the most important issues, and the result of the review with the recommendation of the reviewers. In a more formal review, the fulfillment of formal exit criteria may be documented. If there was no physical meeting and, for example, electronic communication was used instead, there should definitely be a protocol.

Rework

The manager decides whether to follow the recommendation or do something else. A different decision is, however, the sole responsibility of the manager. Usually, the author will eliminate the defects on the basis of the review results and rework the document. More formal reviews additionally require updating the defect status of every single found defect.

Follow-Up

The proper correction of defects must be followed up, usually by the manager, moderator, or someone especially assigned this responsibility.

Second review

If the result of the first review was not acceptable, another review should be scheduled. The process described here can be rerun, but usually it is done in an abbreviated manner, checking only changed areas.

The review meetings and their results should then be thoroughly evaluated to improve the review process, to adapt the used guidelines and checklists to the specific conditions, and to keep them up-to-date. To achieve this, it is necessary to collect and evaluate measurement data.

Find and fix deficiencies in the software development process

Recurring, or frequently occurring, defect types point to deficiencies in the software development process or lack of technical knowledge of the people involved. Necessary improvements of the development process should be planned and implemented. Such defect types should be included in the checklists. Training must compensate for lack of technical knowledge.

For more formal reviews, the final activity is checking the exit criteria. If they are met, the review is finished. Otherwise, it must be determined whether rework can be done or if the whole review was unsuccessful.

4.1.4 Roles and Responsibilities

The description of the general approach included some information on roles and responsibilities. This section presents the people involved and their tasks.

Manager

The manager selects the objects to be reviewed, assigns the necessary resources, and selects the review team.

Representatives of the management level should not participate in review meetings because management might evaluate the qualifications of the author and not the document. This would inhibit a free discussion among the review participants. Another reason is that the manager often lacks the necessary detailed understanding of technical documents. In a review, the technical content is checked, and thus the manager would not be able to add valuable comments. Management reviews of project plans and the like are a different thing. In this case, knowledge of management principles is necessary.

Moderator

The moderator is responsible for executing the review. Planning, preparation, execution, rework, and follow-up should be done in such a way that the review objectives are achieved.

The moderator is responsible for collecting review data and issuing the review report.

This role is crucial for the success of the review. First and foremost, a moderator must be a good meeting leader, leading the meeting efficiently and in a diplomatic way. A moderator must be able to stop unnecessary discussions without offending the participants, to mediate when there are conflicting points of view, and be able to see “between the lines.” A moderator must be neutral and must not state his own opinion about the review object.

Author

The author is the creator of the document that is the subject of a review. If several people have been involved in the creation, one person should be appointed to be responsible; this person assumes the role of the author. The author is responsible for the review object meeting its review entry criteria (i.e., that the document is reviewable) and for performing any rework required for meeting the review exit criteria.

It is important that the author does not interpret the issues raised on the document as personal criticism. The author must understand that a review is done only to help improve the quality of the product.

Reviewer

The reviewers, sometimes also called inspectors, are several (usually a maximum of five) technical experts that participate in the review meeting after necessary individual preparation.

They identify and describe problems in the review object. They should represent different viewpoints (for example, sponsor, requirements, design, code, safety, test). Only those viewpoints pertinent to the review of the product should be considered.

Some reviewers should be assigned specific review topics to ensure effective coverage. For example, one reviewer might focus on conformance with a specific standard, another on syntax. The manager should assign these roles when planning the review.

The reviewers should also label the good parts in the document. Insufficient or deficient parts of the review object must be labeled accordingly, and the deficiencies must be documented for the author in such a way that they can be corrected.

Recorder

The recorder (or scribe) shall document the issues (problems, action items, decisions, and recommendations) found by the review team.

The recorder must be able to record in a short and precise way, correctly capturing the essence of the discussion. This may not be easy because contributions are often not clearly or well expressed. Pragmatic reasons may make it meaningful to let the author be recorder. The author knows exactly how precisely and how detailed the contributions of the reviewers need to be recorded in order to have enough information for rework.

Possible difficulties

Reasons for less successful reviews

Reviews may fail to achieve their objectives due to several causes:

§ The required persons are not available or do not have the required qualifications or technical skills. This may be solved by training or by using qualified staff from consulting companies. This is especially true for the moderator, because he must have more psychological than technical skills.

§ Inaccurate estimates during resource planning by management may result in time pressure, which then causes unsatisfactory review results. Sometimes, a less costly review type can bring relief.

§ If reviews fail due to lack of preparation, this is mostly because the wrong reviewers were chosen. If a reviewer does not realize the importance of the review and its great effect on quality improvement, then figures must be shown that prove the productive benefit of reviews. Other reasons for review failure may be lack of time and lack of motivation.

§ A review can also fail because of missing or insufficient documentation. Prior to the review, it must be verified that all the needed documents exist and that they are sufficiently detailed. Only when this is the case should a review be performed.

§ The review process cannot be successful if there is lack of management support because the necessary resources will not be provided and the results will not be used for process improvement. Unfortunately, this is often the case.

Detailed advice for solving these problems is described in [Freedman 90].

4.1.5 Types of Reviews

Two main groups of reviews can be distinguished depending on the review object to be examined:

§ Reviews pertaining to products or intermediate products that have been created during the development process

§ Reviews that analyze the project itself or the development process

Excursion

Reviews in the second group are called →management reviews3 [IEEE 1028] or project reviews. Their objective is to analyze the project itself or the development process. For example, such a review determines if plans and rules are followed, if the necessary work tasks are executed, or the effectiveness of process improvements or changes.

The project as a whole and determining its current state are the objects of such a review. The state of the project is evaluated with respect to technical, economic, time, and management aspects.

Management reviews are often performed when reaching a milestone in the project, when completing a main phase in the software development process, or as a “postmortem” analysis to learn from the finished project.

In the following sections, the first group of reviews is described in more detail. We can distinguish between the following review types: →walkthrough, inspection, →technical review, and →informal review. In the descriptions, the focus is on the main differences between the particular review type and the basic review process (see section 4.1.3).

Walkthrough

A walkthrough4 is a manual, informal review method with the purpose of finding defects, ambiguities, and problems in written documents. The author presents the document to the reviewers in a review meeting.

Educating an audience regarding a software product is mentioned in [IEEE 1028] as a further purpose of walkthroughs. Further objectives of walkthroughs are to improve the product, to discuss alternative implementations, and to evaluate conformance to standards and specifications.

The main emphasis of a walkthrough is the review meeting (without a time limit). There is less focus on preparation compared to the other types of reviews; it can even be omitted sometimes.5

Discussion of typical usage situations

In most cases, typical usage situations, also called scenarios, will be discussed. Test cases may even be “played through.” The reviewers try to find possible errors and defects by spontaneously asking questions.

Suitable for small development teams

The technique is useful for small teams of up to five persons. It does not require a lot of resources because preparation and follow-up are minor or sometimes not even required. The walkthrough is useful for checking “noncritical” documents.

The author chairs the meeting and therefore has a great amount of influence. This can have a detrimental effect on the result if the author impedes an intensive discussion of the critical parts of the review object.

The author is responsible for follow-up; there is no more checking.

Before the meeting the reviewers prepare, the results are written in a protocol, and someone other than the author records the findings. In practice there is a wide variation from informal to formal walkthroughs.

Objectives

The main objectives of a walkthrough are mutual learning, development of an understanding of the review object, and error detection.

Inspection

Formal process

The inspection is the most formal review. It follows a formal, prescribed process. Every person involved, usually people who work directly with the author, has a defined role. Rules define the process. The reviewers use checklists containing criteria for checking the different aspects.

The goals are finding unclear items and possible defects, measuring review object quality, and improving the quality of the inspection process and the development process. The concrete objectives of each individual inspection are determined during planning. The inspectors (reviewers) prepare for only a specific number of aspects that will be examined. Before the inspection begins, the inspection object is formally checked with respect to entry criteria and reviewability. The inspectors prepare themselves using procedures or standards and checklists.

Traditionally, this method of reviewing has been called design inspection or code or software inspection. The name points to the documents that are subject to the inspection (see [Fagan 76]). However, inspections can be used for any document in which formal evaluation criteria exist.

Inspection meeting

A moderator leads the meeting. The inspection meeting follows this agenda:

§ The moderator first presents the participants and their roles as well as a short introduction to the topic of the inspection object.

§ The moderator asks the participants if they are adequately prepared. In addition, the moderator might ask how much time the reviewer used to prepare and how many and how severe were the issues found.

§ The group may review the checklists chosen for the inspection in order to make sure everyone is well prepared for the meeting.

§ Issues of a general nature concerning the whole inspection object are discussed first and written into the protocol.

§ A reviewer presents6 the contents of the inspection object quickly and logically. If it’s considered useful, passages can also be read aloud.

§ The reviewers ask questions during this procedure, and the selected aspects of the inspection are thoroughly discussed. The author answers questions. The moderator makes sure that a list of issues is written. If author and reviewer disagree about an issue, a decision is made at the end of the meeting.

§ The moderator must intervene if the discussion is getting out of control. The moderator also makes sure the meeting covers all aspects to be evaluated as well as the whole document. The moderator makes sure the recorder writes down all the issues and ambiguities that are detected.

§ At the end of the meeting, all recorded items are reviewed for completeness.

§ Discussions are conducted to resolve disagreements, for example, whether or not something can be classified a defect. If no resolution is reached, this is written in the protocol. There should be no discussion on how to solve the issues. Any discussion should be limited in time.

§ Finally, the reviewers judge the inspection object as a whole.

§ They decide if it must be reworked or not. In inspections, follow-up and reinspection are formally regulated.

Additional assessment of the development and inspection process

In an inspection, data are also collected for general quality assessment of the development process and the inspection process. Therefore, an inspection also serves to optimize the development process, in addition to assessing the inspected documents. The collected data are analyzed in order to find causes for weaknesses in the development process. After process improvement, comparing the collected data before the change to the current data checks the improvement effect.

Objective

The main objective of inspection is defect detection or, more precisely, the detection of defects causes and defects.

Technical Review

Does the review object fulfill its purpose?

In a technical review, the focus is compliance of the document with the specification, fitness for its intended purpose, and compliance to standards. During preparation, the reviewers check the review object with respect to the specified review criteria.

Technical experts as reviewers

The reviewers must be technically qualified. Some of them should not be project participants in order to avoid “project blindness.” Management does not participate. Basis for the review is only the “official” specification and the specified criteria for the review. The reviewers write down their comments and pass them to the moderator before the review meeting.7 The moderator (who ideally is properly trained) sorts these findings based on their presumed importance. During the review meeting, only selected remarks are discussed.

High preparation effort

Most of the effort is in preparation. The author does not normally attend the meeting. During the meeting, the recorder notes all the issues and prepares the final documentation of the results.

The review result must be approved unanimously by all involved reviewers and signed by everyone. Disagreement should be noted in the protocol. It is not the job of the review participants to decide on the consequences of the result; that is the responsibility of management. If the review is highly formalized, entry and exit criteria of the individual review steps may also be defined.

In practice, very different versions of the technical review are found, from a very informal to a strictly defined, formal process.

Objective

Discussion is expressly requested during a technical review. Alternative approaches should be considered and decisions made. The specialists may solve the technical issues. The conformity of the review object with its specifications and applicable standards can be assessed. Technical reviews can, of course, reveal errors and defects.

Informal Review

The informal review is a light version of a review. However, it more or less follows the general procedure for reviews (see section 4.1.3) in a simplified way. In most cases, the author initiates an informal review. Planning is restricted to choosing reviewers and asking them to deliver their remarks at a certain point in time. Often, there is no meeting or exchange of the findings. In such cases, the review is just a simple author-reader-cycle. The informal review is a kind of cross reading by one or more colleagues. The results need not be explicitly documented; a list of remarks or the revised document is in most cases enough. Pair programming, buddy testing, code swapping, and the like are types of informal review. The informal review is very common and highly accepted due to the minimal effort required.

Objective

An informal review involves relatively little effort and low costs. Discussion and exchange of information among colleagues are welcome “side effects” of the process.

Selection Criteria

Selecting the type of review

The type of review that should be used depends very much on how thorough the review needs to be and the effort that can be spent. It also depends on the project environment; we cannot give specific recommendations. The decision about what type of review is appropriate must be made on a case-by-case basis. Here are some questions and criteria that should help:

§ The form in which the results of the review should be presented can help select the review type. Is detailed documentation necessary, or is it good enough to present the results informally?

§ Will it be difficult or easy to find a date and time for the review? It can be difficult to bring together five or seven technical experts for one or more meetings.

§ Is it necessary to have technical knowledge from different disciplines?

§ What level (how deep) of technical knowledge is required for the review object? How much time will the reviewers need?

§ Is the preparation effort appropriate with respect to the benefit of the review (the expected result)?

§ How formally written is the review object? Is it possible to perform tool-supported analyses?

§ How much management support is available? Will management curtail reviews when the work is done under time pressure?

Testers as reviewers

It makes sense to use testers as reviewers. The reviewed documents are usually used as the test basis to design test cases. Testers know the documents early and they can design test cases early. By looking at documents from a testing point of view, testers may check new quality aspects, such as testability.

Notes

As we said in the beginning of the chapter, there are no uniform descriptions of the individual types of review. There is no clear boundary between the different review types, and the same terms are used with different meanings.

Company-specific reviews

Generally, it can be said that the types of reviews are very much determined by the organization that uses them. Reviews are tailored to the specific needs and requirements of a project. This has a positive influence on their efficiency.

A cooperative collaboration between the people involved in software development can be considered beneficial to quality. If people examine each other’s work results, defects and ambiguities can be revealed. From this point of view, pair programming, as suggested in →Extreme Programming, can be regarded as a permanent “two-person-review” [Beck 00].

With distributed project teams, it might be hard to organize review meetings. These days, reviews can be in the form of structured discussion by Internet, videoconferencing, telephone conference calls, etc.

Success Factors

The following factors are crucial for review success and must be considered:

§ Reviews help improve the examined documents. Detecting issues, such as unclear points and deviations, is a wanted and required effect. The issues must be formulated in a neutral and objective way.

§ Human and psychological factors have a strong influence in a review. A review must be conducted in an atmosphere of trust. The participants must be sure that the outcome will not be used to evaluate them (for example, as a basis of their next job assessment). It’s important that the author of the reviewed document has a positive experience.

§ Testers should be used as reviewers. They contribute to the review by finding (testing) issues. When they participate in reviews, testers learn about the product, which enables them to prepare tests earlier and in a better way.

§ The type and level of the examined document, and the state of knowledge of the participating people, should be considered when choosing the type of review to use.

§ Checklists and guidelines should be used to help in detecting issues during reviews.

§ Training is necessary, especially for more formal types of reviews, such as inspections.

§ Management can support a good review process by allocating enough resources (time and personnel) for document reviews in the software development process.

§ Continuous learning from executed reviews improves the review process and thus is important.

4.2 Static Analysis

Analysis without executing the program

The objective of static analysis is, as with reviews, to reveal defects or defect-prone parts in a document. However, in static analysis, tools do the analysis. For example, even spell checkers can be regarded as a form of →static analyzers because they find mistakes in documents and therefore contribute to quality improvement.

The term static analysis points to the fact that this form of checking does not involve an execution of the checked objects (of a program). An additional objective is to derive measurements, or metrics, in order to measure and prove the quality of the object.

Formal documents

The document to be analyzed must follow a certain formal structure in order to be checked by a tool. Static analysis makes sense only with the support of tools. Formal documents can note, for example, the technical requirements, the software architecture, or the software design. An example is the modeling of class diagrams in UML.8 Generated outputs in HTML9 or XML10 can also be subjected to tool-supported static analysis. Formal models developed during the design phases can also be analyzed and inconsistencies can be detected. Unfortunately, in practice, the program code is often the one and only formal document in software development that can be subjected to static analysis.

Developers typically use static analysis tools before or during component or integration testing to check if guidelines or programming conventions are adhered to. During integration testing, adherence to interface guidelines is analyzed.

Analysis tools often produce a long list of warnings and comments. In order to effectively and efficiently use the tools, the mass of generated information must be handled intelligently; for example, by configuring the tool. Otherwise, the tools might be avoided.

Static analysis and reviews

Static analysis and reviews are closely related. If a static analysis is performed before the review, a number of defects and deviations can be found and the number of the aspects to be checked in the review clearly decreases. Due to the fact that static analysis is tool supported, there is much less effort involved than in a review.

Hint

§ If documents are formal enough to allow tool-supported static analysis, then it should definitely be performed before the document reviews because faults and deviations can be detected conveniently and cheaply and the reviews can be shortened.

§ Generally, static analysis should be used even if no review is planned. Each time a discrepancy is located and removed, the quality of the document increases.

Not all defects can be found using static testing, though. Some defects become apparent only when the program is executed (that means at runtime) and cannot be recognized before. For example, if the value of the denominator in a division is stored in a variable, that variable can be assigned the value zero. This leads to a failure at runtime. In static analysis, this defect cannot easily be found, except for when the variable is assigned the value zero by a constant having zero as its value.

All possible paths through the operations may be analyzed, and the operation can be flagged as potentially dangerous. On the other hand, some inconsistencies and defect-prone areas in a program are difficult to find by dynamic testing. Detecting violation of programming standards or use of forbidden error-prone program constructs is possible only with static analysis (or reviews).

The compiler is an analysis tool

All compilers carry out a static analysis of the program text by checking that the correct syntax of the programming language is used. Most compilers provide additional information, which can be derived by static analysis (see section 4.2.1). In addition to compilers, there are other tools that are so-called analyzers. These are used for performing special analyses or groups of analyses.

The following defects and dangerous constructions can be detected by static analysis:

§ Syntax violations

§ Deviations from conventions and standards

§ →Control flow anomalies

§ →Data flow anomalies

Finding security problems

Static analysis can be used to detect security problems. Many security holes occur because certain error-prone program constructs are used or necessary checks are not done. Examples are lack of buffer overflow protection and failing to check that input data may be out of bounds. Tools can find such deficiencies because they often search and analyze certain patterns.

4.2.1 The Compiler as a Static Analysis Tool

Violation of the programming language syntax is detected by static analysis and reported as a fault or warning. Many compilers also generate further information and perform other checks:

§ Generating a cross-reference list of the different program elements (e.g., variables, functions)

§ Checking for correct data type usage by data and variables in programming languages with strict typing

§ Detecting undeclared variables

§ Detecting code that is not reachable (so-called →dead code)

§ Detecting overflow or underflow of field boundaries (with static addressing)

§ Checking interface consistency

§ Detecting the use of all labels as jump start or jump target

The information is usually provided in the form of lists. A result reported as “suspicious” by the tool is not always a fault. Therefore, further investigation is necessary.

4.2.2 Examination of Compliance to Conventions and Standards

Compliance to conventions and standards can also be checked with tools. For example, tools can be used to check if a program follows programming regulations and standards. This way of checking takes little time and almost no personnel resources. In any case, only guidelines that can be verified by tools should be accepted in a project. Other regulations usually prove to be bureaucratic waste anyway. Furthermore, there often is an additional advantage: if the programmers know that the program code is checked for compliance to the programming guidelines, their willingnessto work according to the guidelines is much higher than without an automatic test.

4.2.3 Execution of Data Flow Analysis

Checking the use of data

→Data flow analysis is another means to reveal defects. Here, the usage of data on →paths through the program code is checked. It is not always possible to decide if an issue is a defect. Instead, we speak of →anomalies, or data flow anomalies. An anomaly is an inconsistency that can lead to failure but does not necessarily do so. An anomaly may be flagged as a risk.

An example of a data flow anomaly is code that reads (uses) variables without previous initialization or code that doesn’t use the value of a variable at all. The analysis checks the usage of every single variable. The following three types of usage or states of variables are distinguished:

§ Defined (d): The variable is assigned a value.

§ Referenced (r): The value of the variable is read and/or used.

§ Undefined (u): The variable has no defined value.

Data flow anomalies

We can distinguish three types of data flow anomalies:

§ ur-anomaly: An undefined value (u) of a variable is read on a program path (r).

§ du-anomaly: The variable is assigned a value (d) that becomes invalid/undefined (u) without having been used in the meantime.

§ dd-anomaly: The variable receives a value for the second time (d) and the first value had not been used (d).

Example of anomalies

We will use the following example (in C++) to explain the different anomalies. The following function is supposed to exchange the integer values of the parameters Max and Min with the help of the variable Help, if the value of the variable Min is greater that the value of the variableMax:

void exchange (int& Min, int& Max) {
int Help;
if (Min > Max) {
Max = Help;
Max = Min;
Help = Min;
}
}

After the usage of the single variables is analyzed, the following anomalies can be detected:

§ ur-anomaly of the variable Help: The domain of the variable is limited to the function. The first usage of the variable is on the right side of an assignment. At this time, the variable still has an undefined value, which is referenced there. There was no initialization of the variable when it was declared (this anomaly is also recognized by usual compilers, if a high warning level is activated).

§ dd-anomaly of the variable Max: The variable is used twice consecutively on the left side of an assignment and therefore is assigned a value twice. Either the first assignment can be omitted or the programmer forgot that the first value (before the second assignment) has been used.

§ du-anomaly of the variable Help: In the last assignment of the function, the variable Help is assigned another value that cannot be used anywhere because the variable is valid only inside the function.

Data flow anomalies are usually not that obvious

In this example, the anomalies are obvious. But it must be considered that between the particular statements that cause these anomalies there could be an arbitrary number of other statements. The anomalies would not be as obvious anymore and could easily be missed by a manual check such as, for example, a review. A tool for analyzing data flow can, however, detect the anomalies.

Not every anomaly leads directly to an incorrect behavior. For example, a du-anomaly does not always have direct effects; the program could still run properly. The question arises why this particular assignment is at this position in the program, just before the end of the block where the variable is valid. Usually, an exact examination of the program parts where trouble is indicated is worthwhile and further inconsistencies can be discovered.

4.2.4 Execution of Control Flow Analysis

Control flow graph

In figure 4-1, a program structure is represented as a control flow graph. In this directed graph, the statements of the program are represented with nodes. Sequences of statements are also represented with a single node because inside the sequence there can be no change in the course of program execution. If the first statement of the sequence is executed, the others are also executed.

Changes in the course of program execution are made by decisions, such as, for example, in IF statements. If the calculated value of the condition is true, then the program continues in the part that begins with THEN. If the condition is false, then the ELSE part is executed. Loops lead to previous statements, resulting in repeated execution of a part of the graph.

Control flow anomalies

Due to the clarity of the control flow graph, the sequences through the program can easily be understood and possible anomalies can be detected. These anomalies could be jumps out of a loop body or a program structure that has several exits. They may not necessarily lead to failure, but they are not in accordance with the principles of structured programming. It is assumed that the graph is not generated manually but that it is generated by a tool that guarantees an exact mapping of the program text to the graph.

If parts of the graph or the whole graph are very complex and the relations, as well as the course of events, are not understandable, then the program text should be revised, because complex sequence structures often bear a great risk of being wrong.

Excursion: Predecessor-successor table

In addition to graphs, a tool can generate predecessor-successor tables that show how every statement is related to the other statements. If a statement does not have a predecessor, then this statement is unreachable (so-called dead code). Thus a defect or at least an anomaly is detected. The only exceptions are the first and last statements of a program: They can legally have no predecessor or successor. For programs with several entrance and/or exit points, the same applies.

4.2.5 Determining Metrics

Measuring of quality characteristics

In addition to the previously mentioned analyses, static analysis tools provide measurement values. Quality characteristics can be measured with measurement values, or metrics. The measured values must be checked, though, to see if they meet the specified requirements [ISO 9126]. An overview of currently used metrics can be found in [Fenton 91].

The definition of metrics for certain characteristics of software is based on the intent to gain a quantitative measure of software whose nature is abstract. Therefore, a metric can only provide statements concerning the one aspect that is examined, and the measurement values that are calculated are only interesting in comparison to numbers from other programs or program parts that are examined.

Cyclomatic number

In the following, we’ll take a closer look at a certain metric: the →cyclomatic number (McCabe number [McCabe 76]). The cyclomatic number measures the structural complexity of program code. The basis of this calculation is the control flow graph.

For a control flow graph (G) of a program or a program part, the cyclomatic number can be computed like this:11

v(G) = e - n + 2

v(G)

= cyclomatic number of the graph G

e

= number of edges of the control flow graph

n

= number of nodes of the control flow graph

Example for computing the cyclomatic number

A program part is represented by the graph shown in figure 4-1. It is a function that can be called. Thus, the cyclomatic number can be calculated like this:

v (G) = e - n + 2 = 17 - 13 + 2 =6

e

= number of edges in the graph = 17

n

= number of nodes in the graph = 13

Figure 4–1
Control flow graph for the calculation of the cyclomatic number (identical to figure 2–2)

image

The value of 6 is, according to McCabe, acceptable and in the middle of the range. He assumes that a value higher than 10 cannot be tolerated and rework of the program code has to take place.

The cyclomatic number gives information about the testing effort

The cyclomatic number can be used to estimate the testability and the maintainability of a particular program part. The cyclomatic number specifies the number of independent paths in the program part.12 If 100% branch coverage (see section 5.2.2) is intended, then all these independent paths of the control flow graph have to be executed at least once. Therefore, the cyclomatic number provides important information concerning the volume of the test.

Understanding a program is essential for its maintenance.

The higher the value of the cyclomatic number, the more difficult it is to understand the flow in a certain program part.

Excursion

The cyclomatic number has been very much discussed since its publication. One of its drawbacks is that the complexity of the conditions, which lead to the selection of the control flow, is not taken into account. It does not matter for the calculation of the cyclomatic number whether a condition consists of several partial atomic conditions with logical operators or is a single condition. Many extensions and adaptations have been published concerning this.

4.3 Summary

§ Several pairs of eyes see more than a single pair of eyes. This is also true in software development. This is the main principle for the reviews that are performed for checking and for improving quality. Several people inspect the documents and discuss them in a meeting and the results are recorded.

§ A fundamental review process consists of the following activities: planning, kick-off, preparation, review meeting, rework, and follow-up. The roles of the participants are manager, moderator, author, reviewer, and recorder.

§ There are several types of reviews. Unfortunately, the terminology is defined differently in all literature and standards.

§ The walkthrough is an informal procedure where the author presents her document to the reviewers in the meeting. There is little preparation for the meeting. The walkthrough is especially suitable for small development teams, for discussing alternatives, and for educating people in the team.

§ The inspection is the most formal review type. Preparation is done using checklists, there are defined entry and exit criteria, and a trained moderator chairs the meeting. The objective of inspections is checking the quality of the document and improvement of development, the development process, and the inspection process itself.

§ In the technical review, the individual reviewers’ results must be given to the review leader prior to the meeting. The meeting is then prioritized by assumed importance of the individual issues. The evaluators usually have access to the specifications and other documentation only. The author can remain anonymous.

§ The informal review is not based on a formal procedure. The form in which the results have to be presented is not prescribed. Because this type of review can be performed with minimal effort, its acceptance is very high, and in practice it is commonly used.

§ Generally, the specific environment, i.e., the organization and project for which the review is used, determines the type of review to be used. Reviews are tailored to meet specific needs and requirements, which increases their efficiency. It is important to establish a cooperative and collaborative atmosphere among the people involved in the development of the software.

§ In addition to the reviews, a lot of checks can be done for documents that have a formalized structure. These checks are called static analyses. The test object is not executed during a static analysis.

§ The compiler is the most common analysis tool and reveals syntax errors in the program code. Usually, compilers provide even more checking and information.

§ Analysis tools that are dependent on programming language can also show violation of standards and other conventions.

§ Tools are available for detecting anomalies in the data and control flows of the program. Useful information about control and data flows is generated, which often points to parts that could contain defects.

§ Metrics are used to measure quality. One such metric is the cyclomatic number, which calculates the number of independent paths in the checked program. It is possible to gain information on the structure and the testing effort.

§ Generally, static analyses should be performed first, before a document is subjected to review. Static analyses provide a relatively inexpensive means to detect defects and thus make the reviews less expensive.