Best Practice SQA, Auditing and Reviews

ę Copyright 1978-2017 BAL Consulting«P/LAll rights reserved. Citation Policy.

Google
 
Web BAL Consulting

Something is rotten in the state of Denmark. - William Shakespeare, "Hamlet", Act 1 scene 4

Ask | Contact | Email | Feedback | Legal | Map | Privacy | Search+ | Secure Message | Talent | Tools
Australia | Check | Currency | Market | News | Phonebook | Sensis´┐Ż | Time | Units | Weather | White Pages | Yellow Pages

Links

Best Practice Software Quality Assurance

BAL Methodology BALM

BAL Risk Analysis Methodology BRAM

Best Practice Software Engineering

Publications

Survey of U.S. Software Tools & Technology 1981 {PDF 3.5MB}

Ben Livson Archive 1970s R&D Papers         

GUIDELINES FOR AUDITING AND REVIEWS

Skepticism, like chastity, should not be relinquished too readily. - George Santayana

 Content

Introduction

How to Organise Audits

Preparations

Conduct

Psychology

Checklists

Requirements Review

Architectural Review

Design Review

Implementation

Formal Qualification Audit

Documentation Review

Summary

References

NB. This document brings fond memories - was originally prepared in 1978 using a 30 cps Teletype 43 terminal with UPPER CASE characters when writing 3GL programs was adventure in the embedded computer systems world where most software was written in assembler. The document was provided to the IEEE Software Engineering Standards Sub-Committee in 1981 - the IEEE guidelines for audits and reviews evolved many years later.

ABSTRACT

Brevity is the soul of wit. - William Shakespeare, "Hamlet", Act 2 scene 2

A set of guidelines for review and auditing of software documents is provided. The guidelines contain checklists for auditing the documents produced at the various phases of the life cycle. Also, rules for conducting reviews are given and comments about possible pitfalls are included. Psychological aspects of the review process are emphasized. The guidelines are organized in a way that enables systematic application for the analysis of a given document.

KEYWORDS: audit, review, software quality assurance, software configuration management, walk-through, software standards.

1. INTRODUCTION top

It is quality rather than quantity that matters. - Seneca, Epistles

Auditing and reviews are both a major software quality assurance activity and one of the four functions of configuration management: Extensive software tool support can be provided to all the other configuration management functions except auditing. Auditing and reviews are by nature a manual process that can be supported by proper guidelines and some minor software tools.

The process of auditing and reviews is the main means of feedback for management and buyer. This is especially true for material that is not readable by the computer.

2. ORGANIZING FOR AUDITS top

2.1 ROLES

The lady doth protest too much, methinks. - William Shakespeare, "Hamlet", Act 3 scene 2

2.2 PREPARATIONS BEFORE THE MEETING top

I have left orders to be awakened at any time in case of national emergency, even if I'm in a cabinet meeting. - Ronald Reagan (40th US President)

The coordinator has to gather the material and distribute it. This gathering activity must be done if there are more than one source of documents. In this case the 'producer' will not be the actual author but he will be THE PRESENTER, rather than the 'producer'. The coordinator sends the invitation and assures the appropriate time for the meeting.

2.3 CONDUCTING THE REVIEW top

No legacy is so rich as honesty. - William Shakespeare, "All's Well that Ends Well", Act 3 scene 5

 2.4 PSYCHOLOGICAL PROBLEMS top

Do not speak ill of the dead. - The Seven Sages, from Diogenes Laertius, Lives of Eminent Philosophers

3. Guidelines and Checklists

I never cease being dumbfounded by the unbelievable things people believe. - Leo Rosten

3.1 PHASES OF AUDITS

The phases of the reviews to be discussed here are:

3.2 Checklists top

The following requirements are applicable to all phases of reviews:

 3.2.1 FORMAT CHECKLIST

3.2.2 GENERAL CHECKLIST

The general checklist is suited to all phases of the reviews. Each item in the checklist can be checked for each paragraph for each level of the document internal structure.

(1) ACCURACY.

(2) COMPLETENESS (3) CONSISTENCY (4) USEFULNESS (5) TESTABILITY (6) MODULARITY (7) CLARITY (8) FEASIBILITY

For higher level documentation like requirements - consideration should be given to implementation issues. If implementation is not clear some indication that it is possible must be given. It may suffice to point out similar requirements that were solved elsewhere.

(9) RELIABILITY

(10) MAINTAINABILITY (11) HUMAN ENGINEERING FACTORS

(12) MINIMAL COMPLEXITY

Minimize complexity across:

CONCEPTUAL INTEGRITY

COMPONENT INDEPENDENCE

The system has to be partitioned into its components so that the dependence of the components is minimized. The high frequency dynamics of the system falls within the components and the inter-component interactions represent the lower frequency dynamics.

 HIERARCHY

-- The hierarchy is a method for simplifying the description of a system. Check if this method was used properly.

(13) TRACEABILITY

This is an important feature. Each item of documentation must have its source clearly stated.

 3.2.3 AMBIGUOUS WORDS TO BE OBSERVED

The following words may have ambiguous meaning:

Some different possibilities are given for each example below:

A (AN): (one and only one? some? at least one?)

"The data set will contain an end of file character. "

AFTER: (immediately? somewhere after?)

"The control number comes after the quantity."

ALL THE TIME: (initially and never reset? Every time?)

"The interrupt status is set to 'optional' all the time."

AND: (either...or? double condition?)

"The sequence ends on a flag and an end of file."

CURRENT: (current for program? current for real system? current in other sense?)

"The control total is taken form the current record."

FOLLOWING: (following immediately? Following somewhere away?)

"The checksum is on the following summary card."

FROM: (starts from designated number? Starts somewhere after the number?)

"The registry number start from 100."

LARGEST: (what is the set and sequence?)

"The field will be set to the largest possible value." (character or number?)

LAST: (last for the program read? last referenced? last updated? last in file?)

"The control total is taken form the last record."

LATEST: see previous word.

MAY: (can it be something else not mentioned too?)

"The return code may contain an integer or a blank."

OR: (either...or? only one?)

"The sequence ends on a flag or an end of file."

SAME: (same value? same format?)

"All customers have the same control field."

SHOULD: (must? ought?)

"The operator should mount the designated disk pack."

SMALLEST: see LARGEST.

THEIR: (whose exactly?)

"A family unit consists of a mother and father and all their children."

THEY: see THEIR.

TO: see FROM

UNTIL: (stopped now? stopped by other condition also?)

"Elements are added until the element with the present date has been added."

WE: (who exactly - the user, the developer and the user, who else?)

"We will create an operator's guidebook."

WHEN: (the only way? are there other ways?)

"The terminal session ends when the sign-off command is issued."

WHEN: (by the time? the first time? only when? whenever?)

"The counter is set to zero when the subroutine is invoked."

WILL: (who will? is it a fact?)

"The pointer will be set before the data value is set."

- HOW TO USE THE LIST OF WORDS:

 A. Each paragraph can be read with the list of items in mind. This is

possible after the list has been more or less memorized.

B. If the document is in machine readable form - a computer search can be made for statements containing the listed words. Then they can be examined for ambiguities.

3.2.4 AMBIGUOUS EXPRESSIONS TO BE OBSERVED

1. UNPROVED CERTAINTY.

"There can be no more than fifty lines per invoice."

Is it a hope? What is the proof?

Push the search for a proof as far back as possible.

That is - the proof, too, needs a proof.

2. REQUIREMENT OR FACT?

"The flag is set to zero."

Is it a fact or a requirement?

3. VAGUE WORDS:

SOME, SOMETIMES, OFTEN, USUALLY, ORDINARILY, CUSTOMARILY, MOST, MOSTLY.

"Exception processing will be required for some of the line items."

What is the exact number?

4. INCOMPLETE LISTS.

Watch for: ETC., AND SO FORTH, AND SO ON, SUCH AS.

"The permitted inputs are 'A', 'B', 'C', etc.

The rest of the list may contain two more items or A-Z letters.

Try to complete the lists.

5. UNSTATED ASSUMPTIONS.

"The valid codes range from 100 to 1000."

Is it only integers? hexadecimals?

6. LISTS WITHOUT EXAMPLES (OR TOO FEW EXAMPLES)

See previous rule.

7. VAGUE VERBS.

HANDLED, PROCESSED, REJECTED, SKIPPED, ELIMINATED.

"Unmatched detail records are to be rejected."
What has to be done on rejection?

8. PERSUASIVE WORDS.

CERTAINLY, THEREFORE, CLEARLY, OBVIOUSLY, AS ANY FOOL CAN PLAINLY SEE.

"Clearly, the dataset control pointer will be set before any data value is set."

Is it a fact? Perhaps a requirement?

9. PASSIVE VOICE.

"The parameters are initialized."

By whom? may be by nobody?

10. COMPARATIVES WITHOUT REFERENTS.

"The final total is to be placed in the NEW file."

NEW - compared to what? It may depend upon the time of reference.

11. PRONOUNS.

IT, HE, SHE, THEY, HIS, HERS, ITS, THEIR, WE, US, OUR, YOU, YOUR.

"Module ZAP is called by module ZIP. It must be given the security code."

Who - ZIP or ZAP?

3.2.5. TRANSFORMATIONS FOR CLARIFICATION

The meaning of a paragraph may be clarified or ambiguities revealed by subjecting the text to various transformations:

1. VARY STRESS PATTERNS.

"The EXCEPTIONAL cases are handled by the terminal operator." (Only those cases!)

"The exceptional cases are handled by the TERMINAL operator." (No other operator!)

2. TERM SUBSTITUTION.

Substitute the definition of the terms instead of the terms themselves.

"The 'current status file' becomes the 'old status file'."

"The 'direct access file containing the current status of all active accounts' becomes the 'sequential file of accounts carried over from the previous period'."

Notice the disparities in the definitions.

3. PICTURE THE STRUCTURE.

When a structure or a relation is described in words - try to picture it. See if the relations among the objects are unique. Try to change the relations and see if it fits the description in words.

4. EXPRESS EQUATIONS IN WORDS.

5. EXPRESS CALCULATIONS IN AN EQUATION.

6. WORK TWO EXAMPLES OF AN EQUATION BY HAND.

4. PHASE GUIDELINES

My salad days, when I was green in judgment. - William Shakespeare, "Antony and Cleopatra", Act 1 scene 5

For each phase there are specific guidelines in addition to the general guidelines mentioned above:

4.1 SOFTWARE REQUIREMENTS REVIEW top

(1) Documents required:

(2) Review checklist 4.2 ARCHITECTURE -PRELIMINARY DESIGN AUDIT top

The documents required are:

The 'strength' of the module must be maximized. That is the relationship among the sub-parts of the module must be such that they perform only one clearly defined function.

The 'coupling' of the modules should be minimized - the data relationships among the modules have to be by passing data as parameters and as elements of data and not as data structure.

The MYERS terminology for strength and coupling:

STRENGTH (weak to strong): Co-incidental, logical, classical (termination, initialization), procedural, communicational, functional.

COUPLING (strong to weak): Content, common, external, control, stamp, data.

Observe: Two kinds of reserves must be allocated:
  1. Reserve-1 needed to cope with erroneous design estimates and
  2. Reserve-2 needed for possible addition of functions in future.
Check degradation and recovery procedures: 4.3 DETAILED DESIGN AUDIT top

(1) Necessary documents

(2) Detailed Design Review Checklist The checklist of the Preliminary Design is also applicable to the Detailed Design. In addition check the following: 4.4 Implementation Review - INSPECTION top

4.4.1 DATA REFERENCE CHECKLIST

4.4.2 DATA DECLARATIONS CHECKLIST  4.4.3 COMPUTATION CHECKLIST 4.4.4 COMPARISON CHECKLIST 4.4.5 CONTROL FLOW CHECKLIST 4.4.6 INTERFACE CHECKLIST 4.4.7 INPUT/OUTPUT CHECKLIST  4.4.8 OTHER CHECKS FORM ECONOMY 4.5. FORMAL QUALIFICATION AUDIT top

A judge is a law student who marks his own examination papers. - H L Mencken

The FQA is used to approve the final product. It is therefore not necessary to review the product with respect to content, this is supposed to be done during the development by internal walk-through, and periodic reviews and testing.

The FQA is mainly a formal check of the contract implementation and other requirements.

 
5. THE DOCUMENT REVIEW PROCESS
top

Drama is life with the dull bits cut out. - Alfred Hitchcock

Review the document as a whole. Scan the format checklist and take notes as to adherence to requirements.

5.1. Read each section and paragraph. Scan the general checklist for each section and see if any of the items are appropriate.

Special reports and tools as the index and the glossary may be used.

5.2. For a section higher in the hierarchy the more specific checklist may be applied. These checklists address the document at a higher level or apply to the document as a whole.

5.3. If the document is written in a special language like PSL (Program Statement Language) - generate additional reports. These reports may help you answer questions like: Does every data item have source? Is every process invoked?

5.4. Note any discrepancies. Key them according to the category of checklist that is appropriate. This may later be used for statistical purposes.


6. SUMMARY
top

Love all, trust a few. Do wrong to none. - William Shakespeare

The process of auditing is the primary feedback to management. Management needs to commit resources necessary for this process.

 
7. BIBLIOGRAPHY top

1. Software Configuration Management. Edward H. Bersoff, Vilas D. Henderson, Stanley G.Siegel, Prentice-Hall, N.J., 1980.

2. Software Reliability: Glenford J. Myers. Wiley & Sons, Inc. 1979.

3. Structured Walkthrough: Edward Yourdon. Yourdon Press, Fourth Edition 1989.

4. Quality Assurance for Computer Software. Robert Dunn, Richard Ullman. McGraw-Hill, 1982.

5. Mil-Std-1521 (USAF), Military Standard - Technical Reviews and Audits for Systems, Equipment, and Computer Programs superseded by DoD 5000 system in 1995.

6. IEEE Std 830-1993 IEEE Recommended Practice for Software Requirements Specifications

7. IEEE Std 1028-1997 IEEE Standard for Software Reviews

8. Lateral Thinking: Edward De Bono. Penguin Books, 1990.

9. MIL-STD-483 Configuration Management Practices for Systems – a classic from 1970s. Also, see the Configuration Management Resource Guide.

10. Software engineering – Wikipedia 2010, The IEEE's Guide to the Software Engineering Body of Knowledge and The Joint Task Force on Software Engineering Association ... Software quality • Software quality assurance

11. Handbook of Walkthroughs, Inspection, and Technical Reviews: Daniel P. Freedman, Gerald M. Weinberg. Little, Brown and Company, 1982.

12. A Guide to the Project Management Body of Knowledge (PMBOK´┐Ż Guide), Fourth Edition, Project Management Institute, released on 31 December 2008 contains a large number of references to Audits and Reviews as Tools & Techniques e.g. Performance and Documentation Reviews, Quality, Risk and Procurement Audits.

 

End of Guidelines for Auditing and Reviews

A man never reaches that dizzy height of wisdom that he can no longer be led by the nose. - Mark Twain

Ask | Contact | Email | Feedback | Legal | Map | Privacy | Search+ | Secure Message | Talent | Tools | top

Australia | Check | Currency | Market | News | Phonebook | Sensis´┐Ż | Time | Units | Weather | White Pages | Yellow Pages