Best Practice Software Quality Assurance

Ben U. Livson

	Web BAL Consulting

Links 

Best of SQA, Audit and Review Guidelines

Best of Software Engineering

BAL Risk Assessment Methodology BRAM

Overall BAL Methodology BALM

Publications

 Content
 

Summary

Corporate Management Policy

Authority

Budgeting

Conflict

Customer

Profile

Conformance Profile

Audit Profile

VV&T Profile

Software Configuration Management

Legalise

Roles

Methods, Standards and Tools

Conclusions

References

SUMMARY top

This report describes a comprehensive and practical approach to software quality assurance (SQA). The emphasis is on SQA organization,' management and task profile on corporate basis. The relationships between SQA and systems engineering, software developer, project office, subcontractors,' customer, and independent verification, validation and testing IVV&T are discussed. The SQA role in software configuration management (SCM) is covered. Software engineering methods ', standards and tools that contribute to software quality are described. SQA manpower qualifications and the assignments SQA is capable of fulfilling as a function of the quality and quantity of available manpower are analyzed. The paper is based on the author's experience as manager of the Israel Aircraft Industries Ltd Engineering Division SQA from 1978-1986. No project specific data is presented and the approach presented should be valid both for embedded computer systems and data processing quality assurance.

Key Words: Software Quality Assurance (SQA), Software Configuration Management (SCM), Independent Verification, Validation and Testing (IVV&T).

CORPORATE MANAGEMENT, POLICY AND SQA top

Software projects cannot be managed on a voluntary basis. Software quality assurance cannot function without an explicitly defined corporate policy on SQA authority and responsibility. Corporate management shall enforce corporate policy on SQA the same way as corporate policy is enforced on all other issues. SQA cannot function on an advisory consulting basis. There shall be no one time subcontracting of SQA to a third party without a clearly defined corporate policy. In particular, a company should not be run on the bases of a project office without the enforcement of corporate policy. The interests of a project office are usually short-term and in many cases conflicts sound corporate policy and the most elementary SQA disciplines. Corporate policy on SQA should address the issues of authority, budgeting and resolution of conflicts.

SQA Authority top

Corporate bodies that cannot assume responsibility are redundant and should be eliminated. No body can assume responsibility without explicitly defined and enforced authority. SQA authority shall as a minimum include release approval of. all deliverable items, and approval of software reviews. The refusal of SQA to approve a deliverable item shall prevent its release to customer. There shall be no sending of drafts to customers without prior SQA approval. Any further work depending on the approval of the deliverable item shall be halted. The same shall apply to SQA review approval. SQA refusal to approve a review shall halt any further work. A project office bypass of SQA approval to the extent permitted by corporate policy shall lead to conflict resolution by corporate management. There shall be a customer notification about any bypass of SQA approval by project office. The basic issue is whether SQA has the authority to control software development and support process or not.

Failure by corporate management to define SQA authority and to enforce this authority means that the SQA activity should be terminated in the given corporation.

SQA Budgeting top

Corporate management shall define criteria about the relative size of SQA budgeting as a function of its task profile and the budgeting sources for SQA. SQA budgeting shall be part of the Request For Proposal (RFP) cost evaluation according to corporate policy and not dependent on the goodwill of project offices. Corporate policy shall state that SQA approval of RFP is required. The sources of SQA shall be such that the independence of SQA will be guaranteed. In particular, corporate management shall allow overhead budgeting for SQA organization, research and training to build SQA capability on a continuous basis rather than subject to a one-time project.

Conflict Resolution top

Conflicts between SQA and software developer, configuration management, systems engineering and project office should be resolved by corporate management.

The author sees SQA as the in-house arm of corporate management, whereas independent verification, validation and testing organization is the vehicle of the customer. Conflict resolution by project office is a poor alternative for SQA. Project office considerations are dictated by time table constraints, budget cuts, the fact that without the developer there is NO PRODUCT, and the fact that most project engineers leave the project before delivery to the customer. Project Office experiences SQA in many cases as an unpleasant auxiliary function, needed to placate the attention of a hostile customer.

Customer and SQA top

Customer shall make sure that supplier has an explicitly stated and enforced corporate policy on SQA. Customer shall have inside information about supplier. It is only the tip of the iceberg to state software engineering standards in the RFP and then forget about SQA. Customer shall evaluate supplier SQA personnel capability to fulfill the required task profile before selecting a supplier. Customer shall request on contractual basis SQA plan with deliverable items time table. Customer shall monitor supplier SQA activity on a continuous basis. Customer interface with supplier SQA shall be stated as part of the contract between customer and supplier. Combined in-house SQA and customer IVV&T is the best customer solution for product quality. Interaction of in-house SQA and customer IVV&T should be minimized or even prevented. Both functions shall be independent.

SQA TASK PROFILES top

SQA Definition: A planned and systematic pattern of all actions necessary to provide adequate confidence that the item or product conforms to established technical REQUIREMENTS. (ANSI/IEEE Std 730-1984)

The SQA task definition leads to three possible task profiles:

The SQA definition of providing adequate confidence that the product conforms to REQUIREMENTS cannot be achieved by mechanical checks that documents have paragraph titles as listed in standards. The means for providing the required confidence must be provided by SQA software engineering. The minimal SQA conformance profile is attractive to many companies, since it requires a minimal budget allocation to SQA, usually less than five percent of the software development budget. Also, SQA work (of this kind) is not attractive to senior software engineers, so that even if budgets are available this is the only task profile which the available SQA manpower is capable of providing. The image of SQA in the seventies was a librarian checking the labels of magnetic tapes. Unfortunately, this image has not been completely erased despite extensive educational efforts by the military and professional organizations. Most DOD-STD-2168^* assessments are covered by the minimal profile.

NB> DOD-STD-2168^* cancelled on 20 March 1995 and merged into Mil-STD-498, which superseded DOD-STD-2167A, DOD-STD-7935A, and DOD-STD-1703. MIL-STD-498 in turn was cancelled in June 1998 when IEEE/EIA 12207 was released.

SQA must have senior Systems and software engineers capable of auditing the CONTENTS of deliverable documentation. The qualifications of SQA manpower for this profile are most demanding. SQA engineers must have prior experience and proven capability in defining and implementing projects audited. For instance, auditing of an avionics operating flight program O.F.P project would require a team consisting of pilots, systems engineers and senior software engineers that have specified previous or similar projects. Surely, a junior programmer or librarian is useless in this profile. Furthermore, it is emphasized that an SQA team in the auditing profile must have an interdisciplinary capability and it must be understood that junior people cannot grow to such tasks at the expense of projects. The excuse of hiring junior programmers for SQA, because nobody else is available, is not acceptable. In such circumstances it is preferable to refrain from SQA auditing and not to audit punctuation in documents.

It is the responsibility of both the customer, corporate management, project office and, of course, the SQA manager to evaluate the capability of the SQA team to fulfill the required auditing profile. The auditing profile itself is not resource intensive. Usually around ten per cent of budgets is sufficient. The real problem is that top quality manpower is required. Both ANSI/IEEE Std 730-1984 and MIL-S-52779A require SQA auditing profile. SQA auditing is a major issue in configuration management. DOD-STD-2168^* auditing profile is limited and most of this DoD standard relates to issues covered by the minimal conformance profile. The DoD standard nevertheless requires an independent assessment role.

Neither ANSI/IEEE Std 730-1984 nor MIL-S-52779A explicitly state that SQA shall conduct verification, validation and testing. The definition that SQA shall provide adequate confidence that the product conforms to established REQUIREMENTS is usually interpreted that SQA plan shall reference or document procedures to assure the accomplishment of VV&T objectives. Thus, MIL-S-52779A requires SQA assurances for the following objectives:

a. Analysis of software requirements to determine testability (a major challenge for the SQA auditing profile).

b. Review of test requirements and criteria for adequacy, feasibility, traceability and satisfaction of requirements (requirements for the SQA auditing profile).

c. Review of test plans, procedures, and specifications for compliance with contractor and contractual requirements, to assure that all authorized and only authorized changes are implemented (SQA auditing profile with interface to configuration management).

d. Verification that tests are conducted in accordance with approved test plans and procedures (requirement for SQA at least to participate in testing).

e. Certification that test results are the actual findings of the tests (requirement for SQA to participate in testing and to monitor testing).

f. Review and certification of test reports (as above).

g. Assurance that test related media and documentation are maintained to allow repeatability of tests (SQA assurance of auto regression testing requires involvement both in testing and configuration management).

h. Assurance that support software and computer hardware to be used to develop and test software and hardware under the contract are acceptable.

MIL-S-52779A does not explicitly require SQA to perform tasks a-h, but SQA is held responsible for the accomplishment of a-h. It is the task of both customer, corporate management, project offices and, of course, SQA management to evaluate as to whether SQA is capable of undertaking the above responsibility.

ANSI ' /IEEE Std 983-1986 Guide for SQA Plans states software verification and validation plan and report as compulsory deliverable items, states the required contents, and explicitly states that SQA is responsible for the reviews of these deliverable items. Again, SQA role for VV&T shall include auditing but need not be restricted to auditing according to both military and ANSI/IEEE standards.

The VV&T profile of SQA should in the author's opinion be entrusted to an independent organization external to the supplier company. The IVV&T shall be directly subordinated to customer. Basically, full scale IVV&T for critical projects requires management of a parallel project with a complete life-cycle. The resources required for IVV&T may even exceed those allocated to supplier software development.

Extensive IEEE/ANSI standardization efforts for verification and validation plans and guide for implementing such plans are under way.

SQA and Software Configuration Management top

SQA must function in an identified and controlled software engineering process. Both ANSI/IEEE Std 730-1984 and MIL-S-52779A require SQA to ascertain that SCM plan complies to certain standards, but do not require SQA to write the SCM plan. In the author's opinion SQA shall be responsible for the SCM auditing function and SQA shall participate in software configuration control boards. The above SQA plan standards request SQA plan to assure the detection, documentation, and correction of software problems. This function is best implemented as part of the configuration management system and should not be implemented as a separate system.

SQA should have the necessary privileges as configuration management system user to ascertain that problem reporting is handled properly. In the author's opinion the practice of quality assurance organizations to maintain a Failure Reporting and Corrective Action System (FRACAS) separate from the corporate or project configuration management system is not a good practice for software projects. It is not easy to separate problem reports from change proposals, and it is not easy to separate the operations and maintenance field service from continuous software development and support. There should be one and only one system to handle the entire flow of change proposals and problem reports. The above viewpoints are necessary to Comply to DOD-STD-2168^*.

Contractual, Legislative and Liability Aspects top

SQA is by definition responsible for product conformance to REQUIREMENTS. SQA cannot undertake such a responsibility unless it participates in the contractual phase. SQA has to approve both cost and technical proposals in the request for proposal phase. SQA approval of the cost proposal is important to ensure that SQA is properly budgeted to uphold its responsibility. Efforts in the contractual phase should not be saved. It is better to invest three more months in the contractual phase than scrap a project after five years and to continue court battles for the next decade. Software engineering will approach systems engineering. The process of implementation will be more and more automated. Future software engineering will be a massive dialogue between customer and supplier to reach a joint product specification. Major advances are being made in rapid prototyping and application generators to support the joint customer and supplier product specification phase. Software engineering is slowly approaching the stage of automatic code generation from specification also in the area of embedded computer systems, although this was considered a virtual impossibility only a few years ago. The number one problem will be how the supplier would be able to understand what the customer wants and how the customer would be able to ascertain that the supplier has understood what he wants. The future thrust of both in-house SQA and customer IVV&T shall be in the contractual specification phase.

 The issue of liability shall be addressed much more strictly. Liability shall not be only on a corporate basis, but in the author's opinion there shall be personal liability for corporate management, project office management, software development management and SQA management. The issue is not about this or that software engineering method or tool being adequate but about personal responsibility. There shall be appropriate legislation in the criminal code to enforce the personal responsibility for safety critical projects. The gap between legislation and the computer era has to be closed.

The Roles of a SQA Person top

Fletcher Buckley [7] gives three roles for a SQA person: Information-Gatherer, Policeman and Helper. The Information-Gatherer role can covered by the minimal conformance and audit profiles. The same @applies to the Policeman role. The SQA role as helper or even as advisor is rejected by the author. Helping and giving advice are the responsibility of all employees that shall not be an excuse for failing to carry out a person's task obligations.

Software Engineering Methods, Standards and Tools top

SQA methods, standards and tools are a strict subset of software engineering methods, standards and tools. There are no unique SQA methods. Simply, SQA is interested in the enforcement of methods, standards and tools that contribute to product quality. Certain hardware reliability methods such as redundant fault tolerant software or fault tree analysis of critical systems have been attempted by SQA. If these methods will have merit, they will be adopted as part of general software engineering methods. SQA has been by nature very active in software engineering standardization. The basic idea is to reduce software art into a controlled and well-documented engineering process. In the author's opinion partisan SQA activity on corporate basis should be stopped.

The most comprehensive software standardization project is being carried out by IEEE. Corporate tuning of these standards is both a waste of time and even harmful. IEEE provides comprehensive guides for the use of its SQA, SCM and SVV standards. The issue of software engineering methods and the tools that implement some of these methods are very complicated. None of the software standards address to any specific method, and for a very good reason.

There is no firm proof that any of the software requirements and design methods contribute to product quality. on the contrary, there is ample evidence that a number of projects have lost time and sustained damage, because a lot of energy was wasted in trying to experiment on new methods and tools at the expense of projects. Particularly harmful are experiments adopted at more advanced project stages. Methods and tools should be frozen at an early project stage. The argument against such a freeze is that the project life-cycle may be very long, say ten years, and methods and tools are at a very dynamic stage of development. However, the customer should be consulted about methods and tools not stated explicitly in the contract. The common approach that the supplier may do whatever is not explicitly against the terms of the contract is a likely path to failure. The following trends in software methods and tools show the best promise for product quality in the author's opinion:

0. Rapid Prototyping to support joint customer and supplier specification.
1. Application Generators that produce a target code from specification. The enormous success of application generators in commercial data processing is slowly spreading to wider application areas, including computer embedded systems. Application generators free software engineering from low level coding and implementation into high level systems engineering of product specification.
2. Software engineering embedded in Relational Computer Information Architecture. Relational database should cover requirements, architectural design, interface control definition, acceptance test planning and configuration management. SQA should have an important role both in the definition and use of the project relational database. The SQA advantage of this approach is that traceability of testing and design to requirements is gained with minimal effort. Target code data structures can be automatically created from an interface control database, thus eliminating a of lot annual development effort and the need for mechanical checks by SQA against interface errors.

4 - Software configuration management tools will be integrated with the software environment in real time. SCM tools that work as off-line post mortem utilities shall be scrapped. Low level configuration management operations shall be generated automatically in parallel to the software development and support process. High level SCM operations shall be part of the relational project information architecture. A new generation of SCM tools is about to be introduced into service within the next two or three years.

5 - Testing Tools that support auto regression testing will enter into extensive use in the next few years. It is likely that most computer vendors will include both static and dynamic testing tools as basic software.

The key issue for SQA is corporate management policy on SQA. Corporate management shall define and enforce SQA AUTHORITY, budgeting and conflict resolution. SQA shall be the vehicle of corporate management the way IVV&T is directly subordinated to customer. Both corporate management and customer jointly shall ascertain SQA capability of undertaking the responsibility of assuring product conformance to REQUIREMENTS. SQA management shall approve all deliverable items and reviews starting from the contractual phase. A software project contract shall not be signed without SQA approval. SQA shall have the authority to freeze project development at all milestones. Corporate management and customer shall monitor SQA on a continuous basis. The interface between SQA and customer shall be defined as part of the project contract. SQA shall approve the configuration management plan, and monitor SCM problem reporting and handling, participate in software configuration control board, and SQA shall be responsible for the SCM auditing function. contractual liability issues shall receive appropriate attention. Added emphasis shall be given to the contractual phase in general and to the active SQA participation in the contractual phase in particular. Finally, the author's viewpoint is that SQA methods, standards and tools are a subset of general software engineering methods, standards and tools. Adoption of ANSI ' /IEEE standards is strongly urged. Likely software tool developments in the next few years will result in large scale integration of software tools in a relational computer information architecture with total automation of low level configuration management. Rapid prototyping and application generators shall assist the joint customer and supplier project specification phase.