© Copyright 1978-2023 BAL Consulting^®P/L™. All rights reserved. Citation Policy.

	Web BAL Consulting

Something is rotten in the state of Denmark. - William Shakespeare, "Hamlet", Act 1 scene 4

Links

Best Practice Software Quality Assurance

BAL Methodology BALM

BAL Risk Analysis Methodology BRAM

Best Practice Software Engineering

Publications

Survey of U.S. Software Tools & Technology 1981 {PDF 3.5MB}

Ben Livson Archive 1970s R&D Papers         

GUIDELINES FOR AUDITING AND REVIEWS

Skepticism, like chastity, should not be relinquished too readily. - George Santayana

 Content

Introduction

How to Organise Audits

Preparations

Conduct

Psychology

Checklists

Requirements Review

Architectural Review

Design Review

Implementation

Formal Qualification Audit

Documentation Review

Summary

References

NB. This document brings fond memories - was originally prepared in 1978 using a 30 cps Teletype 43 terminal with UPPER CASE characters when writing 3GL programs was adventure in the embedded computer systems world where most software was written in assembler. The document was provided to the IEEE Software Engineering Standards Sub-Committee in 1981 - the IEEE guidelines for audits and reviews evolved many years later.

ABSTRACT

Brevity is the soul of wit. - William Shakespeare, "Hamlet", Act 2 scene 2

A set of guidelines for review and auditing of software documents is provided. The guidelines contain checklists for auditing the documents produced at the various phases of the life cycle. Also, rules for conducting reviews are given and comments about possible pitfalls are included. Psychological aspects of the review process are emphasized. The guidelines are organized in a way that enables systematic application for the analysis of a given document.

KEYWORDS: audit, review, software quality assurance, software configuration management, walk-through, software standards.

1. INTRODUCTION top

It is quality rather than quantity that matters. - Seneca, Epistles

• identification;
• configuration control;
• status accounting;
• auditing.

The process of auditing and reviews is the main means of feedback for management and buyer. This is especially true for material that is not readable by the computer.

• This is to assure that specifications conform to requirements and the code to the specs.
• The audits are methods by which management and buyer can give their approval to continuation of the development effort.
• At a less formal level - it is used for internal checks and verifications.
• There are several types of reviews:
• REVIEW: used to check progress on a periodic basis
• AUDIT: used for checking all aspects of the product, usually at a milestone.
• CODE INSPECTION: a peer inspection of code during development.
• WALKTHROUGH: a simulation of what the system is supposed to do. Preparing a set of test cases and tracing system transformation in response to assumed conditions does it. Although it is done usually at a code level, it may also be done at a higher level.

2.1 ROLES

The lady doth protest too much, methinks. - William Shakespeare, "Hamlet", Act 3 scene 2

• The people taking part in an audit may vary according to type of the audit (or review). At least the following should participate:
• PRODUCER. The person responsible for the product being reviewed. This may be the manager of the department - if it is a big product - or the actual writer of the document.
• Her role is to present the product and to respond to comments of other participants.
• If the review is an informal one, the producer is not expected to react immediately to comments but to elaborate them afterwards.
• COORDINATOR. The person summoning the meeting and scheduling the audit. The coordinator is the manager of the audit. His duties include deciding who will be summoned, distribution of invitations, distributing of material and preparing of the room. During the actual meeting the coordinator will act as moderator, and after the meeting he will prepare minutes and distribute them to the participants.
• SCRIBE. The person taking notes, he may be the coordinator.
• USER REPRESENTATIVE.
• QUALITY ASSURANCE REPRESENTATIVE. Will consider issues of testability, reliability, maintainability, conformance to standards and other quality criteria.
• Other reviewers may be included according to the nature of the product, managerial responsibilities or contractual agreement. The configuration manager and the person maintaining the system are likely participants for reviews.
• The reviewer team should preferably not exceed 5-6 people.
• Reviewers must be highly qualified personnel and capable of writing the material they review.
• As development proceeds according to a hierarchical manner - each document is a refinement of a document at a previous level and sources to next document - the reviewers may include representatives of the authors of the previous level and intended authors of the next level, e.g.: the author of the requirements will be represented in the review of the preliminary design document, and will check if the requirements were correctly understood. The author of the detailed design document will also be present and will analyze the feasibility of the preliminary design.

I have left orders to be awakened at any time in case of national emergency, even if I'm in a cabinet meeting. - Ronald Reagan (40th US President)

The coordinator has to gather the material and distribute it. This gathering activity must be done if there are more than one source of documents. In this case the 'producer' will not be the actual author but he will be THE PRESENTER, rather than the 'producer'. The coordinator sends the invitation and assures the appropriate time for the meeting.

• The PRODUCER has to prepare the presentation. If a walk-through is intended the producer has to prepare some test cases. If the producer is also the author it is better that someone else should prepare the test cases (or scripts).
• The participants must prepare themselves for the meeting by reading the material and analyzing it. It is possible to supply them with such support reports as a 'keyword under context' and cross-references : arranged according to data items, according to various groupings. This can be done if the reports are written in some semi formal language like the Problem Statement language (PSL). In that case there are special programs that can produce the support reports. The structure of the preparation will be described later.

No legacy is so rich as honesty. - William Shakespeare, "All's Well that Ends Well", Act 3 scene 5

• The coordinator calls the meeting to order, and gives a brief explanation of the problem, and then introduces the participants to each other.
• The producer takes the floor. If it is a follow-up of previous meetings the producer may review the action taken to react to the previous comments, and then presents the product, piece by piece. At the end of each piece comments are requested.
• If a walk-through is executed - the order of presentation is not necessarily the order of the document but the order of the test cases. As branching points arise (multiple choices) a certain method has to be devised, assuring that all possibilities are considered. The order may be according to levels (for each point where more than one possibility arise - terminate the review of all the choices of the first level and than proceed for each following level the same way).
• During the walk-through the scribe takes notes. The notes do not have to be taken verbatim unnecessary.
• At the end of the review a vote may be taken upon recommendations concerning the product.

Do not speak ill of the dead. - The Seven Sages, from Diogenes Laertius, Lives of Eminent Philosophers

• The moderator must be aware of various dangers and problems of human nature:
• The meeting may evolve into a personality clash. The coordinator must be careful to direct comments to the product being reviewed rather than to the producer. The producer must also be prevented from defending the product by personal attack. This is rather difficult if both top management and the user are present.
• Care must be taken that the discussion will not drift to other topics or will turn into a social chat, although some casual talk may, of course, be permitted in order to allow for a relaxed atmosphere.
• Arguments about minor topics must also be avoided, especially arguments about style.
• The coordinator must pay attention to shy participants who must be encouraged to participate actively. This may be done by eliciting comments according to some predetermined order (like the order of seating) without waiting for the participant to initiate the comment.
• The coordinator has to be sensitive to the fine psychological inclinations and reactions that reflect political or psychological power struggle but are disguised as content comments.

I never cease being dumbfounded by the unbelievable things people believe. - Leo Rosten

3.1 PHASES OF AUDITS

The phases of the reviews to be discussed here are:

• Software Requirements Review.
• Preliminary Design Review.
• Detailed Design Review.
• Code Inspection.
• Formal Qualification Review.

The following requirements are applicable to all phases of reviews:

 3.2.1 FORMAT CHECKLIST

• The format of the document should be checked and conform to company standards with respect to style and external form and items to be included. The following should be checked:
• Table of contents.
• Glossary.
• Breakdown into small items.
• Proper indentation and spacing.
• Use of tables and drawings.
• Proper identification of pages, chapters, figures and tables.
• Proper date and change designation.
• Bibliography.
• Use of footnotes.
• Check style and use of complicated expressions.
• General appearance and typesetting.
• It is advised that the reviewer becomes familiar with the document from the point of appearance and layout.
• Look first at the table of contents and digest the form and structure of the document.
• Prepare a plan for reading the document. The document may be organized in a way that may be difficult to read sequentially like in the order of the alphabet.
• Study the glossary. This will give the reader also a general view concerning the objects that are dealt with in the document.
• Then study the general format of the document. If it is found to be deficient or not compatible with standards or conventions it should be noted.

The general checklist is suited to all phases of the reviews. Each item in the checklist can be checked for each paragraph for each level of the document internal structure.

(1) ACCURACY.

• Are the facts accurate?
• Check computation (when feasible) on a random basis.

• Is any function missing?
• Is any input, output or condition missing?
• Is any possibility overlooked?
• Avoid mental blocks by using various creativity enhancement methods (see De Bono [8]).

• Usage of terms has to be unique. The same term may not be used for different ideas.
• An idea or object must have only one term describing it. Pay special attention to usage of synonyms, different structures of sentences. The use of abbreviations for long terms is allowed.
• The abbreviations must be defined in an abbreviation table and in the glossary.

• Is there a purpose to each feature?
• What is the end use of an interim feature? (A feature that has no meaning for the user).

• How will each feature be tested?
• How will internal features (not visible externally) be tested?

• The system and documents describing it have to be modular. Modularity means independence of modules:
• Maximize internal binding, minimize coupling.
• Modules must be of limited size.
• Structure should be hierarchical.
• Repeated usage of the same module (a module may be a software module, and documentation module - a paragraph) must be properly cross-referenced.

• The meaning of each item is understood.
• There is a single interpretation of each item.
• Each item's description is exact and not vague.
• The specification is easy to read.

For higher level documentation like requirements - consideration should be given to implementation issues. If implementation is not clear some indication that it is possible must be given. It may suffice to point out similar requirements that were solved elsewhere.

(9) RELIABILITY

• What will happen in case of system failure?
• How is the system going to respond to unexpected input?
• What diagnostics are to be produced?
• For critical software a special reliability checklist may be prepared by Quality Assurance and a special Reliability Review may be held.

• The development group has to consider changes, as follows :
• List the features that are not likely to be changed. Grade them according to probability of change. This can be done before the review by sending questionnaires to all participating groups. During the review these lists may be modified. The items not in the list are likely to be changed.
• For items that can be changed a list of expected changes is compiled. The creativity methods mentioned earlier may be also used. Each item within the system can be checked in view of these changes.

(12) MINIMAL COMPLEXITY

Minimize complexity across:

CONCEPTUAL INTEGRITY

• The system has to have a clear unifying line of thought. Command and system reactions must have the same syntax. Internal tables must have similar form and criteria for the design decisions must be the same.
• The user interface has to show a measure of uniformity and harmony. The semantics of the commands has to be similar. An inherent symmetry with respect to syntax, names, operands, defaults.

The system has to be partitioned into its components so that the dependence of the components is minimized. The high frequency dynamics of the system falls within the components and the inter-component interactions represent the lower frequency dynamics.

 HIERARCHY

-- The hierarchy is a method for simplifying the description of a system. Check if this method was used properly.

(13) TRACEABILITY

This is an important feature. Each item of documentation must have its source clearly stated.

 3.2.3 AMBIGUOUS WORDS TO BE OBSERVED

The following words may have ambiguous meaning:

Some different possibilities are given for each example below:

A (AN): (one and only one? some? at least one?)

"The data set will contain an end of file character. "

AFTER: (immediately? somewhere after?)

"The control number comes after the quantity."

ALL THE TIME: (initially and never reset? Every time?)

"The interrupt status is set to 'optional' all the time."

AND: (either...or? double condition?)

"The sequence ends on a flag and an end of file."

CURRENT: (current for program? current for real system? current in other sense?)

"The control total is taken form the current record."

FOLLOWING: (following immediately? Following somewhere away?)

"The checksum is on the following summary card."

FROM: (starts from designated number? Starts somewhere after the number?)

"The registry number start from 100."

LARGEST: (what is the set and sequence?)

"The field will be set to the largest possible value." (character or number?)

LAST: (last for the program read? last referenced? last updated? last in file?)

"The control total is taken form the last record."

LATEST: see previous word.

MAY: (can it be something else not mentioned too?)

"The return code may contain an integer or a blank."

OR: (either...or? only one?)

"The sequence ends on a flag or an end of file."

SAME: (same value? same format?)

"All customers have the same control field."

SHOULD: (must? ought?)

"The operator should mount the designated disk pack."

SMALLEST: see LARGEST.

THEIR: (whose exactly?)

"A family unit consists of a mother and father and all their children."

THEY: see THEIR.

TO: see FROM

UNTIL: (stopped now? stopped by other condition also?)

"Elements are added until the element with the present date has been added."

WE: (who exactly - the user, the developer and the user, who else?)

"We will create an operator's guidebook."

WHEN: (the only way? are there other ways?)

"The terminal session ends when the sign-off command is issued."

WHEN: (by the time? the first time? only when? whenever?)

"The counter is set to zero when the subroutine is invoked."

WILL: (who will? is it a fact?)

"The pointer will be set before the data value is set."

- HOW TO USE THE LIST OF WORDS:

 A. Each paragraph can be read with the list of items in mind. This is

possible after the list has been more or less memorized.

B. If the document is in machine readable form - a computer search can be made for statements containing the listed words. Then they can be examined for ambiguities.

3.2.4 AMBIGUOUS EXPRESSIONS TO BE OBSERVED

1. UNPROVED CERTAINTY.

"There can be no more than fifty lines per invoice."

Is it a hope? What is the proof?

Push the search for a proof as far back as possible.

That is - the proof, too, needs a proof.

2. REQUIREMENT OR FACT?

"The flag is set to zero."

Is it a fact or a requirement?

3. VAGUE WORDS:

SOME, SOMETIMES, OFTEN, USUALLY, ORDINARILY, CUSTOMARILY, MOST, MOSTLY.

"Exception processing will be required for some of the line items."

What is the exact number?

4. INCOMPLETE LISTS.

Watch for: ETC., AND SO FORTH, AND SO ON, SUCH AS.

"The permitted inputs are 'A', 'B', 'C', etc.

The rest of the list may contain two more items or A-Z letters.

Try to complete the lists.

5. UNSTATED ASSUMPTIONS.

"The valid codes range from 100 to 1000."

Is it only integers? hexadecimals?

6. LISTS WITHOUT EXAMPLES (OR TOO FEW EXAMPLES)

See previous rule.

7. VAGUE VERBS.

HANDLED, PROCESSED, REJECTED, SKIPPED, ELIMINATED.

"Unmatched detail records are to be rejected."
What has to be done on rejection?

8. PERSUASIVE WORDS.

CERTAINLY, THEREFORE, CLEARLY, OBVIOUSLY, AS ANY FOOL CAN PLAINLY SEE.

"Clearly, the dataset control pointer will be set before any data value is set."

Is it a fact? Perhaps a requirement?

9. PASSIVE VOICE.

"The parameters are initialized."

By whom? may be by nobody?

10. COMPARATIVES WITHOUT REFERENTS.

"The final total is to be placed in the NEW file."

NEW - compared to what? It may depend upon the time of reference.

11. PRONOUNS.

IT, HE, SHE, THEY, HIS, HERS, ITS, THEIR, WE, US, OUR, YOU, YOUR.

"Module ZAP is called by module ZIP. It must be given the security code."

Who - ZIP or ZAP?

3.2.5. TRANSFORMATIONS FOR CLARIFICATION

The meaning of a paragraph may be clarified or ambiguities revealed by subjecting the text to various transformations:

1. VARY STRESS PATTERNS.

"The EXCEPTIONAL cases are handled by the terminal operator." (Only those cases!)

"The exceptional cases are handled by the TERMINAL operator." (No other operator!)

2. TERM SUBSTITUTION.

Substitute the definition of the terms instead of the terms themselves.

"The 'current status file' becomes the 'old status file'."

"The 'direct access file containing the current status of all active accounts' becomes the 'sequential file of accounts carried over from the previous period'."

Notice the disparities in the definitions.

3. PICTURE THE STRUCTURE.

When a structure or a relation is described in words - try to picture it. See if the relations among the objects are unique. Try to change the relations and see if it fits the description in words.

4. EXPRESS EQUATIONS IN WORDS.

5. EXPRESS CALCULATIONS IN AN EQUATION.

6. WORK TWO EXAMPLES OF AN EQUATION BY HAND.

4. PHASE GUIDELINES

My salad days, when I was green in judgment. - William Shakespeare, "Antony and Cleopatra", Act 1 scene 5

For each phase there are specific guidelines in addition to the general guidelines mentioned above:

4.1 SOFTWARE REQUIREMENTS REVIEW top

(1) Documents required:

• Software Development Plan
• Quality Assurance Plan.
• Configuration Management Plan
• Software Requirements Document.
• User Manual Draft
• Acceptance Test Plan Draft
• Analysis and Simulations
• Applicable Standards.

• Are there written objectives?
• Are objectives measurable, clear, reasonable?
• Are 'non objectives' specified (in order to eliminate hidden assumptions)?
• Are requirements defined to the appropriate level?
• Is the connection to system requirements clear?
• Software requirements must go to a deeper level than the system requirements, but they should not address new requirements that were not mentioned at the system level.
• Were all contractual demands fulfilled?
• Is there agreement among the representatives of user, developer, and software quality assurance?
• What was the basis for understanding user needs? (Reports, specifications, interviews)
• What are the alternatives to the suggested configuration?
• Was there consideration given to support software during the decision process that led to the computer choice?
• How are system failures handled?
• What are the critical design elements?
• What is the analysis of these elements?
• Is the maintenance requirement compatible with organization capabilities?
• Are computer resources sufficient?
• (Computer resources: memory, CPU load, transmission rate, response rate)
• Check interface.
• Interface to external equipment.
• Interface to software packages and systems.
• Interface to other sub-systems.
• Check definition of the interface between the sending and the receiving unit.
• Is it defined according to standard?
• (units, range, format, scale, resolution, rate of transmission, most significant bit, defaults, meaning of discrete values, validity checks)
• Is consideration given to illegal input?
• Check special algorithms.
• Was response time analysed?
• Were global constants defined? (Like earth radius, atmospheric constants)
• Was the initial state of the system defined?
• Are recovery requirements defined?
• Are self-tests and diagnostic requirements defined?
• Is there a distinction between necessary requirements and thoughts about implementation?
• Is the description of every function complete according to the standard?
• The description should include at least:
• Input description (see interface description).
• Output description (see interface description).
• System transformations (if relevant). This has to be described from external point of view in user relevant terms.
• Constraints.
• Is system's functional flow clearly and completely described?
• Is the hierarchy of the functions described?
• Is there a complete definition of software functions requiring human and operator response?
• Is a 'help' facility provided?
• Is immediate acknowledgment to user input given?
• What is the reaction of the system to illegal input from the user?
• Are timing requirements given?
• Are memory requirements given?
• Is description of executive system required?
• What standards are available?

The documents required are:

• Preliminary Design Document
• Requirements Document
• Change proposals
• User Manual
• Preliminary acceptance test plan
• Integration plan
• Standards
• Analysis
• Hierarchical decomposition of software
• Plans for manuals and documentation
• Preliminary Design Review Checklist
• Are computer languages defined?
• Is database (external database and internal tables) defined? Is it defined according to the standard?
• Is the functional role of each table described?
• Is the structure of each table clearly described?
• Does the description include pictorial representation?
• Does the description of each item in the table include field's name, size, location, contents, level, and purpose?
• Is the scope of each table defined?
• Is there a list of modules that may change a table?
• Is there a list of modules that are using a table?
• Does every item have a source?
• Is the usage of every item unambiguous?
• Is the depth of the design appropriate?
• The depth of the design must be deeper than the decomposition depth of the requirements.
• Does each module description comply with the standard?
• A module description must include at least:
• NAME.
• FUNCTION.
• PARAMETER LIST (INPUT, OUTPUT, INPUT/OUTPUT).
• EXTERNAL EFFECT (such as printing, device handling).
• ERROR HANDLING.
• Has the decision criteria for the functioning of each module been given, whether system status, operational mode, or computes?
• Can the module be tested as designed?
• Are all conditions of system's status that call for a module response described?
• Is binding among the modules according to good design practices:

The 'coupling' of the modules should be minimized - the data relationships among the modules have to be by passing data as parameters and as elements of data and not as data structure.

The MYERS terminology for strength and coupling:

STRENGTH (weak to strong): Co-incidental, logical, classical (termination, initialization), procedural, communicational, functional.

COUPLING (strong to weak): Content, common, external, control, stamp, data.

• Is the control structure well defined?
• Reactions to faults in equipment.
• Reactions to faulty input.
• Prevention of SYSTEM DEADLOCK.
• Exit from infinite loop and unexpected wait.
• Protection mechanism for table and resources.
• Memory load. Is there sufficient reserve?
• Other resources load. Is there sufficient reserve?

1. Reserve-1 needed to cope with erroneous design estimates and
2. Reserve-2 needed for possible addition of functions in future.

• Are any additional tools necessary?
• Were any additional constraints discovered?
• Were the mathematical algorithms checked?
• Check interface from both ends - the receiving unit and the sending unit for compatibility of definition.
• External interface.
• Interface among modules.
• Check definition of data items transferred: (units, range, format, scale, resolution, rate of transmission, most significant bit, defaults, meaning of discrete values, validity checks, accuracy, quantity)
• Is the communication code (ASCII, EBCIDIC) compatible?
• Is there notification of interface failure?
• Are there provisions for rerouting data in case of failure?
• Check timing requirements. Check volume of data.
• Have overload facilities been built in?
• Check data traffic on links for loading and timing.
• Are input/output rates and the speed of device adequate?
• Are devices suitably buffered?
• Has timing of recovery procedures been considered?
• Is there a requirement for additional processor time?
• What confidence is there in the timing data?

• How will system tolerate operator error?
• How will system prevent hardware or software errors to cause loss of data?
• Are there facilities for recording system's state in the event of failure?
• Have acceptable degraded facilities been defined?
• Is 'power up' state considered?
• Is automatic reporting of failures adequately catered?
• Are there checkpoints (check sums) in software?
• Is software informing users that system has recovered?
• Is the operation sequence clearly defined?
• Check queuing scheduling and interrupts.
• Is interrupt handling compatible with the required operation sequence?
• Is scheduling algorithm dynamic? Are there facilities to alter the parameters?
• Do scheduling routines cater to degraded system?
• Can queuing be monitored (especially during system build up)?
• What was done to increase reliability?
• Were ready-made packages used? Was any check made to assure their compatibility?
• Check conformance of operating system to design.
• Is the operating system adequately documented?
• Does it provide all the services needed?
• Does it have features not required? Can they be removed to increase efficiency?
• Does it have task scheduling services, priority controls, I/O services?
• How much system overhead can be allowed?
• Were all changes incorporated in the design?
• Are planned manuals adequate?
• Are future extensions considered?
• Increase number of external lines.
• Build up of data flow greater than specified.
• Extended storage.
• Extended records.
• Extra processes acting on data.
• Is there a complete listing of symbols and definitions?

(1) Necessary documents

• Detailed Design document
• Updated documents for
• Development Plan
• Test Plan
• Requirements
• Analysis

• Is the design compatible with the principle of 'structured programming'?
• Is the 'Design Language Unit' (a symbol of flowchart, a statement in PDL or UML) small enough to enable direct coding?
• Was there any consideration given to exception condition like - division by zero, difference between the value as expressed in binary form and the true exact value?
• Is methodology clear and utilized?
• Is design of sufficient detail to allow coding?
• Are there adequate margins for min/max data values?
• Are all referenced data items - declared?
• Are data items used only after they have received a value?
• Are all declared data items used?
• Is there un-executable code?
• Are there endless loops?
• Does module logic flow from top to bottom?
• Is indentation used correctly?
• Does this module determine the values of all inputs to modules it calls?
• Does this module use all outputs returned by the module it calls?

4.4.1 DATA REFERENCE CHECKLIST

• Unset variables used?
• Subscripts within bounds?
• Noninteger subscripts legal?
• Dangling reference (reference to un-allocated storage)?
• Correct attribute when using alias?
• Record and structure attribute match?
• Structure definitions match across procedures?
• String limit exceeded?
• Off by one errors in indexing or subscripts?

• All variables declared?
• Default attributes understood?
• Arrays and strings properly initialized?
• Correct length, types, and storage classes assigned?
• Initialization consistent with storage class?
• Any variables with similar names?

• Computation on nonarithmetic variables?
• Computation on variables of different length?
• Illegal mixed mode arithmetic?
• Target size less than size of assigned value?
• Intermediate result overflow or underflow?
• Division by zero?
• Base-2 inaccuracies of floating value?
• Variable value outside of meaningful range?
• Operator precedence understood?
• Integer division correct?

• Comparison between inconsistent variables?
• Mixed mode comparison?
• Comparison relationship correct?
• Boolean expressions correct?
• Comparison of base-2 fractional values?
• Operator precedence understood?
• Compiler evaluation of boolean expressions understood?

• Multiple branches (case range) exceeded?
• Endless loops?
• Will program terminate?
• Correct loop bypasses because of entry conditions?
• Loops fall-throughs correct?
• Off-by-one iteration errors?
• DO/END statements match?
• Any non-exhaustive decisions based upon assumptions concerning the possible values a variable may have?
• Misuse of nested IF statements?
• Undefined combinations of flags?

• Parameters and argument match as to:
• Number.
• Attributes.
• Units.
• Order.
• Any references to parameters not associated with current point of entry?
• Input-only arguments altered?
• Are global variable definitions consistent across modules?
• Constants passed as arguments?
• Returned error code checked?

• File attributes correct?
• OPEN statements correct?
• Format specification match I/O statement?
• Buffer size matches record size?
• Files opened before use?
• End-of-file conditions handled?
• I/O errors handled?
• Any textual errors in output information?

• Is there a concept that can easily be expressed in plain language?
• Does the function in this part have a clear place in the overall function?
• Can the routine perform properly despite misuse?

• Is the style clean and clear?
• Is it meaningful to all class of readers?
• Are there repeated code segments?
• Are comments useful?
• Is the level of detail consistent?
• Are standard practices used?
• Is there proper initialization?
• Does the routine clean up after itself?

• Unnecessary redundant operations?
• Is storage used consistently?
• How much will it cost to modify? (Consider the three most likely future modifications.)
• Is it simple?

A judge is a law student who marks his own examination papers. - H L Mencken

The FQA is used to approve the final product. It is therefore not necessary to review the product with respect to content, this is supposed to be done during the development by internal walk-through, and periodic reviews and testing.

The FQA is mainly a formal check of the contract implementation and other requirements.

• Do all representations of the software represent the same software: Does the object correspond to source, are the tape formats, labels and checksums correct? In short, there must be a physical configuration audit.
• Were all acceptance test executed and are results acceptable?
• Have walk-throughs been executed regularly?
• Is the identification complete and compliance with the configuration management plan?
• Is documentation of the appropriate quality?
• Have all deliverable items been delivered as required?

Drama is life with the dull bits cut out. - Alfred Hitchcock

Review the document as a whole. Scan the format checklist and take notes as to adherence to requirements.

5.1. Read each section and paragraph. Scan the general checklist for each section and see if any of the items are appropriate.

Special reports and tools as the index and the glossary may be used.

5.2. For a section higher in the hierarchy the more specific checklist may be applied. These checklists address the document at a higher level or apply to the document as a whole.

5.3. If the document is written in a special language like PSL (Program Statement Language) - generate additional reports. These reports may help you answer questions like: Does every data item have source? Is every process invoked?

5.4. Note any discrepancies. Key them according to the category of checklist that is appropriate. This may later be used for statistical purposes.

6. SUMMARY top

Love all, trust a few. Do wrong to none. - William Shakespeare

The process of auditing is the primary feedback to management. Management needs to commit resources necessary for this process.

• The qualification of the auditors must be at least on the level of the developers.
• The user should participate in the auditing process.
• Tools may be of great help. It will be easier to use tools if the documents are written in a semi formal language with an appropriate processor to manipulate it, than if it is written in natural language. It may be tempting to use formal language only superficially and to resort to various features of the language like DESCRIPTION and COMMENT clauses in order to write the document in the natural language. This should be avoided. It is very difficult to process natural language by computer and very few simple tools are available. The effort of gaining proficiency of the language and of trying to express everything within its premises will be rewarded later during auditing and analysis.

1. Software Configuration Management. Edward H. Bersoff, Vilas D. Henderson, Stanley G.Siegel, Prentice-Hall, N.J., 1980.

2. Software Reliability: Glenford J. Myers. Wiley & Sons, Inc. 1979.

3. Structured Walkthrough: Edward Yourdon. Yourdon Press, Fourth Edition 1989.

4. Quality Assurance for Computer Software. Robert Dunn, Richard Ullman. McGraw-Hill, 1982.

5. Mil-Std-1521 (USAF), Military Standard - Technical Reviews and Audits for Systems, Equipment, and Computer Programs superseded by DoD 5000 system in 1995.

6. IEEE Std 830-1993 IEEE Recommended Practice for Software Requirements Specifications

7. IEEE Std 1028-1997 IEEE Standard for Software Reviews

8. Lateral Thinking: Edward De Bono. Penguin Books, 1990.

9. MIL-STD-483 Configuration Management Practices for Systems – a classic from 1970s. Also, see the Configuration Management Resource Guide.

10. Software engineering – Wikipedia 2010, The IEEE's Guide to the Software Engineering Body of Knowledge and The Joint Task Force on Software Engineering Association ... Software quality • Software quality assurance

11. Handbook of Walkthroughs, Inspection, and Technical Reviews: Daniel P. Freedman, Gerald M. Weinberg. Little, Brown and Company, 1982.

12. A Guide to the Project Management Body of Knowledge (PMBOK^� Guide), Fourth Edition, Project Management Institute, released on 31 December 2008 contains a large number of references to Audits and Reviews as Tools & Techniques e.g. Performance and Documentation Reviews, Quality, Risk and Procurement Audits.

End of Guidelines for Auditing and Reviews

A man never reaches that dizzy height of wisdom that he can no longer be led by the nose. - Mark Twain