Download this review

Author	Linda Westfall
Publisher	ASQ Quality Press
Published	2009
ISBN	978-0-87389-730-3
# of Pages	570 + supplementary CD
CQSE BOK	All

There are those books you expect will be wonderful reading and you look forward to the time doing so. These are usually fiction or non-fiction. Then there are the technical books that you expect will be boring and tedious and you force yourself to begin reading. This was a book I expected to be boring and plodding. In fact, it has turned out to be anything but that.

Linda Westfall has done a masterful job of taking material that could have been dull and boring and presenting it in such a way that it is actually interesting and extremely useful. More than being a rag-tag list of items and their accompanying examples or illustrations, the book is actually a well thought through compendium of user needs and quality-based techniques to fulfill those needs.

Yet, along with the kudos, this reviewer has taken some issues with the book. The primary issue concerns the failure to synch the contents of this book with the referenced IEEE Standard. This, it turns out, is not the author’s fault as much as it is the failure of the CSQE BOK to conform to the applicable standards. What is the author’s fault is the occasional reference to a recalled IEEE Standard.

Linda Westfall methodically follows the CSQEBOK from beginning to end as she divides this book into seven (7) parts, each part corresponding to a section of the BOK. Within each section there may be as few as two or as many as six chapters with each chapter providing information about a single topic. Following the last chapter is an Appendix containing the CSQE BOK, a glossary of terms, and an immense list of references to enhance the reader’s understanding.

The seven parts of the book include:
Part I — General Knowledge
Part II — Software Quality Management
Part III — Systems and Software Engineering Processes
Part IV — Project Management
Part V — Software Metrics and Analysis
Part VI — Software Verification and Validation
Part VII — Software Configuration Management
Appendix A — Software Quality Engineer Certification Body of Knowledge

Assessment of Parts:

Each chapter defines or describes the topic to be discussed. Each section of a chapter is introduced with the applicable text taken from the BOK. The chapter may contain definitions, examples, methodologies, and techniques that taken together form a framework for the reader. Each chapter also contains graphics that illustrate the points made in the text. Each new topic is introduced by the applicable citation from the BOK.

Part I — General Knowledge:

The topics in this section encompass: quality principles; ethical and legal compliance; standards and models; leadership skills; and, team skills. I thought the first two chapters provided a good overview of the covered topics. However I was surprised that the chapter describing standards and models did not mention the relationship of standards to the identified models. For example, the CMMI model recommends that there be a configuration management plan but the reader may not be aware that IEEE Standard 828-2005 should be used to generate a configuration management plan.

Part II — Software Quality Management:

The topics in this section include: quality management system; methodologies for quality management; and, audits.

While reading the chapter describing a quality management system, I was chagrined to discover that it addressed not a quality management system but rather described a testing overview. I thought the examples would show the totality of a quality management system rather than focus, exclusively on the test process. One of the diagrams shows the hierarchical relationships of standards, policies, process, procedures and instructions. I understand that these are components of the SQA Process; however I was looking for additional information that showed how these items link together.

Audits were covered comprehensively. However, missing was an identification of those things that might trigger an audit as well as what types of audits might be done on a routine basis and included on the project schedule. I recall one situation when I noticed that a document I was given to evaluate as part of the verification process contained a lower level identification number than the previously reviewed document of the same name. This led to a full review of the configuration management process. This review identified numerous inherent weakness of the existing process and led to numerous changes being implemented.

This section would be enhanced by including a template (outline) of an audit document and an audit report such as might be found in IEEE Std. 1028-2008

Part III — Systems and Software Engineering Processes:

The topics in this section include: life cycles and process models; system architecture; requirements engineering; requirements management; software analysis; design and development; and, maintenance management.

The first thing this reviewer found confusing was that the illustrations of the various development models did not immediately follow the textual description. For example, the text described the V-model and the illustration was that of the W-model. Following the textual description of the spiral-model was the illustration of the W-model.

A helpful graphic for this section would have been a grid that showed the distinguishing characteristics of each to the development models. For example, which ones require all the documentation to be completed in advance and which ones require the system requirements to be completed in advance. Which ones require continuous system testing and which do not.

The description of techniques for requirements elicitation was excellent, especially the detail needed to develop use cases. Also useful were the use of diagrams normally associated with object oriented programming but which are of great use for requirements engineering. Mentioning that use cases and object oriented diagrams are exceedingly useful in developing test cases for user acceptance testing as well as for other levels of testing would have been a nice inclusion.

This reviewer was disappointed to see that requirements analysis and bi-directional traceability were included in the chapter concerning requirements management rather than the chapter on verification, where they more correctly belong. Again this appears to be a problem associated with the BOK rather than with the author’s judgment. Although identifying that the change control board was responsible for handling changes to the requirements, there was no mention of the need for all requirements to be baselined and controlled. Nor was there any mention of the need not to re-use requirement numbers or to create a new requirement number whenever the wording in a requirement is changed or modified. These would be the minimum tasks required to manage the requirements.

To counter the random discussions that software maintenance activities are a separate and distinct part of the SDLC, it would be helpful for the author to have mentioned that software maintenance activities take place concurrently with operations activities. While this may be considered a truism by experience software quality assurance individuals, it may not be considered a truism by those who are less experienced.

Part IV — Project Management:

This section includes: planning, scheduling, and deployment; tracking and controlling; and, risk management.

This reviewer takes a strong exception to the author’s statement that many tasks cannot be planned to any level of detail until predecessor activities are completed. It would be more accurate for the author to state that some tasks cannot be executed until predecessor activities have been completed. However, the author then continues to describe situations in which lower level plans may have to be modified based on new information and previously unforeseen conditions. Yet, even the modification of a plan is very different than the absence of a plan or the absence of a level of detail.

If the identification of project-level standards is not accomplished during the project planning process, when will they be identified? The answer to this question is not contained within this section, yet it can be crucial to the ultimate implementation of the software development project.

Thanks go to the author for including the artifacts required for each of the Phase Gate Reviews. This reviewer has been asked to participate in numerous Milestone Reviews that lacked the required artifacts.

Part V — Software Metrics and Analysis:

This section includes: metrics and measurements theory; process and product measurement; and, analytic techniques.

Kudos to the author for making readily understandable what easily might have been not so understandable. The author has captured the heart of any metrics discussion by stating, and I paraphrase “that any metric that does not feed a decision is a metric not worth collecting. However, having collected the metric the next logical questions might be” what does it mean and how will it best be represented.

Part VI — Software Verification and Validation:

The topics included in this section are: theory of verification and validation; test planning and design; reviews and inspections; test execution documentation; and, customer deliverables.

Rather than cover the broad aspect s of verification and validation, this section makes it appear that testing covers both of these concepts. Creating an illusion that testing covers both, does a great disservice to the reader of this handbook. Unfortunately this is a problem of the BOK rather than ignorance on the part of the handbook’s author.

For a good understanding of the distinction between verification vs. validation and the tasks that go into each, the reader is urged to read IEEE Standard 1012-2004. These tasks encompass the full development life cycle, are appropriate to all types of life cycles, and recommended tasks are determined by the integrity level of the project. It further states that an integrity level may be determined by risk or by any other criterion deemed important by the organization.

Again, the author (or the BOK) considers testing to be the same as V&V by stating that each test plan “can be incorporated as part of a single V&V plan. Neither the IEEE Standard for Verification and Validation nor the IEEE Standard for Test Documentation mention this possibility. Both standards, and my own 25 years of experience, indicate that V&V will create documentation and execute tests for only software with a high integrity level and that for all other software V&V will verify the software documentation produced by others and may witness test execution. In each of these possible instances, there will be a V&V Plan and one or multiple test plans, test designs test case, and test procedure documents. There may be a Master Test Plan that describes the management and strategy of the test effort, especially if test encompasses multiple levels of test, multiple iterations of test, and multiple systems.

The chapter covering review and inspections may do a disservice to the reader only because it may lead the reader to believe that having a peer review ameliorates that need for verification and validation to evaluate the contents of the product against one or more agreed upon standards.

Part VII — Software Configuration Management:

The topics included in these last chapters include: configuration infrastructure; configuration identification; configuration control and status accounting; configuration audits; and product release and distribution.

I was pleased to note that the description of configuration management tools and their uses included those that allowed related documents to be baselined along with the source code. Documents that might be baselined include requirements, design, and test. The ability to roll-back both code and related documentation is extremely important for organizations that may have multiple versions of code in use.

Conclusion:

The writing of a guide for the CSQEBOK started many years ago when the ASQ Software Division decided it would be helpful for the members to have such a guide. The project was begun and abandoned numerous times until Linda Westfall accepted the challenge. In the preface the she states that this book was written not only to be a guide in preparing for the CSQE exam but also as a resource for software quality engineers and others who need to understand how software quality impacts their work. In both of those objectives, Linda Westfall has succeeded in a bountiful fashion.

This book now occupies a prominent place on this reviewer’s bookshelf.

First published in Software Quality Professional, Sept. 2010

Download this review

Author	Laura Pullum, Brian J. Taylor, Marjorie A. Darrah
Publisher	John Wiley & Sons, Inc.
Published	2007
ISBN	-13: 978-0-470-08457-1-10:  047008457X
# of Pages	128
CQSE BOK	Standards and Models, Systems and Softwre Engineering Processes,Software Verification and Validation, Process and Product Measurement

While traditional software performs its intended function without changing its behavior or its functionality during normal system usage, this is not the case for adaptive software.  Adaptive systems, neural networks being only one example, can and do change their behavior, functionality, internal parameters or realization while in their normal operational state.  Thus answering the question of how to perform verification and validation (V&V) on adaptive systems looms as an imperative.  That, each adaptive system may have its own intrinsic V&V problems and some adaptive software algorithms are so new that they have never been applied to production or high-assurance projects creates a dilemma for the V&V practitioner.

Answering the question and the dilemma are the foundation for this book which definitely fulfills the need with its useful and practical guidelines accompanied by wonderful examples.  This book provides a foundation for guidance to V&V practitioners, who may be confronted with the various types of adaptive systems whether these be neural networks, genetic algorithms, fuzzy logic, machine learning, expert systems, autonomous planning software, formal logic or some combination of these.

The guidance provided is based on lessons learned and applied research from the development and V&V of a safety- and mission-critical adaptive flight control system.  In presenting this guidance, the authors rely on their V&V research, experience with adaptive systems and the lessons learned from practitioners at the NASA IV&V facility.  They have bolstered their research and experience with pertinent information derived from basic and applied research and industry publications.  Parts of the guidance are applicable to any form of adaptive system and other parts target neural networks, specifically.  The book has been written to align with the IEEE Standard for Software Verification and Validation, IEEE Std 1012-1998, thus confirming the versatility of the standard.  As the authors state just as IEEE Std 1012 is intended for software being developed, maintained, or reused, so is this guidance.  The producer of adaptive software should consider V&V as part of the adaptive software life cycle process and not something done solely at the end of development.

This small and powerfully written book contains a scant four chapters.  Chapter One provides an introduction and overview of the book.  Chapter Two provides a summary of the areas of consideration when using adaptive systems/neural networks.  Chapter Three provides the detailed, specific guidance for all life cycle processes, activities and tasks related to adaptive systems/neural networks.  The organization of the chapter facilitates mapping (cross-walking) between the standard and the provided guidance.  Chapter Four provides a high-level description of the updates in IEEE Std 1012 from the 1998 version to the 2004 version.  It describes how these updates impact the V&V practitioner using the guidance provided in this book.

In Chapter Two, the authors identify numerous areas that need special consideration from the V&V practitioner.  These include, but are not limited to: systems requiring special attention to prevent harm to humans or to the environment; system control and knowledge requirements; system’s training set analysis; and, stability analysis. Each of these areas is fully described and examples are provided.  In addition the chapter provides, for each area, information on situations that are more or less appropriate for each identified adaptive system.  The authors state that it is important, early in the project, to conduct an assessment to determine the appropriateness of using an adaptive system or neural network.  They continue, “[neural networks]” are like multipurpose tools–generally used in statistical problem and  different architectures are usable for different problems types.

While Chapter Two presents a mini-tutorial on adaptive systems especially neural networks, Chapter Three follows this up by presenting all of the activities and tasks required to evaluate these systems and  places the activities and tasks into the framework of IEEE 1012-1998.  For the V&V practitioner this chapter forms the heart of the book.  For each task recommended by the standard, the authors describe a specific adaptive system perspective that emanates from the information provided in the prior chapter.

The chapter continues with examples of high-level goals and system requirements for neural networks related to Control and other system requirements related to Knowledge.  This allows the reader to follow the required thought and relationship process.  In the discussion related to the risk analysis and hazard analysis tasks, the authors include a synopsis of a case study in which nine (9) hazards associated with neural networks are identified.  Each of these hazards is then evaluated for frequency of occurrence and severity of failure to plan for sufficient mitigation. The authors also identify a number of metrics that have proven useful in determining the pass/fail performance of the neural network as a whole.  There is a definite relationship between these metrics and the included table identifying and describing the neural network quality characteristics.  Later in the life cycle, the authors suggest test plans scope and test designs for the test levels (acceptance, system, integration, and component) identified in the standard.

Prior to reading this book, this reviewer had no specific knowledge either of adaptive systems or neural networks other than a lay person’a understanding of swarming theory and adaptive technology as used in flight-systems, prototype automobiles, and science fiction books.  However, after reading this book I had a bit more understanding of the importance and use of adaptive systems.  As an independent verification and validation (IV&V) practitioner I had an extremely good general understanding of the need for and the complexity of performing V&V on adaptive systems, especially neural networks.  This was best stated by the authors when they wrote traditional software development does not adequately address the development of adaptive systems, specifically neural networks because one does not write code for these systems, rather they are formed as they are trained or in real-time as they adapt; and because the risks associated with neural networks are heavily tied to the development activity.

This book was not an easy read because of the complexity of the subject matter.  However the authors did a masterful job of explaining and describing each step along the way.  Unlike many authors they did not skip any steps in explaining the concepts and constructs.  Because I wished that they had included some practical examples of adaptive software systems I contacted the leading author and requested that such information be provided for this review.  I had expected some esoteric or futuristic applications.  Instead I learned that there are actual working and practical examples of adaptive software systems.

At the time this book was written, there was no similar book available.  To this reviewer’s knowledge, there is still no similar book available.  Whether readers want to have a general understanding of adaptive software systems or whether they are required to verify and validate any type of adaptive software system, this is a book that must be read and kept close at hand for easy reference.

Download this review

Author	Manish Gupta & Raj Sharman
Publisher	Information Science Reference
Published	2009
ISBN	978-1-60566-036-3
# of Pages	383
CQSE BOK	Systems architecture,requirements engineering, risk management

Imagine my surprise when I saw that this book is actually a collection of essays written by a wide variety of people. This book brings together articles from researchers and practitioners in the financial, legal, technology, information security fields through original papers on all aspects of roles and effects of human and social dimensions of information security. As stated in the Preface, the key objective of this book is to “fill the gap in existing literature on human and social dimensions of information security by providing the readers one comprehensive source of latest trends, issues and research in the field.”

Each paper constitutes a chapter within the section. As a result the same or similar concepts are presented with different perspectives. I find this very enlightening.  The papers are arranged in sections with each section containing 5 or more papers authored by persons representing academics and practitioners from the international community. The common thread through each section concerns the human element.

Section I: Human and Psychological Aspects – This section begins with  the notion of humans and their frailties as related to the security practices for individuals, businesses, and organizations. It continues with the impact of humans on information security and why humans make poor security decision. The last paper in this section describes the incompatibility of software quality assurance procedures with the use of automatically generated code.

Section II: Social and Cultural Aspects – Beginning with a description of the complex nature of information security culture in a networked environment and a paper that examines the knowledge that might be available if both technology and human activity were seen as being equally important, this section brings together diverse elements of discourse. These papers are followed by a paper discussing social engineering as a technique for compromising information systems. The final paper in this section describes a model of a social paradigm for security and software engineering.

Section III: Usability Issues – The difficulty of using security configuration interfaces and how those interfaces can be improved is the subject of this first paper. It is followed by a paper describing the need for and the challenges of security usability and a paper describing the need and techniques for distinguishing between humans and automated computer programs on the Internet. One of these techniques is the use of CAPTCHAs. The last paper argues that it is possible to find a good compromise between quality of predictions and protection of personal data.

Section IV: Organizational Aspects – This final section begins with a paper describing the incorporation of human and social factors in the threat-vulnerability model of risks and the management of vulnerabilities. Following this are papers addressing workplace monitoring and its implication for privacy concerns and aligning risk management with business requirements as a strategy for developing effective enterprise information security management. The ending papers highlight the issues resulting from the coalescence of system requirements elicitation, information security, and human factors and then the management of information as a critical corporate asset.

In addition to a detailed Table of Contents providing an overview of each chapter within each section of the book, the authors have included a Forward that lays the foundation for this book and its very contemporary and relevant papers. The theme is that although many organizations are relying purely upon technical solutions to implement their security policies this is an inadequate solution. The authors believe that it is a lack of understanding that prevents them from addressing security from the people, processes, and technology standpoints to implement a successful security strategy. This book is an attempt to close the information gap between technology and human factors.

By providing high quality research papers and industrial and practice articles on social and human aspects of securing information systems and infrastructure from social engineering attacks and real-world implications and implementations (practice) of the research the authors achieve their objective.

Introduction:

I was attending a networking meeting not too long ago. As is the custom I circulated through the crowd, introduced myself, and attempted to hand my business card to everyone I met and to get their card in return. At some point, someone announced that we were all to get into a circle and introduce ourselves and our companies to everyone in the group. This was the time for the three-minute elevator speech with which I present my hopes and aspirations in a short period of time. I took a deep breath. When my time came I presented it just as I had rehearsed it. I was well spoken and succinct. However I was not brilliant because I had neglected to take the audience into account. I neglected to frame my information for a non technical audience. As a result no one understood what I did or what kind of contacts I needed.

Afterward, I had the opportunity to speak with a number of people and to explain exactly what kind of contacts I needed and what I did. Since that day I have been striving to put the words of that one-on-one into another elevator speech. The speech should summarize the nature of the services offered, the benefits to the user, and the type of organization that could benefit from these services.

Services:

We provide a variety of IV&V services. The important thing is that we are independent. That means that we have a budget that is independent from the development budget, a management that does not report to the manager of development, and that the staff performing IV&V tasks is not the same staff as those performing development tasks. As a result we are able to provide non-biased feedback to the client organization. We base our assessments on consensus standards and so the feedback is both substantive and informed. We provide feedback on both the quality of the product and the likelihood that the system being produced will meet the business needs of the organization.

Our IV&V services include: Management Support; Engineering Support; Test Support; Review and Audits Support; and Process/Procedures Support. Thus the client organization can acquire all or some of these services. The details of the specific tasks included in each of these services can be found in another “musings on the deck” — IV&V Services, published on July 4, 2008.

This means that in each of these categories of IV&V services we evaluate documents and other products that provide a foundation for the system under development.

Management Support Services

For example, the management category generally involves program/project plans, schedules, milestones, and proposals. It also may include various management plans such as quality assurance and configuration management. By evaluating these, IV&V identifies inconsistencies, errors, or other issues that may present a risk to the program/project, thus providing management with an early identification of potential problems.

Engineering Support Services

In the engineering category, IV&V evaluates individual technical/development products to insure that the content, at a minimum, conforms to the applicable IEEE or other consensus standard. IV&V also evaluates the adequacy and completeness of the products based on good engineering practices. One of the earliest assessments performed by IV&V and the most important is the assessment of the system requirements because these are incorporated in the subsequent design documents and ultimately into the software code or the computer hardware. These evaluations identify the adequacy, completeness, consistency and correctness of the content so that technical risks are identified prior to the creation of software code or manufacture/assembly of computer hardware.

Test Support Services

In the test category, IV&V evaluates developer and/or integrator test documentation including plans, test cases and test procedures. These are assessed against the applicable IEEE or other consensus standards for consistency, completeness, and correctness. IV&V determines that the test documents correctly link to each other and correctly reflect the business needs, security, performance and other requirements based on the level of testing. Additionally, IV&V often witnesses the actual testing performed by the development and/or integration contractor. After testing is completed, IV&V may assess the test report prepared by the development and/or integration contractor. As a result of the test activity, IV&V identifies possible risks, if any, for taking the system under test, into production.

There may be occasions when IV&V is tasked to perform its own testing. Whenever this occurs, IV&V must generate its own test plans, test cases, and test procedures. Once test execution is completed IV&V will write its own test report.

Review and Audit Support Services

Although projects may have many reviews and audits these are typically either management reviews or technical reviews. Management reviews may be performed at different intervals by various level of management and are most often concerned with schedule and budget issues. IV&V often provides input concerning the adequacy of project artifacts or lifecycle processes, in the form of reports or presentations, in support of these reviews. IV&V may provide a special report for that specific review or an IV&V End of Activity Report may be provided in support of the management review. Management has the responsibility for deciding if the project should go forward and if so  what is the risk or the cost and if the project should not go forward  what is the risk or the cost.

Technical reviews are sometimes called milestone reviews. Technical reviews are held based on the project schedule and/or at the completion of one or more specific products. IV&V provides input, in the form of task reports, in support of technical reviews. Technical reviews are held to determine if a specific product meets its requirements and to determine if the project can move to the next activity based on completion of approved criteria. One technical review is often called the critical design review. This event determines if the requirements and the design documents are complete and that the project can move to the implementation (code generation) process. The technical review provides project manage to assess the technical risks involved with going forward.

One type of audit may examine a process and the results of that process such as a Configuration Audit. Another type of audit may compare the system as-built to the system as-tested  such as a Functional Configuration Audit. IV&V may participate in these audits.

Benefits:

An immediate benefit of IV&V is that upper management is made aware, on an on-going basis, of potential risk to the budget or schedule while project management is appraised on a continuing basis on the quality of the deliverables. Some deliverables, if defective, will have substantial negative impact on the ability to build the right system and to do so in a correct manner. Other deliverables, if defective, will have a substantial negative impact on the ability to monitor and control the project. All defective deliverables should be corrected as early in the life cycle as possible in order to mitigate the risk to the project and/or system.

A more long-term benefit of IV&V is the avoidance of costs associated with defective systems or software. These costs might include excessive maintenance and support costs due to an increase in HELP requests and/or the unplanned software releases. Additional costs may include those associated with the creation of a negative corporate image among potential customers based on negative reviews by professional review organizations or bloggers. There may be future costs associated with current customers leaving and going to competitors. Finally, defective systems or software may result in lawsuits for breach of contract or even damage to organization assets.

The cost of IV&V over the development life cycle is said to average 15%; however, once the cost of IV&V is spread over the life of the system or software then the actual cost is significantly lower. A study at the NASA IV&V facility compared the Return on Investment (ROI) of IV&V on several different projects to be 1.9 on the low side and 4.9 on the high side.

Benefitting Organizations:

If you own or represent an organization that produces software for its own use or for others your organization will benefit from allocating funds for IV&V because testing the product to assure its functionality only assures the functionality of the system or software as-built. It does not assure the functionality that should-be. IV&V will assure the functionality that you need is present.

If you own or represent an organization that acquires systems and or software produced by others you will benefit from allocating funds for IV&V because you are not in the business of developing systems or software and likely have neither the skills nor the time to oversee the development. At the same time you probably have a low level of trust that the development contractor will deliver what you need and what you paying for without stringent oversight of the development products. IV&V will provide the oversight so that your get the system or software that you need and that you are paying for.

You can acquire as many or as few IV&V services as you need or your budget permits. No matter how much or how little IV&V you acquire you will enhance the quality of your product.

It has been more than 10 years since The Office of Management and Budget (OMB) issued Circular A-119 (Revised) directing federal agencies to use voluntary consensus standards, both domestic and international, in its regulatory and procurement activities. This circular defined voluntary consensus standards as having the following attributes: openness; balance of interest; due process; an appeals process; and consensus. Standards that meet these criteria include ISO, IEEE, and ANSI standards. It was expected that the agencies, including the Department of Defense, would cease using their own agency-specific standards which were often modeled after the proprietary standards of the contractors hired to develop those standards.

Although the circular is not specific as to its use in the external development of systems and/or software, it is specific in stating that the agencies must use voluntary consensus standards in its regulatory and procurement activities. For example, agencies could, if they desired, state via the RFP that their evaluation of a contract deliverable would be based on its conformance to the applicable voluntary consensus standard. Since the release of Circular A-119 (Revised) few if any of the federal agencies have attempted to bring their System Development Life Cycle standards into alignment with the circular. Recently, OMB issued Circular A-11 which references the use of voluntary consensus standards in Section 53. However, legacy systems continue to be upgraded and new systems continue to be built with little or no standardization across agencies or within individual agencies.

Introduction:

Assuming the federal agencies want to conform to OMB Circular A-119(Revised), the unanswered question is – what precludes conformance? Some possible answers include: it will be too expensive; our operations and maintenance contractors won’t cooperate; our CPIC group doesn’t care how IT is done; our project managers don’t know how to do it or think it will take too much effort to implement. These arguments are all realistic albeit not very logical.

Perhaps the unspoken reason is that the federal agencies do not know how to implement the use of voluntary consensus standards. This paper is specific to the use of such standards in the development of systems and software and will propose several implementation options ranging from directive to consensus.

Implementation Options:

The implementation that is correct for a particular federal agency is the one that is in-synch with the agency culture and its way of managing IT. One agency may feel comfortable with a top-down approach whereas another agency may be more comfortable with a bottom-up approach. Yet another agency may find that a combination or hybrid of these two diverse approaches is most effective. This paper describes four models – Directive, Project, Consensus, and Hybrid.

Directive Model

Description: The CIO determines that it is advantageous for the agency to adopt voluntary consensus standards and signs that directive. The CIO staff will then develop an implementation strategy and guidelines for implementation. The directive is binding for all internal system or software development. Although OMB Circular A-119 does not mandate the use of voluntary consensus standards by external contractors, it does mandate their use in regulatory and procurement activities, and the CIO could make a determination that the agency will evaluate contractor deliverables against these standards. Also, the CIO could make the determination that any externally produced system utilizing the agency backbone or connecting to an in-house system must be developed using voluntary consensus standards.

Advantages: The specifics of implementation are specified. Project managers know what they have to do. All members of projects immediately are held accountable. This model is best used by an organization with a well-defined chain-of-command.

Disadvantages: Some IT program managers may push back because they have not been consulted and/or because their programs are developed outside of the CIO’s arena. They may neglect to use the standards or they may not monitor the use of the standards. There is no opportunity to validate the guidelines before they are implemented.

Project Model

Description: Pilot the transition to voluntary consensus standards using a highly visible project to learn from and improve the implementation process. In this model, the program manager declares that voluntary consensus standards are to be used on the project, and the applicable voluntary standards are provided by the agency. As a result, all documentation generated by the agency is based on voluntary consensus standards and the RFP states that all contract deliverables will be assessed and accepted or rejected based on voluntary consensus standards.

Advantages: Using a single project reduces the potential impact of an ill-planned implementation. A single project allows for the validation of the concept and provides a feed-back mechanism that enhances the final implementation across all IT projects. This ultimately gains buy-in from the stakeholders.

Disadvantages: Other projects continue to have the option of not using voluntary consensus standards. A highly visible project that requires more than a year will delay implementation of the use of voluntary consensus standards throughout the agency’s IT projects.

Consensus Model

Description: A champion, whether at the level of the CIO or lower down, introduces the idea to relevant managers. In an agency with a high level CIO and division level CIOs the relevant managers would be the low-level CIOs. In another agency, the relevant managers may be the IT Program Officers. Once that group agrees that the use of voluntary consensus standards would be beneficial, they may request the CIO to issue the policy statement and to have staff develop implementing procedures and guidelines. In addition to achieving consensus on the need for the use of voluntary consensus standards, the consensus model may include the need to achieve consensus on the implementation procedures and guidelines.

Advantages: The consensus model ensures that all stakeholders have a sense of ownership in the process to be followed. This is especially true if the consensus model is used for the development of the implementing procedures and guidelines as well as in the selection of the model. The consensus model may reflect the requirements of small as well as large and complex projects.

Disadvantages: The process of achieving consensus may require the inclusion of a waiver process that dilutes the potential impact of using the voluntary consensus standards. The time required to affect this model may be the lengthiest of any of the options as stakeholders struggle to exempt their area or to determine criteria for exception.

Hybrid Model

Description: A hybrid model may involve the Directive Model and the Project Model, or it may involve the Consensus Model and the Project Model. In the first instance, a highly visible project may be identified to implement voluntary consensus standards while the CIO staff is developing the procedures and guidelines for the agency. In the second instance, a highly visible project may be identified to implement voluntary consensus standards while the management team is achieving the consensus required to bring the use of voluntary consensus standards to the agency as a whole. Alternatively, an agency may identify a highly visible project as a way of implementing procedures and guidelines that have been drawn up as a result of either the directive or the consensus model.

Advantages: The hybrid model provides the opportunity to implement the model in a limited fashion while mitigating potential risks associated with an all-agency implementation. The model affords the organization needed experience to modify the method of implementation, as well as the specific procedures and techniques based on the experience of the highly visible project.

Disadvantages: A highly visible project ,that requires more than a year to complete, may delay implementation of the use of voluntary consensus standards throughout the agency’s IT projects. The relative ease of implementing voluntary consensus standards in a single project compared with the difficulty of an all agency implementation may deter the all-agency implementation.

Summary:

OMB Circular A-119 directs the federal agencies to remove themselves from the business of generating and maintaining standards. The availability of voluntary consensus standards allows them to do just that. The use of voluntary consensus standards allows the federal agencies to manage their IT staff and contractors better by setting a consistent level of expectations for them and the agency program manager. The OMB Circular does not specify the method by which the agencies are to implement the directive to use voluntary consensus standards. Rather the OMB Circular leaves it up to each federal agency to determine the best way to implement the requirement.

This paper has defined four models (Directive, Project, Consensus, Hybrid) that can be implemented. In addition this paper has described some advantages and disadvantages of each model. It is up to the agency to identify the model that best suits its needs and its culture. Once the agency has selected this model, they may desire to use an outside contractor to develop the process and procedures required for implementation.

Both in the classroom and on the client site I am asked to describe the difference between test and verification and validation. It is always difficult to explain that test performs a number of test tasks and that sometimes verification and validation performs duplicative test tasks.

Test is responsible for a wide-range of test tasks that occur throughout the system development life cycle. Early in the life cycle test will evaluate the requirements (system, software, interface) to insure that they can be validated. Test will become familiar with the concept documentation and the business needs. Later in the life cycle test will, based on the integrity level of the system, determine the necessary breadth and depth of testing. After preparing the required test documentation test will commence testing. This may result in testing each component, the integration of the components, system testing against the system requirements, and acceptance testing to validate that the system meets user needs. Or a determination may be made that test will do only acceptance testing while development is responsible for the earlier testing. After testing is completed test may prepare interim test reports or merely a final test report. These and other test tasks are fully described in IEEE Std. 829-2008.

If the development effort has a verification and validation component then, based on the identified integrity level of the system, verification and validation will evaluate the adequacy of the test tasks and/or perform its own test tasks in addition to the ones performed by test. If the verification and validation evaluation tasks are performed by persons outside of the development structure, budget, and management then they are called “independent.”

The Verification and validation component will accomplish various tasks in addition to evaluating the adequacy of the test tasks. The verification and validation tasks entail assessing the adequacy of development tasks. This determination is based on the assessment of development artifacts against applicable voluntary consensus standards (IEEE, ISO, ANSI). An assessment of the system development process would use ISO 15288 while an assessment of a requirements document would use IEEE Std. 1233. In addition to assessing the artifact against the applicable standard Verification and Validation assesses the artifact in terms of good engineering practices. Thus an assessment of a design document would not only include the applicable standard but also would include the flow-down of the architecture document, the business needs of the organization, and the system or software requirements. This engineering perspective distinguishes Verification and Validation from Quality Assurance. Another musing will provide additional discussion on the intersect of Quality Assurance and (Independent)Verification and Validation.

Information on determining the integrity level is fully described in IEEE Std. 829-2008 and IEEE Std. 1012-2004. If merited by the identified integrity level, verification and validation will also engage in testing. Their testing may be in parallel with that being done by test or it may be after that done by test.

Just as test has tasks that are performed within the context of the development life cycle so does verification and validation. Only some of these tasks will intersect with the tasks being performed by test. And just as the test tasks will vary according to the identified integrity level of the system being developed so will the verification and validation tasks vary according to the identified integrity level. The table below shows the intersection of recommended tasks for a project with a level 3 criticality level. Projects with lower levels, indicating they are less critical, will have fewer tasks. The selected system development life cycle phases are only examples. Every project will have its own life cycle phases.

Phase: Management

Phase: Acquisition

Phase: Supply

Phase: Development/Concept

Phase: Development/Requirements

Phase: Development/Design

Phase: Development/Implementation

Phase: Development/Test

Phase: Development/Installation & Checkout

Phase: Operation

Phase: Maintenance

As can be seen above verification and validation is responsible for many of the same tasks as is test albeit from a different perspective. Both efforts have to be planned, managed, and executed. In both instances there will be interfaces with other organizational and supporting entities.

For systems with a lower integrity level verification and validation only may review the documentation created by test and witness test execution. For systems with a higher integrity level, verification and validation may review the documentation created by test and generate its own test documentation. It may witness test execution and perform its own tests based on its own test documentation.

Conclusion:

Even though many organizations think of verification as being the same as test, I have shown that for high integrity systems both should be implemented. For low integrity systems that provide a low risk to the acquiring organization having only project level testing may be adequate.

For about twenty years, I have been involved in Information Technology (IT) in both the public and private sectors. I have been a business analyst, a software tester, a test manager, and an independent verification and validation (IV&V) manager. I have taught software quality assurance at a Washington DC University. I co-authored the IEEE Std 829-2008, System and Software Test Documentation and I was involved in the development of the IEEE Std 1012-2004, Standard for Verification and Validation and I am currently part of the working group that is developing IEEE Std 1012-x that will enhance best practices in software verification and validation.

Many federal government agencies are looking at independent verification and validation (IV&V) as the panacea to all their ills — or at least to some of their ills. Much of this interest in IV&V is driven by a new breed of CIO’s who understand the business value of having IV&V as well as by members of Congress who attach a requirement for IV&V to funding for large IT acquisition. As a result, requests for proposal (RFP) are being issued for IV&V services. Unfortunately for the acquirer and the potential supplier the requests and the resulting statement of work are either very general such as “provide IV&V services” or overly detailed such as “provide an approach for completing the IV&V strategy,” or “provide an overview of what will be contained in the IV&V Plan.” The later came from an organization that expected IV&V products to be consistent with IEEE Std. 1012-2004.

IV&V Taxonomy

After a number of years of these recurring experiences, I have come to the conclusion that the agencies know what they want but they don’t know how to ask for it. To resolve this problem I have developed a taxonomy of IV&V services. This taxonomy is not complicated and involves only five categories of IV&V services. These include: management and support; engineering documentation; test documentation and test witnessing; reviews and audits; and process and procedures. Within each category are a number of specific services. Presenting these categories with specific services to the IV&V sponsor and/or to the contracting office ensures a “win-win” situation for both the acquiring agency and the supplying contractor. The taxonomy provides a means for the acquiring agency to articulate its needs and it provides specifics that allow the supplier to meet those needs. These categories and their contents provide an orderly core set of IV&V services for selection and allow the acquiring agency to identify services they may have overlooked.

In the management and support category the acquirer can choose from:

• Assessment of the Statement of Need
• Assessment of the Program/Project Plan
• Assessment of the Configuration Management Plan
• Assessment of the Quality Assurance Plan
• Assessment of the Security Plan
• Assessment of the Master Test Plan

Just as the Program/Project Plan describes how the program/project will be managed including the organization and identification of the high-level activities so does the Master Test Plan describe the organization of the test effort and the identification of the high-level activities. Additional services in this category may include an assessment of the certification and accreditation package prior to submission for approval or an assessment of the transition plan. It may include an assessment of the project schedule or even the EVMS process. IV&V may be requested to evaluate the Program Management Office (PMO) and its processes or even the IT acquisition process.

Within the engineering documentation category the acquirer can choose from:

• Assessment of System Requirements
• Assessment of Architectural Design
• Assessment of Interface Requirements
• Assessment of Software Design
• Assessment of Database Design

Additional services in this category may include an assessment of the conversion plan, the migration plan, the correctness of data conversion, or the integration plan.

In the test documentation and test witnessing category the acquirer can choose from:

• Assessment of Test Plans (integration, system, acceptance)
• Assessment of Test Designs ( integration, system, acceptance)
• Assessment of Test Cases (integration, system, acceptance)
• Assessment of Test Procedures (integration, system, acceptance)
• Assessment of Test Execution (integration, system, acceptance)
• Assessment of Test Reports (integration, system, acceptance)

Within this category the acquirer may be interested in an assessment of the test logs or even the anomalies that have been generated throughout the development process. The assessment of test execution will include whether or not each test run provided the expected results and whether an anomaly report was written if it the test did not provide the expected results. IV&V may also report on how well the test process as described in the documentation was followed especially regarding test suspension and test resumption.

In the review and audits category the acquirer can choose from:

• Support of Requirements Review
• Support of Preliminary and Critical Design Review
• Support of Test Readiness Review (integration, system, acceptance)
• Support of Function and Physical Configuration Audit
• Support Management Reviews

Each of these reviews and audits may be supported by IV&V. When supporting a review or audit, IV&V will determine if the required inputs have been base-lined prior to the review or audit. For example, IV&V will ascertain whether the requirements documents and the applicable design documents have been base-lined prior to the Critical Design Review which assesses the detailed design documents. Applicable IV&V task reports for each the documents provided for each review will be included. IV&V also will participate in these technical reviews.

The acquiring organization may request that IV&V participate in management reviews.If so, the acquiring organization will identify whether it wants participation at all project level management meetings, selected meetings, invited participation at senior management reviews or some other combination of regular and ad hoc management meetings.

In the process and procedures category the acquirer can choose from:

• Assessment of the Configuration Management Process
• Assessment of the Quality Assurance Process
• Assessment of Test Process
• Assessment of the Development Process

Conclusion

Not every organization or every project will require the full range of IV&V services. For example, some projects will be development projects while others will be up-grade projects. However, by categorizing the possible services the acquiring organization will be in a better position to identify if it wants IV&V for its engineering effort or for its process evaluation. Once the needed category has been identified the acquiring organization will then understand what specific core services are available within each category and will be able to select those that are best suited for the project or program at hand.

Download this review

Author	Mary Beth Chrissis, Mike Konrad, and Sandy Shrum
Publisher	Addison-Wesley
Published	2007
ISBN	0-321-27967-0
# of Pages	595 plus appendices
CQSE BOK	Software Engineering Processes; Program and Project Management

There is a plethora of maturity models, standards, methodologies, and guidelines available to help organizations improve the way they do business. Most of them focus on a specific part of the business and do not take a systematic approach to the problems faced by the total organization. The CMMI provides an integrated model that transcends disciplines. It addresses practices that cover the product (or service) life cycle from conception through delivery and maintenance. CMMI emphasizes the work needed to build and maintain the total product (or service).

If you are new to process improvement or perhaps new to the concepts governing CMMI then you should read Chapter 1 because it provides an overview of process improvement and describes CMMI. Then you should skim Part Two, describing both generic and specific goals and practices. After that you may want to review the references in Appendix A to identify additional sources of beneficial information. Follow this with a reading of the acronyms and glossary to become familiar with the language of CMMI and then return to Part Two.

If you are new to CMMI but you do have experience with other process improvement models, you will notice many similarities. Read Part One to learn how CMMI is different from other process
improvement models. Then as you read Part Two notice the best practices from other models with which you are familiar. Study the tips, hints, and cross references to see details and relationships that will assist in understanding CMMI.

If you are not new to CMMI you will quickly recognize the concepts and best practices. Focus on the tips, hints, and cross references in the process areas (Part Two) to discover ideas, relationships or details you may have overlooked previously.

Part One of this book describes the various approaches and models that govern the capability maturity model. It also describes the process area components and the multiple maturity levels. These are then followed by a description of the relationships among the process areas and information on using the CMMI models. The final section of Part One is a case study showing how Raytheon applied CMMI to services.

The authors have included practices that cover project management, process management, systems engineering, hardware engineering, software engineering, and other supporting processes used in development and maintenance. In addition, the construct may cover the use of integrated product teams for maintenance and development activities. The practices described are used by organizations in many diverse industries, including aerospace, banking, computer hardware, software, defense, automotive, and telecommunications.

There are 22 CMMI process areas and each one consists of required components, expected components, and informative components. Required components describe what the organization must do in order to satisfy the process area. Expected components guide those who implement improvements or perform appraisals. Expected components include the specific and generic practices.

The authors treat each process area as a component; and each one, in alphabetic order, consists of:
* Introductory notes that describe the major concepts covered in the process area.
* Related process areas lists references to related process areas and reflects the high level relationships among the process areas.
* Specific goals describe the unique characteristics that must be present to satisfy the process area.
* Generic goals describe the characteristics that must be present to institutionalize the processes that implement the process area.
* Specific goal and practices summary provides a high-level summary of goals and practices that must be accomplished. This is informative.
* Specific practices describe the activities that are considered important in achieving the specific goal.
* Typical work products lists sample output from a specific practice. This is informative.
* Sub-practices provide a detailed description that provides guidance for interpreting and implementing a specific or generic practice. This is informative.
* Generic practices contain a description of an activity that is important in achieving the generic goal.
* Generic practice elaboration provides guidance on how the generic practice should be applied to the process area.
* Supporting informative components provide further information to describe the concept. This may be in the form of notes, examples, amplifications, or references.
o Notes are text that provides detail, background, or rationale.
o Examples consist of text that clarify a concept or described activity.
o Amplifications are notes or examples that are relevant to a particular discipline.
o References are a pointer to additional or more detailed information in related process areas.
* Additions can be informative material, a specific practice, a specific goal, or a process area that extends the scope of the model or emphasizes a particular aspect of its use.
* Specific work products describe artifacts that are developed while implementing the specific practices.

While Part One provides the models for implementing CMMI, Part Two describes in detail all of the generic goals and practices of CMMI. It focuses on those model components that address process institutionalization. Institutionalization implies that the process is ingrained in the way the work is performed and there is commitment and consistency in the way the process is performed. The progression of processes is as follows:

* A performed process accomplishes the work necessary to produce work products. This is equivalent to a Level 1 process.
* A managed process is a performed process that is planned and the performance of the process is managed against the plan. This is equivalent to a Level 2 process.
* A defined process is a managed process that is tailored from the organization’s set of standard processes and provides a basis for planning, performing, and improving the project’s tasks and activities. This is equivalent to a Level 3 process.
* A quantitatively managed process is a defined process that is controlled using statistical and other quantitative techniques. This is equivalent to a Level 4 process.
* An optimizing process is a quantitatively managed process that is changed and adapted to meet relevant current and project business objectives and is continuously improved by addressing common causes of process variation. This is equivalent to a Level 5 process.

Because of my own interest in verification and validation, these were the first process areas I turned to. Verification is the process of determining that something meets its requirements and validation is the process of determining that something meets the user needs and is usable by the user. At first look, they appeared to be correct and with an appropriate level of detail. However, further examination led to dismay when I noted that under SP 1.1 Select Work Products for Verification it identified various types of testing as work products. The Verification process should have identified the test artifacts (e.g., Integration Test Plan) for each type of testing as work products and should have stated that the various types of testing actually are encompassed in the Validation process. One of the “tips” in this section states that verification often means testing. Testing is a technique, not for verification, but rather for validation as are inspection, demonstration, and analysis.

Artifacts are typically verified against applicable standards. Artifacts such as a training guide or an operations manual may first be verified to determine that the contained information is adequate and complete and then be validated to determine that the information is useable. This reviewer considers it problematic that this book is at odds with verification and validation as described in IEEE Std 1012-2004, IEEE Standard for Software Verification and Validation. Systems and software are validated. In addition, the book, in the Part Two section describing the Validation process states that the end users are typically involved in validation. This is true when validation is for User Acceptance Test. However, it is not true when validation involves integration, system, performance, or stress testing.

While other process areas appear to conform to “best practices” I was disappointed that the authors frequently recommend that the process information be documented in the Project Plan. A process such as Configuration Management or Process and Product Quality Assurance might be better served by having its own plan. Perhaps the need for separate plans, as appropriate, and the identification of consensus standards to assist in the preparation of the plans will be addressed in the next version of this book.

Yet even with these points of disagreement, I still recommend this book because of its depth and detail of coverage.

Download this review:

Author	Jessica Keyes
Publisher	Auerbach Publications
Published	2004
ISBN	0-201-74868-1
# of Pages	640
CQSE BOK	7 Configuration Management; 4 project management; standards, specifications, and models 5; software metrics

Perhaps the most striking relevance of this book is that while the process of SCM has not fundamentally changed very much in the past 30 years, the program elements being managed have changed immensely. The IT environment has moved from centralized mainframes with a scattering of programming languages to an environment replete with decentralized, networked, WEB-enabled systems employing thousands of clients using hundreds of software packages and dozens of programming languages. This is not your mother’s SCM.

To manage SCM in our present environment one needs to understand that having a finely tuned SCM process remains key to implementing and managing SCM in this diverse environment. To further complicate the mix, add in automated tools along with their library systems. A well-integrated CM process works to ensure that SCM will not exist in isolation. Thus a CM program will require solid project management inherent with a CM Plan describing the CM activities and tasks; the work breakdown structure; the CM schedule, and also the risks and metrics.

By adopting the processes and concepts outlined in this book, you will have everything needed to implement and execute a sound SCM organization. Whether you are a project manager with responsibility for Configuration Management (CM), a CM manager, or a CM Team Lead, this book has something for you. From documenting the planning results in the CM Plan, using consensus standards, to evaluating and selecting a CM tool, to identifying CM metrics, and to identifying and then managing a CM process, the reader may find value within its pages.

This extensive book has two sections. The first is composed of 14 chapters describing every facet of SCM as it relates to software engineering. Each chapter consists of text, a summary, and applicable references. The second section consists of more than 20 appendices containing a plethora of SCM templates that; if used, may lead to achieving an SEI-CMM Level 2 equivalent CM organization employing repeatable processes.

Perhaps, the very best feature of this book is that it is life cycle oriented. For each phase of the generalized life cycle it identifies the relevant SCM activities along with the SCM milestones. The template for the configuration management plan (Appendix T) requires not only the traditional activities of configuration identification, configuration control, configuration status accounting, and configuration audits; but also identifies the engineering objectives and describes the SCM responsibilities for each phase of the development life cycle.

If you need a checklist for executing a functional or a physical configuration audit you only need go to Appendix V or W. If you desire the definition of a term or the meaning of an acronym, go to Appendix U. If you are seeking information on Configuration Management phasing and milestones you only need go to Appendix T4.

Our experienced CM Manager found the extensive references to CM-related consensus standards and the appendices containing templates, checklists, and samples to be the most useful components of this book. She has just placed her order for it.

In summary, using this book as a guide will allow your CM organization to be:

• a support function in that it supports program engineers and developers, the program, the organization, and oftentimes, the customer,
• a control function in that it controls specifications, documents, drawings, requirements, tools, software, and other deliverables, and
• a service provider in that it supports people and controls data.

Download this review:

As one of the early takers of the CSQE examination I entered the examination room with a notebook stuffed with copies of articles and extracts from varied publications. These were supplemented with notes taken at a study session organized by a colleague. Looking around the room I saw that some individuals had wheeled carts filled with reference books. I wondered if they really expected to have the time to search for the correct answer to a difficult question.

The CSQE Primer contains everything I had and much more. The material is organized to follow the CSQE Body of Knowledge (BOK). The layout of each page makes the material easy to read and easy to follow. The ample margins provide space for personal notes. The authors make no assumptions about the reader’s prior knowledge and thus everything is defined, explained, and illustrated.

Each section of the primer addresses a different section of the CSQE BOK. Each chapter contains descriptive information on the identified models, techniques, and processes. This is followed by a list of recommended readings and multiple choice questions (answers are contained in the index). One reader may choose to answer the questions in order to determine which sections need further study. Another reader may review the material and then answer the questions to determine their own level of knowledge and comprehension. As stated in the Preface, “This text is designed to provide a review of fundamental knowledge and skills.” It is also a Primer for those interesting in taking the certification examination.

This Primer has several strengths. These include:

• Identification of the authors of the many definitions, checklists, and templates
• Prolific references to consensus standards
• Identifies the cognitive level at which the sample questions will be written

The primer would be stronger if the authors had used IEEE Standard 1012, Software Verification and Validation, rather than 1059 which was a Guide to Software Verification and Validation and was withdrawn several years ago. Had they used IEEE Std. 1012-1998, the Primer might have included the concept of integrity levels. This concept states that the breadth and depth of V&V tasks is determined by the level of impact of a software or system failure. The Primer also would have described the criteria for evaluating documentation such as requirements and design documents as well as for doing requirements traceability.

If I were taking the CSQE examination tomorrow, I would take this book and Software Engineering: A Practitioner’s Approach by Roger S. Pressman