Download this review

Author	Manish Gupta & Raj Sharman
Publisher	Information Science Reference
Published	2009
ISBN	978-1-60566-036-3
# of Pages	383
CQSE BOK	Systems architecture,requirements engineering, risk management

Imagine my surprise when I saw that this book is actually a collection of essays written by a wide variety of people.  This book brings together articles from researchers and practitioners in the financial, legal, technology, information security fields through original papers on all aspects of roles and effects of human and social dimensions of information security.  As stated in the Preface, the key objective of this book is to “fill the gap in existing literature on human and social dimensions of information security by providing the readers one comprehensive source of latest trends, issues and research in the field.”

Each paper constitutes a chapter within the section.  As a result the same or similar concepts are presented with different perspectives.  I find this very enlightening.   The papers are arranged in sections with each section containing 5 or more papers authored by persons representing academics and practitioners from the international community.  The common thread through each section concerns the human element.

Section I:  Human and Psychological Aspects – This section begins with I  the notion of humans and their frailties as related to the security practices for individuals, businesses, and organizations.  It continues with the impact of humans on information security and why humans make poor security decision.  The last paper in this section describes the incompatibility of software quality assurance procedures with the use of automatically generated code.

Section II:  Social and Cultural Aspects – Beginning with a description of the complex nature of information security culture in a networked environment and a paper that examines the knowledge that might be available if both technology and human activity were seen as being equally important, this section brings together diverse elements of discourse.  These papers are followed by a paper discussing social engineering as a technique for compromising information systems.  The final paper in this section describes a model of a social paradigm for security and software engineering.

Section III:  Usability Issues – The difficulty of using security configuration interfaces and how those interfaces can be improved is the subject of this first paper.  It is followed by a paper describing the need for and the challenges of security usability and a paper describing the need and techniques for distinguishing between humans and automated computer programs on the Internet.  One of these techniques is the use of CAPTCHAs.  The last paper argues that it is possible to find a good compromise between quality of predictions and protection of personal data.

Section IV:  Organizational Aspects – This final section begins with a paper describing the incorporation of human and social factors in the threat-vulnerability model of risks and the management of vulnerabilities.  Following this are papers addressing workplace monitoring and its implication for privacy concerns and aligning risk management with business requirements as a strategy for developing effective enterprise information security management.  The ending papers highlight the issues resulting from the coalescence of system requirements elicitation, information security, and human factors and then the management of information as a critical corporate asset.

In addition to a detailed Table of Contents providing   an overview of each chapter within each section of the book, the authors have included a Forward that lays the foundation for this book and its very contemporary and relevant papers.  The theme is that although many organizations are relying purely upon technical solutions to implement their security policies this is an inadequate solution.  The authors believe that it is a lack of understanding that prevents them from addressing security from the people, processes, and technology standpoints to implement a successful security strategy.  This book is an attempt to close the information gap between technology and human factors.

By providing high quality research papers and industrial and practice articles on social and human aspects of securing information systems and infrastructure from social engineering attacks and real-world implications and implementations (practice) of the research the authors achieve their objective.

Introduction:

I was attending a networking meeting not too long ago. As is the custom I circulated through the crowd, introduced myself, and attempted to hand my business card to everyone I met and to get their card in return. At some point, someone announced that we were all to get into a circle and introduce ourselves and our companies to everyone in the group. This was the time for the three-minute elevator speech with which I present my hopes and aspirations in a short period of time. I took a deep breath. When my time came I presented it just as I had rehearsed it. I was well spoken and succinct. However I was not brilliant because I had neglected to take the audience into account. I neglected to frame my information for a non technical audience. As a result no one understood what I did or what kind of contacts I needed.

Afterward, I had the opportunity to speak with a number of people and to explain exactly what kind of contacts I needed and what I did. Since that day I have been striving to put the words of that one-on-one into another elevator speech. The speech should summarize the nature of the services offered, the benefits to the user, and the type of organization that could benefit from these services.

Services:

We provide a variety of IV&V services. The important thing is that we are independent. That means that we have a budget that is independent from the development budget, a management that does not report to the manager of development, and that the staff performing IV&V tasks is not the same staff as those performing development tasks. As a result we are able to provide non-biased feedback to the client organization. We base our assessments on consensus standards and so the feedback is both substantive and informed. We provide feedback on both the quality of the product and the likelihood that the system being produced will meet the business needs of the organization.

Our IV&V services include: Management Support; Engineering Support; Test Support; Review and Audits Support; and Process/Procedures Support. Thus the client organization can acquire all or some of these services. The details of the specific tasks included in each of these services can be found in another “musings on the deck…” — IV&V Services, published on July 4, 2008.

This means that in each of these categories of IV&V services we evaluate documents and other products that provide a foundation for the system under development.

Management Support Services

For example, the management category generally involves program/project plans, schedules, milestones, and proposals. It also may include various management plans such as quality assurance and configuration management. By evaluating these, IV&V identifies inconsistencies, errors, or other issues that may present a risk to the program/project, thus providing management with an early identification of potential problems.

Engineering Support Services

In the engineering category, IV&V evaluates individual technical/development product to insure that the content, at a minimum, conforms to the applicable IEEE or other consensus standard. IV&V also evaluates the adequacy and completeness of the products based on good engineering practices. One of the earliest assessments performed by IV&V and the most important is the assessment of the system requirements because these are incorporated in the subsequent design documents and ultimately into the software code or the computer hardware. These evaluations identify the adequacy, completeness, consistency and correctness of the content so that technical risks are identified prior to the creation of software code or manufacture/assembly of computer hardware.

Test Support Services

In the test category, IV&V evaluates developer and/or integrator test documentation including plans, test cases and test procedures. These are assessed against the applicable IEEE or other consensus standards for consistency, completeness, and correctness. IV&V determines that the test documents correctly link to each other and correctly reflect the business needs, security, performance and other requirements based on the level of testing. Additionally, IV&V often witnesses the actual testing performed by the development and/or integration contractor. After testing is completed, IV&V may assess the test report prepared by the development and/or integration contractor. As a result of the test activity, IV&V identifies possible risks, if any, for taking the system under test, into production.

There may be occasions when IV&V is tasked to perform its own testing. Whenever this occurs, IV&V must generate its own test plans, test cases, and test procedures. Once test execution is completed IV&V will write its own test report.

Review and Audit Support Services

Although projects may have many reviews and audits these are typically either management reviews or technical reviews. Management reviews may be performed at different intervals by various level of management and are most often concerned with schedule and budget issues. IV&V often provides input concerning the adequacy of project artifacts or lifecycle processes, in the form of reports or presentations, in support of these reviews. IV&V may provide a special report for that specific review or an IV&V End of Activity Report may be provided in support of the management review. Management has the responsibility for deciding if the project should go forward and if so – what is the risk or the cost and if the project should not go forward – what is the risk or the cost.

Technical reviews are sometimes called milestone reviews. Technical reviews are held based on the project schedule and/or at the completion of one or more specific products. IV&V provides input, in the form of task reports, in support of technical reviews. Technical reviews are held to determine if a specific product meets its requirements and to determine if the project can move to the next activity based on completion of approved criteria. One technical review is often called the “critical design review.” This event determines if the requirements and the design documents are complete and that the project can move to the implementation (code generation) process. The technical review provides project manage to assess the technical risks involved with going forward.

One type of audit may examine a process and the results of that process – such as a Configuration Audit. Another type of audit may compare the system as-built to the system as-tested – such as a Functional Configuration Audit. IV&V may participate in these audits.

Benefits:

An immediate benefit of IV&V is that upper management is made aware, on an on-going basis, of potential risk to the budget or schedule while project management is appraised on a continuing basis on the quality of the deliverables. Some deliverables, if defective, will have substantial negative impact on the ability to build the right system and to do so in a correct manner. Other deliverables, if defective, will have a substantial negative impact on the ability to monitor and control the project. All defective deliverables should be corrected as early in the life cycle as possible in order to mitigate the risk to the project and/or system.

A more long-term benefit of IV&V is the avoidance of costs associated with defective systems or software. These costs might include excessive maintenance and support costs due to an increase in HELP requests and/or the unplanned software releases. Additional costs may include those associated with the creation of a negative corporate image among potential customers based on negative reviews by professional review organizations or bloggers. There may be future costs associated with current customers leaving and going to competitors. Finally, defective systems or software may result in lawsuits for breach of contract or even damage to organization assets.

The cost of IV&V over the development life cycle is said to average 15%; however, once the cost of IV&V is spread over the life of the system or software then the actual cost is significantly lower. A study at the NASA IV&V facility compared the Return on Investment (ROI) of IV&V on several different projects to be 1.9 on the low side and 4.9 on the high side.

Benefitting Organizations:

If you own or represent an organization that produces software for its own use or for others your organization will benefit from allocating funds for IV&V because testing the product to assure its functionality only assures the functionality of the system or software as-built. It does not assure the functionality that should-be. IV&V will assure the functionality that you need is present.

If you own or represent an organization that acquires systems and or software produced by others you will benefit from allocating funds for IV&V because you are not in the business of developing systems or software and likely have neither the skills nor the time to oversee the development. At the same time you probably have a low level of trust that the development contractor will deliver what you need and what you paying for without stringent oversight of the development products. IV&V will provide the oversight so that your get the system or software that you need and that you are paying for.

You can acquire as many or as few IV&V services as you need or your budget permits. No matter how much or how little IV&V you acquire you will enhance the quality of your product.

It has been more than 10 years since The Office of Management and Budget (OMB) issued Circular A-119 (Revised) directing federal agencies to use voluntary consensus standards, both domestic and international, in its regulatory and procurement activities. This circular defined voluntary consensus standards as having the following attributes: openness; balance of interest; due process; an appeals process; and consensus. Standards that meet these criteria include ISO, IEEE, and ANSI standards. It was expected that the agencies, including the Department of Defense, would cease using their own agency-specific standards which were often modeled after the proprietary standards of the contractors hired to develop those standards.

Although the circular is not specific as to its use in the external development of systems and/or software, it is specific in stating that the agencies must use voluntary consensus standards in its regulatory and procurement activities. For example, agencies could, if they desired, state via the RFP that their evaluation of a contract deliverable would be based on its conformance to the applicable voluntary consensus standard. Since the release of Circular A-119 (Revised) few if any of the federal agencies have attempted to bring their System Development Life Cycle standards into alignment with the circular. Recently, OMB issued Circular A-11 which references the use of voluntary consensus standards in Section 53. However, legacy systems continue to be upgraded and new systems continue to be built with little or no standardization across agencies or within individual agencies.

Introduction:

Assuming the federal agencies want to conform to OMB Circular A-119(Revised), the unanswered question is – what precludes conformance? Some possible answers include: it will be too expensive; our operations and maintenance contractors won’t cooperate; our CPIC group doesn’t care how IT is done; our project managers don’t know how to do it or think it will take too much effort to implement. These arguments are all realistic albeit not very logical.

Perhaps the unspoken reason is that the federal agencies do not know how to implement the use of voluntary consensus standards. This paper is specific to the use of such standards in the development of systems and software and will propose several implementation options ranging from directive to consensus.

Implementation Options:

The implementation that is correct for a particular federal agency is the one that is in-synch with the agency culture and its way of managing IT. One agency may feel comfortable with a top-down approach whereas another agency may be more comfortable with a bottom-up approach. Yet another agency may find that a combination or hybrid of these two diverse approaches is most effective. This paper describes four models – Directive, Project, Consensus, and Hybrid.

Directive Model

Description: The CIO determines that it is advantageous for the agency to adopt voluntary consensus standards and signs that directive. The CIO staff will then develop an implementation strategy and guidelines for implementation. The directive is binding for all internal system or software development. Although OMB Circular A-119 does not mandate the use of voluntary consensus standards by external contractors, it does mandate their use in regulatory and procurement activities, and the CIO could make a determination that the agency will evaluate contractor deliverables against these standards. Also, the CIO could make the determination that any externally produced system utilizing the agency backbone or connecting to an in-house system must be developed using voluntary consensus standards.

Advantages: The specifics of implementation are specified. Project managers know what they have to do. All members of projects immediately are held accountable. This model is best used by an organization with a well-defined chain-of-command.

Disadvantages: Some IT program managers may push back because they have not been consulted and/or because their programs are developed outside of the CIO’s arena. They may neglect to use the standards or they may not monitor the use of the standards. There is no opportunity to validate the guidelines before they are implemented.

Project Model

Description: Pilot the transition to voluntary consensus standards using a highly visible project to learn from and improve the implementation process. In this model, the program manager declares that voluntary consensus standards are to be used on the project, and the applicable voluntary standards are provided by the agency. As a result, all documentation generated by the agency is based on voluntary consensus standards and the RFP states that all contract deliverables will be assessed and accepted or rejected based on voluntary consensus standards.

Advantages: Using a single project reduces the potential impact of an ill-planned implementation. A single project allows for the validation of the concept and provides a feed-back mechanism that enhances the final implementation across all IT projects. This ultimately gains buy-in from the stakeholders.

Disadvantages: Other projects continue to have the option of not using voluntary consensus standards. A highly visible project that requires more than a year will delay implementation of the use of voluntary consensus standards throughout the agency’s IT projects.

Consensus Model

Description: A champion, whether at the level of the CIO or lower down, introduces the idea to relevant managers. In an agency with a high level CIO and division level CIOs the relevant managers would be the low-level CIOs. In another agency, the relevant managers may be the IT Program Officers. Once that group agrees that the use of voluntary consensus standards would be beneficial, they may request the CIO to issue the policy statement and to have staff develop implementing procedures and guidelines. In addition to achieving consensus on the need for the use of voluntary consensus standards, the consensus model may include the need to achieve consensus on the implementation procedures and guidelines.

Advantages: The consensus model ensures that all stakeholders have a sense of ownership in the process to be followed. This is especially true if the consensus model is used for the development of the implementing procedures and guidelines as well as in the selection of the model. The consensus model may reflect the requirements of small as well as large and complex projects.

Disadvantages: The process of achieving consensus may require the inclusion of a waiver process that dilutes the potential impact of using the voluntary consensus standards. The time required to affect this model may be the lengthiest of any of the options as stakeholders struggle to exempt their area or to determine criteria for exception.

Hybrid Model

Description: A hybrid model may involve the Directive Model and the Project Model, or it may involve the Consensus Model and the Project Model. In the first instance, a highly visible project may be identified to implement voluntary consensus standards while the CIO staff is developing the procedures and guidelines for the agency. In the second instance, a highly visible project may be identified to implement voluntary consensus standards while the management team is achieving the consensus required to bring the use of voluntary consensus standards to the agency as a whole. Alternatively, an agency may identify a highly visible project as a way of implementing procedures and guidelines that have been drawn up as a result of either the directive or the consensus model.

Advantages: The hybrid model provides the opportunity to implement the model in a limited fashion while mitigating potential risks associated with an all-agency implementation. The model affords the organization needed experience to modify the method of implementation, as well as the specific procedures and techniques based on the experience of the highly visible project.

Disadvantages: A highly visible project ,that requires more than a year to complete, may delay implementation of the use of voluntary consensus standards throughout the agency’s IT projects. The relative ease of implementing voluntary consensus standards in a single project compared with the difficulty of an all agency implementation may deter the all-agency implementation.

Summary:

OMB Circular A-119 directs the federal agencies to remove themselves from the business of generating and maintaining standards. The availability of voluntary consensus standards allows them to do just that. The use of voluntary consensus standards allows the federal agencies to manage their IT staff and contractors better by setting a consistent level of expectations for them and the agency program manager. The OMB Circular does not specify the method by which the agencies are to implement the directive to use voluntary consensus standards. Rather the OMB Circular leaves it up to each federal agency to determine the best way to implement the requirement.

This paper has defined four models (Directive, Project, Consensus, Hybrid) that can be implemented. In addition this paper has described some advantages and disadvantages of each model. It is up to the agency to identify the model that best suits its needs and its culture. Once the agency has selected this model, they may desire to use an outside contractor to develop the process and procedures required for implementation.

Both in the classroom and on the client site I am asked to describe the difference between test and verification and validation. It is always difficult to explain that test performs a number of test tasks and that sometimes verification and validation performs duplicative test tasks.

Test is responsible for a wide-range of test tasks that occur throughout the system development life cycle. Early in the life cycle test will evaluate the requirements (system, software, interface) to insure that they can be validated. Test will become familiar with the concept documentation and the business needs. Later in the life cycle test will, based on the integrity level of the system, determine the necessary breadth and depth of testing. After preparing the required test documentation test will commence testing. This may result in testing each component, the integration of the components, system testing against the system requirements, and acceptance testing to validate that the system meets user needs. Or a determination may be made that test will do only acceptance testing while development is responsible for the earlier testing. After testing is completed test may prepare interim test reports or merely a final test report. These and other test tasks are fully described in IEEE Std. 829-2008.

If the development effort has a verification and validation component then, based on the identified integrity level of the system, verification and validation will evaluate the adequacy of the test tasks and/or perform its own test tasks in addition to the ones performed by test. If the verification and validation evaluation tasks are performed by persons outside of the development structure, budget, and management then they are called “independent.”

The Verification and validation component will accomplish various tasks in addition to evaluating the adequacy of the test tasks. The verification and validation tasks entail assessing the adequacy of development tasks. This determination is based on the assessment of development artifacts against applicable voluntary consensus standards (IEEE, ISO, ANSI). An assessment of the system development process would use ISO 15288 while an assessment of a requirements document would use IEEE Std. 1233. In addition to assessing the artifact against the applicable standard Verification and Validation assesses the artifact in terms of good engineering practices. Thus an assessment of a design document would not only include the applicable standard but also would include the flow-down of the architecture document, the business needs of the organization, and the system or software requirements. This engineering perspective distinguishes Verification and Validation from Quality Assurance. Another musing will provide additional discussion on the intersect of Quality Assurance and (Independent)Verification and Validation.

Information on determining the integrity level is fully described in IEEE Std. 829-2008 and IEEE Std. 1012-2004. If merited by the identified integrity level, verification and validation will also engage in testing. Their testing may be in parallel with that being done by test or it may be after that done by test.

Just as test has tasks that are performed within the context of the development life cycle so does verification and validation. Only some of these tasks will intersect with the tasks being performed by test. And just as the test tasks will vary according to the identified integrity level of the system being developed so will the verification and validation tasks vary according to the identified integrity level. The table below shows the intersection of recommended tasks for a project with a level 3 criticality level. Projects with lower levels, indicating they are less critical, will have fewer tasks. The selected system development life cycle phases are only examples. Every project will have its own life cycle phases.

Phase: Management

Phase: Acquisition

Phase: Supply

Phase: Development/Concept

Phase: Development/Requirements

Phase: Development/Design

Phase: Development/Implementation

Phase: Development/Test

Phase: Development/Installation & Checkout

Phase: Operation

Phase: Maintenance

 As can be seen above verification and validation is responsible for many of the same tasks as is test albeit from a different perspective. Both efforts have to be planned, managed, and executed. In both instances there will be interfaces with other organizational and supporting entities.

For systems with a lower integrity level verification and validation only may review the documentation created by test and witness test execution. For systems with a higher integrity level, verification and validation may review the documentation created by test and generate its own test documentation. It may witness test execution and perform its own tests based on its own test documentation.

Conclusion:

Even though many organizations think of verification as being the same as test, I have shown that for high integrity systems both should be implemented. For low integrity systems that provide a low risk to the acquiring organization having only project level testing may be adequate.

For about twenty years, I have been involved in Information Technology (IT) in both the public and private sectors. I have been a business analyst, a software tester, a test manager, and an independent verification and validation (IV&V) manager. I have taught software quality assurance at a Washington DC University. I co-authored the IEEE Std 829-2008, System and Software Test Documentation and I was involved in the development of the IEEE Std 1012-2004, Standard for Verification and Validation and I am currently part of the working group that is developing IEEE Std 1012-x that will enhance best practices in software verification and validation.

Many federal government agencies are looking at independent verification and validation (IV&V) as the panacea to all their ills — or at least to some of their ills. Much of this interest in IV&V is driven by a new breed of CIO’s who understand the business value of having IV&V as well as by members of Congress who attach a requirement for IV&V to funding for large IT acquisition. As a result, requests for proposal (RFP) are being issued for IV&V services. Unfortunately for the acquirer and the potential supplier the requests and the resulting statement of work are either very general such as “provide IV&V services” or overly detailed such as “provide an approach for completing the IV&V strategy,” or “provide an overview of what will be contained in the IV&V Plan.” The later came from an organization that expected IV&V products to be consistent with IEEE Std. 1012-2004.

IV&V Taxonomy

After a number of years of these recurring experiences, I have come to the conclusion that the agencies know what they want but they don’t know how to ask for it. To resolve this problem I have developed a taxonomy of IV&V services. This taxonomy is not complicated and involves only five categories of IV&V services. These include: management and support; engineering documentation; test documentation and test witnessing; reviews and audits; and process and procedures. Within each category are a number of specific services. Presenting these categories with specific services to the IV&V sponsor and/or to the contracting office ensures a “win-win” situation for both the acquiring agency and the supplying contractor. The taxonomy provides a means for the acquiring agency to articulate its needs and it provides specifics that allow the supplier to meet those needs. These categories and their contents provide an orderly core set of IV&V services for selection and allow the acquiring agency to identify services they may have overlooked.

In the management and support category the acquirer can choose from:

· Assessment of the Statement of Need

· Assessment of the Program/Project Plan

· Assessment of the Configuration Management Plan

· Assessment of the Quality Assurance Plan

· Assessment of the Security Plan

· Assessment of the Master Test Plan

Just as the Program/Project Plan describes how the program/project will be managed including the organization and identification of the high-level activities so does the Master Test Plan describe the organization of the test effort and the identification of the high-level activities. Additional services in this category may include an assessment of the certification and accreditation package prior to submission for approval or an assessment of the transition plan. It may include an assessment of the project schedule or even the EVMS process. IV&V may be requested to evaluate the Program Management Office (PMO) and its processes or even the IT acquisition process.

Within the engineering documentation category the acquirer can choose from:

· Assessment of System Requirements

· Assessment of Architectural Design

· Assessment of Interface Requirements

· Assessment of Software Design

· Assessment of Database Design

Additional services in this category may include an assessment of the conversion plan, the migration plan, the correctness of data conversion, or the integration plan.

In the test documentation and test witnessing category the acquirer can choose from:

· Assessment of Test Plans (integration, system, acceptance)

· Assessment of Test Designs ( integration, system, acceptance)

· Assessment of Test Cases (integration, system, acceptance)

· Assessment of Test Procedures (integration, system, acceptance)

· Assessment of Test Execution (integration, system, acceptance)

· Assessment of Test Reports (integration, system, acceptance)

Within this category the acquirer may be interested in an assessment of the test logs or even the anomalies that have been generated throughout the development process. The assessment of test execution will include whether or not each test run provided the expected results and whether an anomaly report was written if it the test did not provide the expected results. IV&V may also report on how well the test process as described in the documentation was followed especially regarding test suspension and test resumption.

In the review and audits category the acquirer can choose from:

· Support of Requirements Review

· Support of Preliminary and Critical Design Review

· Support of Test Readiness Review (integration, system, acceptance)

· Support of Function and Physical Configuration Audit

· Support Management Reviews

Each of these reviews and audits may be supported by IV&V. When supporting a review or audit, IV&V will determine if the required inputs have been base-lined prior to the review or audit. For example, IV&V will ascertain whether the requirements documents and the applicable design documents have been base-lined prior to the Critical Design Review which assesses the detailed design documents. Applicable IV&V task reports for each the documents provided for each review will be included. IV&V also will participate in these technical reviews.

The acquiring organization may request that IV&V participate in management reviews. If so, the acquiring organization will identify whether it wants participation at all project level management meetings, selected meetings, invited participation at senior management reviews or some other combination of regular and ad hoc management meetings.

In the process and procedures category the acquirer can choose from:

· Assessment of the Configuration Management Process

· Assessment of the Quality Assurance Process

· Assessment of Test Process

· Assessment of the Development Process

Conclusion

Not every organization or every project will require the full range of IV&V services. For example, some projects will be development projects while others will be up-grade projects. However, by categorizing the possible services the acquiring organization will be in a better position to identify if it wants IV&V for its engineering effort or for its process evaluation. Once the needed category has been identified the acquiring organization will then understand what specific core services are available within each category and will be able to select those that are best suited for the project or program at hand.

Download this review

Author	Mary Beth Chrissis, Mike Konrad, and Sandy Shrum
Publisher	Addison-Wesley
Published	2007
ISBN	0-321-27967-0
# of Pages	595 plus appendices
CQSE BOK	Software Engineering Processes; Program and Project Management

There is a plethora of maturity models, standards, methodologies, and guidelines available to help organizations improve the way they do business. Most of them focus on a specific part of the business and do not take a systematic approach to the problems faced by the total organization. The CMMI provides an integrated model that transcends disciplines. It addresses practices that cover the product (or service) life cycle from conception through delivery and maintenance. CMMI emphasizes the work needed to build and maintain the total product (or service).

If you are new to process improvement or perhaps new to the concepts governing CMMI then you should read Chapter 1 because it provides an overview of process improvement and describes CMMI. Then you should skim Part Two, describing both generic and specific goals and practices. After that you may want to review the references in Appendix A to identify additional sources of beneficial information. Follow this with a reading of the acronyms and glossary to become familiar with the language of CMMI and then return to Part Two.

If you are new to CMMI but you do have experience with other process improvement models, you will notice many similarities. Read Part One to learn how CMMI is different from other process
improvement models. Then as you read Part Two notice the best practices from other models with which you are familiar. Study the tips, hints, and cross references to see details and relationships that will assist in understanding CMMI.

If you are not new to CMMI you will quickly recognize the concepts and best practices. Focus on the tips, hints, and cross references in the process areas (Part Two) to discover ideas, relationships or details you may have overlooked previously.

Part One of this book describes the various approaches and models that govern the capability maturity model. It also describes the process area components and the multiple maturity levels. These are then followed by a description of the relationships among the process areas and information on using the CMMI models. The final section of Part One is a case study showing how Raytheon applied CMMI to services.

The authors have included practices that cover project management, process management, systems engineering, hardware engineering, software engineering, and other supporting processes used in development and maintenance. In addition, the construct may cover the use of integrated product teams for maintenance and development activities. The practices described are used by organizations in many diverse industries, including aerospace, banking, computer hardware, software, defense, automotive, and telecommunications.

There are 22 CMMI process areas and each one consists of required components, expected components, and informative components. Required components describe what the organization must do in order to satisfy the process area. Expected components guide those who implement improvements or perform appraisals. Expected components include the specific and generic practices.

The authors treat each process area as a component; and each one, in alphabetic order, consists of:
* Introductory notes that describe the major concepts covered in the process area.
* Related process areas lists references to related process areas and reflects the high level relationships among the process areas.
* Specific goals describe the unique characteristics that must be present to satisfy the process area.
* Generic goals describe the characteristics that must be present to institutionalize the processes that implement the process area.
* Specific goal and practices summary provides a high-level summary of goals and practices that must be accomplished. This is informative.
* Specific practices describe the activities that are considered important in achieving the specific goal.
* Typical work products lists sample output from a specific practice. This is informative.
* Sub-practices provide a detailed description that provides guidance for interpreting and implementing a specific or generic practice. This is informative.
* Generic practices contain a description of an activity that is important in achieving the generic goal.
* Generic practice elaboration provides guidance on how the generic practice should be applied to the process area.
* Supporting informative components provide further information to describe the concept. This may be in the form of notes, examples, amplifications, or references.
o Notes are text that provides detail, background, or rationale.
o Examples consist of text that clarify a concept or described activity.
o Amplifications are notes or examples that are relevant to a particular discipline.
o References are a pointer to additional or more detailed information in related process areas.
* Additions can be informative material, a specific practice, a specific goal, or a process area that extends the scope of the model or emphasizes a particular aspect of its use.
* Specific work products describe artifacts that are developed while implementing the specific practices.

While Part One provides the models for implementing CMMI, Part Two describes in detail all of the generic goals and practices of CMMI. It focuses on those model components that address process institutionalization. Institutionalization implies that the process is ingrained in the way the work is performed and there is commitment and consistency in the way the process is performed. The progression of processes is as follows:

* A performed process accomplishes the work necessary to produce work products. This is equivalent to a Level 1 process.
* A managed process is a performed process that is planned and the performance of the process is managed against the plan. This is equivalent to a Level 2 process.
* A defined process is a managed process that is tailored from the organization’s set of standard processes and provides a basis for planning, performing, and improving the project’s tasks and activities. This is equivalent to a Level 3 process.
* A quantitatively managed process is a defined process that is controlled using statistical and other quantitative techniques. This is equivalent to a Level 4 process.
* An optimizing process is a quantitatively managed process that is changed and adapted to meet relevant current and project business objectives and is continuously improved by addressing common causes of process variation. This is equivalent to a Level 5 process.

Because of my own interest in verification and validation, these were the first process areas I turned to. Verification is the process of determining that something meets its requirements and validation is the process of determining that something meets the user needs and is usable by the user. At first look, they appeared to be correct and with an appropriate level of detail. However, further examination led to dismay when I noted that under SP 1.1 Select Work Products for Verification it identified various types of testing as work products. The Verification process should have identified the test artifacts (e.g., Integration Test Plan) for each type of testing as work products and should have stated that the various types of testing actually are encompassed in the Validation process. One of the “tips” in this section states that verification often means testing. Testing is a technique, not for verification, but rather for validation as are inspection, demonstration, and analysis.

Artifacts are typically verified against applicable standards. Artifacts such as a training guide or an operations manual may first be verified to determine that the contained information is adequate and complete and then be validated to determine that the information is useable. This reviewer considers it problematic that this book is at odds with verification and validation as described in IEEE Std 1012-2004, IEEE Standard for Software Verification and Validation. Systems and software are validated. In addition, the book, in the Part Two section describing the Validation process states that the end users are typically involved in validation. This is true when validation is for User Acceptance Test. However, it is not true when validation involves integration, system, performance, or stress testing.

While other process areas appear to conform to “best practices” I was disappointed that the authors frequently recommend that the process information be documented in the Project Plan. A process such as Configuration Management or Process and Product Quality Assurance might be better served by having its own plan. Perhaps the need for separate plans, as appropriate, and the identification of consensus standards to assist in the preparation of the plans will be addressed in the next version of this book.

Yet even with these points of disagreement, I still recommend this book because of its depth and detail of coverage.

For about twenty years I have been involved in Information Technology (IT) in both the public and private sectors. I have been a business analyst, a software tester, a test manager, and an independent verification and validation (IV&V) manager. I have taught software quality assurance at a Washington DC University. And, I have been involved in the development of the two IEEE Standards that foster best practices in software verification and validation. I have been an IV&V subject matter expert on more projects than I can remember. Yet I kept thinking that there must be a better way then to re-invent IV&V for each project when each is within the same organization. I believe that there are many advantages to an enterprise-level IV&V.

Verification and Validation Models

Verification is the assessment of an artifact or product against its requirements. These requirements are most often found in the voluntary consensus standard applicable to that artifact or product. Validation is the determination that the artifact or product can be used by the user to accomplish user needs. Sometimes verification and validation are treated as separate processes as is done in the CMMI. Other times the two are joined together as a single process as occurs in the IEEE Standard for Software Verification and Validation. When joined together and executed by members of the engineering or development organization the process is known as Verification and Validation. When joined together and executed by persons who are not members of the engineering or development organization the process is known as Independent Verification and Validation (IV&V).

Just as there are multiple models for verification and validation there are multiple models for implementing IV&V. The project-model of IV&V is well known and the most often implemented. In this model, each IT project uses its own IV&V consultant who in turn brings to bear its processes, procedures and methodologies. As a result, multiple projects within the same organization may not receive IV&V services in a consistent manner. This occurs even if the same contractor is performing IV&V on several projects. A variation of the project model occurs whenever each project is permitted to modify or “tailor” a process, procedure or methodology including templates for plans, reports or assessments. This occurs when either the client organization or the development organization creates high-level processes, procedures or methodologies and then encourages the projects to “tailor” these, as needed.

Years spent undertaking Independent Verification and Validation (IV&V) in the project-model including the variation described above, have convinced me that this model neither lends itself toward achieving good engineering practices nor achieving CMMI level 5 in either the Verification or the Validation processes. The model I am proposing is the enterprise-model of IV&V. Using this model IV&V provides a supporting function to all IT projects within the organization in much the same way that the finance department, the legal department or the print shop support all entities in the organization. And just as these departments apply the same process , procedures, and methodologies to each entity they service so will the enterprise-model apply the same IV&V process, procedures, and methodologies to each IT project it services.

We believe this model is equally applicable to any organization whether it is in the public or private sector and whether it is small, medium or large.

Enterprise Model

There are a number of advantages to instituting an enterprise-model IV&V program. These include a consistent use of standards, templates and methodologies. Consistent means the same. Consistent does not mean tailored. Consistent means that a unique IV&V Plan will not be created for each IT project but rather an enterprise level IV&V Plan will be executed for each IT project. Consistent means that each IT project will use the same, not tailored, templates and methodologies, procedures, and processes. This consistency translates into reduced costs for the enterprise.

In addition to consistency the organization using an enterprise model can more quickly achieve an institutionalization of the Verification and Validation processes.

Figure 1

Note that the IV&V manager, or their designee, serves as a gateway between the IT projects and the IV&V staff. The IT project manager comes to IV&V and completes a request for IV&V services. These services range from verification of pre-SDLC artifacts to verification of SDLC artifacts, to monitoring system and acceptance testing, to performing IV&V testing. In addition to IT project artifacts and activities, IV&V may provide services such as a verification of the configuration management, risk management or the product and process quality assurance processes. These may be at the project level or at the enterprise level depending on the needs and structure of the organization. Additional IV&V services may include monitoring the security assessment process. The service request will be scheduled and staff resources allocated. As each artifact is reviewed and as each test is monitored a task report is generated and provided to the IT project manager through the IV&V manager.

The service request may include IV&V participation in baseline reviews such as the preliminary design review or the test readiness reviews. The service request also may include IV&V participation in configuration audits such as the functional configuration audit and the physical configuration audit.

In order to implement an enterprise model an infrastructure has to be created. The initial infrastructure for a managed IV&V x(since you use the article ‘a’ isn’t there a noun missing after IV&V?) includes standards, templates, and methodologies that are executed in accordance with an IV&V policy. It also requires highly skilled people who have the resources to produce their controlled outputs. Resources include the standards against which project deliverables will be verified. The author recommends IEEE Std 1012-2004 for an in-depth understanding of IV&V along with the applicable IEEE standards for specific IT project deliverables. For example, IEEE Std 1233 identifies and describes the criteria by which IV&V will verify the system requirements while IEEE Std 829-200x describes the format and contents of the various test documents, including a master test plan. There are applicable IEEE standards for virtually all IT project deliverables.

IV&V outputs will be produced, reviewed and managed according to an approved process. IV&V work products will be available to management at defined points such as at major milestones (e.g. preliminary design review) and at the completion of major tasks (e.g. integration test).

Conclusion

With the basic infrastructure in place the IV&V is set to operate as a managed process (Level 2). A road map for moving to higher levels will be the subject of future musings.

Download this review:

Author	Jessica Keyes
Publisher	Auerbach Publications
Published	2004
ISBN	0-201-74868-1
# of Pages	640
CQSE BOK	7 Configuration Management; 4 project management; standards, specifications, and models 5; software metrics

Perhaps the most striking relevance of this book is that while the process of SCM has not fundamentally changed very much in the past 30 years, the program elements being managed have changed immensely. The IT environment has moved from centralized mainframes with a scattering of programming languages to an environment replete with decentralized, networked, WEB-enabled systems employing thousands of clients using hundreds of software packages and dozens of programming languages. This is not your mother’s SCM.

To manage SCM in our present environment one needs to understand that having a finely tuned SCM process remains key to implementing and managing SCM in this diverse environment. To further complicate the mix, add in automated tools along with their library systems. A well-integrated CM process works to ensure that SCM will not exist in isolation. Thus a CM program will require solid project management inherent with a CM Plan describing the CM activities and tasks; the work breakdown structure; the CM schedule, and also the risks and metrics.

By adopting the processes and concepts outlined in this book, you will have everything needed to implement and execute a sound SCM organization. Whether you are a project manager with responsibility for Configuration Management (CM), a CM manager, or a CM Team Lead, this book has something for you. From documenting the planning results in the CM Plan, using consensus standards, to evaluating and selecting a CM tool, to identifying CM metrics, and to identifying and then managing a CM process, the reader may find value within its pages.

This extensive book has two sections. The first is composed of 14 chapters describing every facet of SCM as it relates to software engineering. Each chapter consists of text, a summary, and applicable references. The second section consists of more than 20 appendices containing a plethora of SCM templates that; if used, may lead to achieving an SEI-CMM Level 2 equivalent CM organization employing repeatable processes.

Perhaps, the very best feature of this book is that it is life cycle oriented. For each phase of the generalized life cycle it identifies the relevant SCM activities along with the SCM milestones. The template for the configuration management plan (Appendix T) requires not only the traditional activities of configuration identification, configuration control, configuration status accounting, and configuration audits; but also identifies the engineering objectives and describes the SCM responsibilities for each phase of the development life cycle.

If you need a checklist for executing a functional or a physical configuration audit you only need go to Appendix V or W. If you desire the definition of a term or the meaning of an acronym, go to Appendix U. If you are seeking information on Configuration Management phasing and milestones you only need go to Appendix T4.

Our experienced CM Manager found the extensive references to CM-related consensus standards and the appendices containing templates, checklists, and samples to be the most useful components of this book. She has just placed her order for it.

In summary, using this book as a guide will allow your CM organization to be:

• a support function in that it supports program engineers and developers, the program, the organization, and oftentimes, the customer,
• a control function in that it controls specifications, documents, drawings, requirements, tools, software, and other deliverables, and
• a service provider in that it supports people and controls data.

Download this review:

As one of the early takers of the CSQE examination I entered the examination room with a notebook stuffed with copies of articles and extracts from varied publications. These were supplemented with notes taken at a study session organized by a colleague. Looking around the room I saw that some individuals had wheeled carts filled with reference books. I wondered if they really expected to have the time to search for the correct answer to a difficult question.

The CSQE Primer contains everything I had and much more. The material is organized to follow the CSQE Body of Knowledge (BOK). The layout of each page makes the material easy to read and easy to follow. The ample margins provide space for personal notes. The authors make no assumptions about the reader’s prior knowledge and thus everything is defined, explained, and illustrated.

Each section of the primer addresses a different section of the CSQE BOK. Each chapter contains descriptive information on the identified models, techniques, and processes. This is followed by a list of recommended readings and multiple choice questions (answers are contained in the index). One reader may choose to answer the questions in order to determine which sections need further study. Another reader may review the material and then answer the questions to determine their own level of knowledge and comprehension. As stated in the Preface, “This text is designed to provide…a review of fundamental knowledge and skills. It is also…a Primer for those interesting in taking the certification examination.”

This Primer has several strengths. These include:

• Identification of the authors of the many definitions, checklists, and templates
• Prolific references to consensus standards
• Identifies the cognitive level at which the sample questions will be written

The primer would be stronger if the authors had used IEEE Standard 1012, Software Verification and Validation, rather than 1059 which was a Guide to Software Verification and Validation and was withdrawn several years ago. Had they used IEEE Std. 1012-1998, the Primer might have included the concept of integrity levels. This concept states that the breadth and depth of V&V tasks is determined by the level of impact of a software or system failure. The Primer also would have described the criteria for evaluating documentation such as requirements and design documents as well as for doing requirements traceability.

If I were taking the CSQE examination tomorrow, I would take this book and Software Engineering: A Practitioner’s Approach by Roger S. Pressman

Download this review:

For several years now the IEEE has been viewed as the guiding light for software engineering standards. And for the same period of time, those involved with system development have been clamoring for system- oriented standards – maybe even a set of system engineering standards comparable to the set of IEEE software engineering standards.

This year the IEEE Standards Association voted to issue this ISO/IEC 15288 standard as part of the IEEE system engineering series of standards. It will complement but not replace existing IEEE standards such as 12207-1995 which provides an overview of software engineering processes. IEEE agrees with the ISO/IEC joint task force that there exists a need for a common framework to improve communications and co-operation within the organization among those who create, utilize, and manage modern systems. The ISO/IEC 15288 provides this common framework.

This framework provides a common process covering the development lifecycle. From the initial conception of the idea, to acquiring and supplying the system, through its ultimate retirement, this standard provides a comprehensive set of processes from which any organization can apply to a system lifecycle model appropriate to its products and services. This standard describes processes that support the definition, control and improvement of new or existing processes of an organization or project. Any organization, depending on its purpose, can select and apply an appropriate subset of these processes to fulfill its purpose. For example:

• An organization can use this standard to help establish an environment of desired processes in order to manage its projects and systems through their lifecycle processes.
• A project can use this standard to help select, structure and employ the elements of an established environment to provide products and services.
• An acquirer and a supplier both can use this standard to help develop an agreement concerning processes and activities, and maintain congruity between both.

This standard identifies four essential lifecycle processes – agreement, enterprise, project, and technical. For each of these, the standard identifies sub-processes. The agreement process consists of the acquisition and supply processes.

The enterprise process consists of the environment management sub-process, the investment management sub-process, the system lifecyle management sub-process, the resource management sub-process and the quality management sub-process.

The project process consists of the sub-processes for project planning, project assessment, project control, decision-making, risk management, configuration management, and information management.

The technical process consists of the following sub-processes: stakeholder requirements, requirements analysis, architectural design, implementation, integration, verification, transition, validation, operation, maintenance, and disposal.

To assist you in using the standard each sub-process identifies the purpose of that sub-process, the activities to be implemented to achieve the purpose of the sub-process, and the outcomes that demonstrate the purpose of the sub-process has been met. For example, an organization looking to enhance its quality management sub-process will implement those activities that seek to achieve the outcomes listed below:

• Quality management policies and procedures are defined.
• Quality goals and objectives are defined.
• Accountability and authority for quality management are well defined.
• Status of customer satisfaction is monitored.
• Appropriate action is taken when quality goals are not achieved.

These outcomes will assure that the products, services and implementations of lifecycle processes meet enterprise quality goals and achieve customer satisfaction. This assurance is the purpose of the Quality Management sub-process.

This reviewer has found ISO/IEC 15288:2002 relatively user-friendly and has been using it to guide a large and complex program that uses an iterative development methodology. Thus the system as a whole will undergo a stakeholder requirements definition process, a requirements analysis process, and an architectural design process, and so forth through the operation process and finally the maintenance process. Each of the multiple iterations will undergo the same processes as appropriate and each release within the iteration will undergo the same processes, as appropriate. For example, the processes for iteration one/release one begins with a preliminary design, followed by a detailed design, and ends with the validation process because release one is not considered to be an operational release.

Looking at the process from another direction – the development of the system as a whole is actually the culmination of the development of multiple smaller systems. Each sub-systems is developed according to the processes and sub-processes. The sub-systems are integrated, verified, and transitioned according to the applicable processes and sub-processes. The final integration will be that of the system as a whole which will be verified, transitioned, placed into operation and maintained.

This recursive use of sub-processes is a key aspect of the application of ISO/IEC 15288. The outputs (information, artifacts, services) of sub-processes at any level become the inputs to the same sub-process used at a level above or a level below. Outputs across all levels of the system architecture can be resolved and consistency achieved.

ISO/IEC 15288 describes lifecycle processes that can be invoked, as needed, any time throughout the lifecycle without a prescribed ordering to their use, thus recognizing that an individual lifecyle is a complex system of processes that possess concurrent, iterative, recursive and time dependent characteristics.

ISO/IEC 15288 provides a level of detail not now provided by IEEE 12207-1995. In addition, it is supportive of SEI-CMM and SEI-CMMI. Thus it is applicable for program and project managers, quality managers, and verification and validation managers at all levels of the organization.