Study on Project Management in NASA

Description
The National Aeronautics and Space Administration (NASA) is the agency of the United States government that is responsible for the nation's civilian space program and for aeronautics and aerospace research.

Report on Project Management in NASA
Page 2
-2-

Table of Contents
Page Signature Page (Board Members) 3 Consultants 4 Acknowledgements 5 Executive Summary 6 1. Introduction 10 2. The Mars Climate Orbiter Mission: Observations and Lessons Learned 15 3. A New Vision for NASA Programs and Projects 24 4. NASA’s Current Program/Project Management Environment 33 5. Recommendations and Metrics 36 6. Checklist for Project Management and Review Boards 44 7. Concluding Remarks 47 Appendixes A. Letter Establishing the Mars Climate Orbiter Mishap Investigation Board B. Mars Climate Orbiter Mishap Investigation Board Phase I Report (dated Nov. 10, 1999) C. Letter Providing Revised Charter for the Mars Climate Orbiter Mishap Investigation Board D. List of Existing Processes and Requirements

Applicable to Programs/Projects E. List of Additional Projects Reviewed by the Mars Climate Orbiter Mishap Investigation Board F. Recurring Themes From Failure Investigations and Studies
Page 3
-3-

Signature Page
__________/s/________________ ____________/s/_____________

Arthur G. Stephenson, Chairman Lia S. LaPiana, Executive Secretary Director, George C. Marshall Program Executive Space Flight Center Office of Space Science NASA Headquarters
__________/s/_______________ ____________/s/_____________

Dr. Daniel R. Mulville Dr. Peter J. Rutledge (ex-officio) Associate Deputy Administrator Director, Enterprise Safety and NASA Headquarters Mission Assurance Division NASA Headquarters
__________/s/_______________ ____________/s/_____________

Frank H. Bauer David Folta Chief, Guidance, System Engineer, Guidance, Navigation and Control Center Navigation and Control Center Goddard Space Flight Center Goddard Space Flight Center
__________/s/_______________ ____________/s/_____________

Greg A. Dukeman Robert Sackheim Guidance and Navigation Specialist Assistant Director for Space Vehicle Flight Mechanics Group Propulsion Systems George C. Marshall Space Flight Center George C. Marshall Space Flight Center

__________/s/_______________

Dr. Peter Norvig Chief, Computational Sciences Division Ames Research Center
__________/s/_______________ ____________/s/_____________

Approved Approved Dr. Edward J. Weiler Frederick D. Gregory Associate Administrator Associate Administrator Office of Space Science Office of Safety & Mission Assurance Advisors: Office of Chief Counsel: MSFC/Louis Durnya Office of Public Affairs: HQ/Donald Savage Page 4
-4-

Consultants
Ann Merwarth NASA/GSFC-retired Expert in ground operations & flight software development Moshe F. Rubinstein Prof. Emeritus, University of California, Los Angeles Civil and Environmental Engineering John Mari Vice-President of Product Assurance Lockheed Martin Astronautics Peter Sharer Senior Professional Staff Mission Concepts and Analysis Group The Johns Hopkins University Applied Physics Laboratory Craig Staresinich Chandra X-ray Observatory Program Manager, TRW Dr. Michael G. Hauser Deputy Director Space Telescope Science Institute

Tim Crumbley Deputy Group Lead Flight Software Group Avionics Department George C. Marshall Space Flight Center Don Pearson Assistant for Advanced Mission Design Flight Design and Dynamics Division Mission Operations Directorate Johnson Space Center
Page 5

Acknowledgements
The Mars Climate Orbiter Mishap Investigation Board wishes to thank the technical teams from Jet Propulsion Laboratory and Lockheed Martin Astronautics for their cooperation, which was essential in our review of the Mars Climate Orbiter project. In addition, the Board wishes to thank the presenters and members of other review boards and projects listed in Appendix E, who shared their thoughts on project management. Finally, the Board wishes to thank Jerry Berg and Rick Smith, of the Marshall Space Flight Center’s Media Relations Department, for their editorial assistance on this report; and Drew Smith, of the Marshall Center, for his invaluable support to the Board.
Page 6

Executive Summary
This second report, prepared by the Mars Climate Orbiter Mishap Investigation Board, presents a vision and recommendations to maximize the probability of success for future space missions. The Mars Climate Orbiter Phase I Report, released Nov. 10, 1999, identified the root cause and factors contributing to the Mars Climate Orbiter failure. The charter for this second report is to derive lessons learned from that failure and from other failed missions — as well as some successful ones — and from them create a formula for future mission success. The Mars Climate Orbiter mission was conducted under NASA’s ?Faster, Better, Cheaper? philosophy, developed in recent years to enhance innovation, productivity and cost-effectiveness of America’s space program. The ?Faster, Better, Cheaper? paradigm has successfully challenged project teams to infuse new technologies and processes that allow NASA to do more with less. The success of ?Faster, Better, Cheaper? is tempered by the fact that some projects and programs have put too much emphasis on cost and schedule reduction (the ?Faster? and ?Cheaper? elements of the paradigm). At the same time, they have failed to instill sufficient rigor in risk management throughout the

mission lifecycle. These actions have increased risk to an unacceptable level on these projects. The Mishap Investigation Board conducted a series of meetings over several months with the Jet Propulsion Laboratory and Lockheed Martin Astronautics to better understand the issues that led to the failure of the Mars Climate Orbiter. The Board found that the Mars Surveyor Program, agreed to significant cuts in monetary and personnel resources available to support the Mars Climate Orbiter mission, as compared to previous projects. More importantly, the project failed to introduce sufficient discipline in the processes used to develop, validate and operate the spacecraft; nor did it adequately instill a mission success culture that would shore up the risk introduced by these cuts. These process and project leadership deficiencies introduced sufficient risk to compromise mission success to the point of mission failure. It should be noted that despite these deficiencies, the spacecraft operated as commanded and the mission was categorized as extremely successful until right before Mars orbit insertion. This is a testament to the hard work and dedication of the entire Mars Climate Orbiter team. The Board recognizes that mistakes and deficiencies occur on all spacecraft projects. It is imperative that all spacecraft projects have sufficient processes in place to catch mistakes before they become detrimental to mission success. Unfortunately for the Mars Climate Orbiter, the processes in place did not catch the root cause and contributing navigational factors that ultimately led to mission failure. Building upon the lessons learned from the Mars Climate Orbiter and a review of seven other failure investigation board results, this second report puts forth a new vision for NASA programs and projects — one that will improve mission success within the Page 7 context of the ?Faster, Better, Cheaper? paradigm. This vision, Mission Success First, entails a new NASA culture and new methods of managing projects. To proceed with this culture shift, mission success must become the highest priority at all levels of the program/project and the institutional organization. All individuals should feel ownership and accountability, not only for their own work, but for the success of the entire mission. Examining the current state of NASA’s program and project management environment, the Board found that a significant infrastructure of processes and requirements already is in place to enable robust program and project management. However, these processes are not being adequately implemented within the context of ?Faster, Better, Cheaper.? To move toward the ideal vision of Mission Success First, the Board makes a series of observations and recommendations that are grouped into four categories, providing a guide by which to measure progress.

1) People
The Board recognizes that one of the most important assets to a program and project is its people. Success means starting with top-notch people and creating the right cultural environment in which they can excel. Thus, Mission Success First demands that every individual on the program/project team continuously employ solid engineering and scientific discipline, take personal ownership for their product development efforts and continuously manage risk in order to design, develop and deliver robust systems capable of supporting all mission scenarios. Teamwork is critical for mission success. Good communication between all project

elements — government and contractor, engineer and scientist — is essential to maintaining an effective team. To ensure good teamwork, the project manager must guarantee an appropriate level of staffing, and all roles and responsibilities must be clearly defined.

2) Process
Even the best people with the best motivation and teamwork need a set of guidelines to ensure mission success. In most cases NASA has very good processes in place, but there are a few areas for improvement. A concise set of mission success criteria should be developed and frozen early in the project life cycle. During the mission formulation process, the program office and the project should perform the system trades necessary to scope out the expected costs for mission success. This should be accomplished independently of any predefined dollar cap. If necessary, consider mission scope changes to drive the costs to a level that the program can afford. Scope should never be decreased below a minimum threshold for science and for technical achievement as defined by the mission success criteria. Page 8
-8-

Both the project and the program should hold adequate contingency reserves, to ensure that mission success is achievable. Projects and programs that wind up with inadequate funding should obtain more funds or consider cancellation before proceeding with inadequate funds. Close attention should be paid from project outset to the plan for transition between development and operations. Adequate systems engineering staffing, particularly a mission systems engineer, should be in place to provide a bridge during the transition between development and operations, and also to support risk management trade studies. Greater attention needs to be paid to risk identification and management. Risk management should be employed throughout the life cycle of the project, much the way cost, schedule and content are managed. Risk, therefore, becomes the ?fourth dimension? of project management — treated equally as important as cost and schedule. Project managers should copy the checklist located in the back of this report, putting it to constant use and adding to it in order to benchmark the performance of their project team. Moreover, this checklist should be distributed to all members of the project team as a 360-degree benchmark tool, to identify and reduce potential risk areas.

3) Execution
Most mission failures and serious errors can be traced to a breakdown in existing communication channels, or failure to follow existing processes — in other words, a failure in execution. To successfully shift to the Mission Success First culture, it is necessary for the institutional line management to become more engaged in the execution of a project. As such, line managers at the field centers need to be held accountable for the success of all missions at their centers. Let us be clear that this role of institutional line management accountability should not be construed as a return to the old management formula, wherein NASA civil servants provided oversight for every task performed by the contractor or team. Instead, we recommend that NASA conduct more rigorous, in-depth reviews of the contractor’s and

the team’s work — something that was lacking on the Mars Climate Orbiter. To accomplish this, line management should be held accountable for asking the right questions at meetings and reviews, and getting the right people to those reviews to uncover mission-critical issues and concerns early in the program. Institutional management also must be accountable for ensuring that concerns raised in their area of responsibility are pursued, adequately addressed and closed out. Line organizations at the field centers also must be responsible for providing robust mechanisms for training, mentoring, coaching and overseeing their employees, project managers and other project team leaders. An aggressive mentoring and certification Page 9
-9-

program should be employed as the first step toward nurturing competent project managers, systems engineers and mission assurance engineers for future programs. Line organizations, in conjunction with the projects, also must instill a culture that encourages all internal and external team members to forcefully and vigorously elevate concerns as far as necessary to get attention within the organization. Only then will Mission Success First become a reality.

4) Technology
Technological innovation is a key aspect in making the ?Faster, Better, Cheaper? approach a reality. Through such innovation, smaller, lighter, cheaper, and betterperforming systems can be developed. In addition, innovative processes enable quicker development cycles. To enable this vision, NASA requires adequately funded technology development, specifically aimed at Agency needs. Programs and projects must conduct long-range planning for and champion technology infusions resulting in delivery of low-risk products for project incorporation. Mechanisms which minimize technology infusion risk, such as the New Millennium Program, should be employed to flight-validate high risk technologies prior to their use on science missions.

Agenda for the Future
The Mars Climate Orbiter Mishap Investigation Board perceives its recommendations as the first step in an agenda that will be revisited and adjusted on an ongoing basis. The aim is to make Mission Success First a way of life — a concern and responsibility for everyone involved in NASA programs. The recommendations of this report must trigger the first wave of changes in processes and work habits that will make Mission Success First a reality. To implement this agenda with a sense of urgency and propagate it throughout the Agency, NASA Headquarters and the NASA centers must address the recommendations presented in this report. NASA must further assign responsibility to an organization (such as the Office of the Chief Engineer) for including the recommendations in Agency policy and in training courses for program and project management. These actions will ensure that Mission Success First serves as a beacon to guide NASA as the future unfolds. Page 10
- 10 -

1. Introduction
Background
In 1993, NASA started the Mars Surveyor Program, with the objective of conducting a series of missions to explore Mars. A Mars Program Office was established and given the responsibility of defining objectives for sending two missions to Mars at each biennial launch opportunity, culminating in return of a sample of Martian material to Earth. For each launch opportunity, the Jet Propulsion Laboratory established a project office to manage development of specific spacecraft and mission operations. In 1995, the Mars Program Office identified two missions for launch in late 1998/early 1999: the Mars Climate Orbiter and the Mars Polar Lander. The Jet Propulsion Laboratory created the Mars Surveyor Project ’98 Office, which was responsible for designing the missions, developing both spacecraft and all payload elements, and integrating, testing and launching both flight systems. In March of 1996, subsequent to the formation of the project office, the Mars Surveyor Program established the Mars Surveyor Operations Project, which was tasked to perform operations of all Mars Surveyor Program missions. The Mars Climate Orbiter was launched Dec. 11, 1998, atop a Delta II launch vehicle from Cape Canaveral Air Force Station, Florida. Nine and a half months after launch, in September 1999, the spacecraft was to fire its main engine to achieve an elliptical orbit around Mars. It then was to skim through Mars’ upper atmosphere for several weeks, in a technique called aerobraking, to move into a low circular orbit. Friction against the spacecraft’s single, 5.5-meter solar array was to have lowered the altitude of the spacecraft as it dipped into the atmosphere, reducing its orbital period from more than 14 hours to 2 hours. On Sept. 23, 1999 the Mars Climate Orbiter mission was lost when it entered the Martian atmosphere on a lower than expected trajectory. On Oct. 15, 1999, the NASA Office of Space Science established the Mars Climate Orbiter Mission Failure Mishap Investigation Board — hereafter referred to as ?the Board? — and appointed Arthur G. Stephenson, Director of the Marshall Space Flight Center, as chairman of the Board. A copy of the letter establishing the Board is contained in Appendix A. On Nov. 10, 1999, the Board’s Phase I Report was released in response to the letter of October 15. That report focused on identifying the root cause and contributing factors of the Mars Climate Orbiter failure and made observations related to the Mars Polar Lander’s entry, descent and landing activities, which were planned for Dec. 3, 1999. A copy of the Phase I Report is contained in Appendix B. Page 11
- 11 -

On Jan. 3, 2000, the Office of Space Science revised the Board’s charter (see Appendix C) to broaden the area of investigation beyond the Mars Climate Orbiter failure in order to derive lessons learned and develop recommendations to benefit future NASA missions. To learn from other failure experiences, the Board looked at the additional projects listed in Appendix E. This report responds to the revised charter by first presenting findings related to the failure of the Mars Climate Orbiter — going beyond those developed in Phase I. The

report accomplishes the following actions: ??Summarizes lessons learned from the Mars Climate Orbiter,? ??Provides an idealized vision of project management,? ??Describes how NASA is currently performing project? management, ??Identifies common themes contributing to recent mission failures,? and ??Makes recommendations for improving the likelihood of mission? success in future NASA missions.

The “Faster, Better, Cheaper” Paradigm
The aim of the ?Faster, Better, Cheaper? philosophy is to encourage doing more with less. This is accomplished by enhancing innovation and productivity, while enabling new safe, cost-effective approaches to achieving mission success. The initiative in recent years has led to significant restructuring of programs and a number of successful missions. Costs were reduced and program scope — including both content and the infusion of new technology — increased at the same time. As implementation of this strategy evolved, however, the focus on cost and schedule reduction increased risk beyond acceptable levels on some NASA projects. Even now, NASA may be operating on the edge of high, unacceptable risk on some projects. These trends of increasing scope, decreasing cost and eventual, significant increase in risk are notionally illustrated in the figure below.
Increasing

Cost and Schedule Risk Scope
Desired state Evolution of Faster, Better, Cheaper Missions

Page 12 The desired state, as indicated in the figure, is the region where cost is well matched to the desired scope and risk is not significantly affected by changes in cost, schedule and scope. Ideally, cost should not be reduced — nor content increased — beyond the point where risk rises rapidly. The Board finds that implementation of the ?Faster, Better, Cheaper? philosophy must be refined at this stage in a new context: Mission Success First. For the purposes of this report, a proper emphasis on mission success encompasses the following principles: ??Emphasis on definition of a minimum set of mission success criteria? and rigorous requirements derived therefrom, ??Sufficient analysis and verification prior to launch, ensuring a high? probability of satisfying the mission success criteria, ??Assurance of sufficient robustness in the design of the mission to? maintain the health and safety of the flight systems until the mission science and/or technology objectives are achieved, even in the event of off-nominal conditions, and ??Ensuring that we will be able to learn from mission failure or? abnormalities, by being able to obtain sufficient engineering data to understand what happened and thereby design future missions to avoid

a repeat occurrence. The ?Faster, Better, Cheaper? paradigm has enabled NASA to respond to the national mandate to do more with less. In order for this paradigm to succeed in the future, we face two key challenges: the timely development and infusion of new technology into our missions, and the fostering of the Mission Success First mentality throughout the workforce, ensuring safe, cost-effective mission accomplishment. Mission Success First is the over-arching focus of this report.

The Changing Environment
Significant change has taken place in the environment for NASA projects over the past five to seven years. The ?Faster, Better, Cheaper? paradigm has been extremely successful in producing a greater number of smaller missions, with significantly shortened development cycles. Many of these missions are selected on the basis of proposals from principal investigators, who become responsible for managing all aspects of the mission through a NASA center. With freedom to operate outside traditional, NASA-specified management approaches, managers may use smaller teams and a strict ?design-to-cost? philosophy in implementing projects. One of the consequences of this approach has been increased partnering between NASA, industry, academia and other government agencies, necessitating increased and improved communications. New and innovative teaming arrangements and contracting approaches have been employed in the procurement processes. These changes have shifted accountability and required the various participants to learn new roles. Page 13
- 13 -

During the same period, the size, experience and focus of the NASA workforce and industry have also undergone significant change. The workforce has been reduced, resulting in a loss of experienced personnel in all skill categories. The primary focus of in-house work is shifting from spacecraft development and operations to new technology development. NASA management of out-of-house missions has changed from ?oversight? to ?insight? — with far fewer resources devoted to contract monitoring. NASA projects have placed increased emphasis on public education and outreach. In addition, the public is more engaged in NASA missions because there are more of them. While this has delivered the desired results — heightening public interest in our missions and increasing public understanding of our scientific advances — it has also made NASA’s failures more visible, along with our successes.

Perpetuating the Legacy
NASA is a national resource. It enjoys a legacy of excellence established by many successes that inspired the nation and the world. Policies that contributed to this legacy must now be assessed because of changes that have occurred in response to the new environment — one characterized by the need to ?do more with less.? Policies must be examined, current processes adjusted and behaviors modified to preserve NASA as a national resource and perpetuate its legacy of success in innovative scientific and technological undertakings.

Outline of the Report
This report is organized as follows. Section 2 addresses the Mars Climate Orbiter

mission. In the Phase I Report by this Board (see Appendix B), the focus was on items deemed particularly important to the Mars Polar Lander mission, then cruising toward Mars. Section 2 describes the lessons learned from the Mars Climate Orbiter mission in general. In Section 3, we offer a vision of an improved NASA culture and the characteristics of an ideal project process aimed at Mission Success First. In Section 4, we present observations of the current project management environment, based upon documented processes (see Appendix D) and our review of a number of projects (see Appendix E). We identify some common causes of project problems. In Section 5, we provide specific recommendations for bridging the gap between where we are now and where we would like to be, and suggest some metrics for measuring our progress toward the desired Mission Success First environment. A checklist for project management is also provided in Section 5. The report addresses broad issues that are important to all parties involved in the NASA program. It is intended to be widely disseminated to NASA employees, contractors and those in academic or other institutions participating in the implementation of NASA projects. Page 14
- 14 -

Agenda for the Future
The Mars Climate Orbiter Mishap Investigation Board perceives its recommendations as the first step in an agenda that will be revisited and adjusted on an ongoing basis in the future. The aim of the agenda is to make Mission Success First a way of life — a concern and responsibility for everyone involved in NASA programs. The recommendations of this report must trigger the first wave of changes in processes and work habits that will make Mission Success First a reality. To implement this agenda with a sense of urgency and propagate it throughout the Agency, NASA Headquarters and the NASA Centers should make plans to address the recommendations presented in this report, as well as other investigative reports (i.e., Spear, McDonald, Young) soon to be released. NASA must further assign an organization (such as the Office of the Chief Engineer) responsibility for including the recommendations in Agency guidance and in training courses for program and project management. These actions will ensure that Mission Success First serves as a beacon to guide NASA decisions as the future unfolds. Page 15
- 15 -

2. The Mars Climate Orbiter Mission: Observations and Lessons Learned
To better understand the issues that led to the failure of the Mars Climate Orbiter, the Mishap Investigation Board conducted a series of meetings over several months with the Jet Propulsion Laboratory and Lockheed Martin Astronautics. As part of its investigation, the Board uncovered several mistakes and deficiencies in the overall Mars Surveyor Program. Despite these deficiencies, the spacecraft operated as commanded and the mission was categorized as extremely successful until just before Mars orbit

insertion. This is a testament to the hard work and dedication of the entire Mars Climate Orbiter team. The Board recognizes that mistakes and deficiencies occur on all spacecraft projects. It is imperative for all spacecraft projects to have sufficient processes in place to catch mistakes and deficiencies before they become detrimental to mission success. Unfortunately for the Mars Climate Orbiter, the processes in place did not catch the root problem and contributing navigational factors that ultimately led to mission failure. As part of its Phase I activity, the Board identified one root cause, eight contributing causes and 10 observations. These are described in the Phase I report (see Appendix B). Subsequent Board investigations and meetings have uncovered additional observations. These observations — as well as the issues identified in the Phase I report — were compiled and consolidated into five primary issue areas:

??Systems Engineering? ??Project Management? ??Institutional Involvement? ??Communication Among Project Elements? ??Mission Assurance?
A top-level description of the observations made during the investigation follows, along with some lessons learned.

Systems Engineering
A necessary condition for mission success in all spaceflight programs is a robust, experienced systems engineering team and well thought-out systems engineering processes. The systems engineering team performs critical trade studies that help optimize the mission in terms of performance, cost, schedule and risk. Throughout mission formulation, design, development and operations, this team leads the subsystemdiscipline teams in the identification of mission risks. The systems engineers work with the project manager and the discipline engineering teams to mitigate these risks. Page 16
- 16 -

The Board saw strong evidence that the systems engineering team and the systems processes were inadequate on the Mars Climate Orbiter project. Some specific observations demonstrating that a robust systems engineering team and processes were not in place included: ??Absence of a mission systems engineer during the operations phase to provide the? bridge between the spacecraft system, the instrument system and the ground/operations system. ??Lack of identification of acceptable risk by the operations team in the context of? the ?Faster, Better, Cheaper? philosophy. ??Navigation requirements set at too high a management level, insufficient? flowdown of requirements and inadequate validation of these requirements. ??Several significant system and subsystem design and development issues,? uncovered after the launch of the Mars Climate Orbiter (the star camera glint issue and the inability of the navigation team to receive telemetry from the ground system for almost six months, for example).

??Inadequate independent verification and validation of Mars Climate Orbiter? ground software (end-to-end testing to validate the small forces ground software performance and its applicability to the software interface specification did not appear to be accomplished). ??Failure to complete — or completion with insufficient rigor — of the interface? control process, as well as verification of specific ground system interfaces. ??Absence of a process, such as a fault tree analysis, for determining ?what could go? wrong? during the mission. ??Inadequate identification of mission-critical elements throughout the mission (the? mission criticality of specific elements of the ground software that impacted navigation trajectory was not identified, for example). ??Inadequate attention, within the system engineering process, to the transition from? development to operations. ??Inadequate criteria for mission contingency planning (without the development of? a fault tree up front, there was no basis for adequate contingency planning). ??Insufficient autonomy and contingency planning to execute Trajectory Correction? Maneuver 5 and other mission-critical operations scenarios. ??A navigation strategy that was totally reliant on Earth-based, Deep Space? Network tracking of the Mars Climate Orbiter as a single vehicle traveling in interplanetary space. Mission plans for the Mars Polar Lander included alternative methods of processing this data — including using ?Near Simultaneous Tracking? of a Mars-orbiting spacecraft. These alternatives were not implemented nor were operational at the time of the Mars Climate Orbiter’s encounter with Mars. The Board found that reliance on single-vehicle, Deep Space Network tracking to support planetary orbit insertion involved considerable systems risk, due to the possible accumulation of unobserved perturbations to the long interplanetary trajectory. Page 17
- 17 -

Lessons Learned

??Establish and fully staff a comprehensive systems engineering? team at the start of each project. Ensure that the systems engineering team possesses the skills to fully engage the subsystem engineers so that a healthy communication flow is present up and down the project elements. ??Engage operations personnel early in the project, preferably during? the mission formulation phase. ??Define program architecture at the beginning of a program by? means of a thorough mission formulation process. ??Develop a comprehensive set of mission requirements early in the? formulation phase. Perform a thorough flowdown of these requirements to the subsystem level. ??Continually perform system analyses necessary to explicitly? identify mission risks and communicate these risks to all segments

of the project team and institutional management. Vigorously work with this team to make trade-off decisions that mitigate these risks in order to maximize the likelihood of mission success. Regularly communicate the progress of the risk mitigation plans and tradeoffs to project, program and institutional management. ??Develop and deploy alternative navigational schemes to single-? vehicle, Deep Space Network tracking for future planetary missions. For example, utilizing ?relative navigation? when in the vicinity of another planet is promising. ??Give consideration to technology developments addressing optical? tracking, relative state ranging and in-situ autonomous spacecraft orbit determination. Such determination should be based on nearby planetary features or Global Positioning System-type tracking.

Project Management
In order to accomplish the very aggressive Mars mission, the Mars Surveyor Program agreed to significant cuts in the monetary and personnel resources available to support the Mars Climate Orbiter mission, as compared to previous projects. More importantly, the program failed to introduce sufficient discipline in the processes used to develop, validate and operate the spacecraft, and did not adequately instill a mission-success culture that would shore up the risk introduced by these cuts. These process and project leadership deficiencies introduced sufficient risk to compromise mission success to the point of mission failure. The following are specific issues that may have contributed to that failure. Roles and responsibilities of some individuals on the Mars Climate Orbiter and Mars Surveyor Operations Project teams were not clearly specified by project management. Page 18
- 18 -

To exacerbate this situation, the mission was understaffed, with virtually no Jet Propulsion Laboratory oversight of Lockheed Martin Astronautics’ subsystem developments. Thus, as the mission workforce was reduced and focus shifted from spacecraft development to operations, several mission critical functions — such as navigation and software validation — received insufficient management oversight. Authority and accountability appeared to be a significant issue here. Recurring questions in the Board’s investigation included ?Who’s in charge?? and ?Who is the mission manager?? The Board perceived hesitancy and wavering on the part of people attempting to answer the latter question. One interviewee answered that the flight operations manager was acting like a mission manager, but is not actually designated as such. The Board found that the overall project plan did not provide for a careful handover from the development project to the very busy operations project. Transition from development to operations — as two separate teams — disrupted continuity and unity of shared purpose. Training of some new, inexperienced development team members was inadequate. Team membership was not balanced by the inclusion of experienced specialists who could serve as mentors. This team’s inexperience was a key factor in the root cause of the

mission failure (the failure to use metric units in the coding of the ?Small Forces? ground software used in trajectory modeling). This problem might have been uncovered with proper training. In addition, the operations navigation team was not intimately familiar with the attitude operations of the spacecraft, especially with regard to the attitude control system and related subsystem parameters. These functions and their ramifications for Mars Climate Orbiter navigation were fully understood by neither the operations navigation team nor the spacecraft team, due to inexperience and miscommunication. The Board found that the project management team appeared more focused on meeting mission cost and schedule objectives and did not adequately focus on mission risk. A critical deficiency in Mars Climate Orbiter project management was the lack of discipline in reporting problems and insufficient follow-up. The primary, structured problem-reporting procedure used by the Jet Propulsion Laboratory — the Incident, Surprise, Anomaly process — was not embraced by the whole team. Project leadership did not instill the necessary sense of authority and responsibility in workers that would have spurred them to broadcast problems they detected so those problems might be articulated, interpreted and elevated to the highest appropriate level, until resolved. This error was at the heart of the mission’s navigation mishap. If discipline in the problem reporting and follow-up process had been in place, the operations navigation team or the spacecraft team may have identified the navigation discrepancies, using the Incident, Surprise, Anomaly process, and the team would have made sure those discrepancies were resolved. Furthermore, flight-critical decisions did not adequately involve the mission scientists who had the most knowledge of Mars, the instruments and the mission science objectives. This was particularly apparent in the decision not to perform the fifth Trajectory Correction Maneuver prior to Mars orbit insertion. Page 19
- 19 -

In summary, the Mars Surveyor Program increased the scope of the operations project and reduced personnel and funding resources. These actions went unchallenged by the project, causing it to operate beyond the edge of acceptable risk. In short, they went beyond the boundaries of Mission Success First.

Lessons Learned

??Roles, responsibilities and accountabilities must be made explicit? and clear for all partners on a project, and a visible leader appointed over the entire operation. ??A cohesive team must be developed and involved in the project? from inception to completion. ??Training and mentoring using experienced personnel should be? institutionalized as a process to preserve and perpetuate the wisdom of institutional memory as well as to reduce mission risk. ??Steps must be taken to aggressively mitigate unresolved problems? by creating a structured process of problem reporting and resolution. Workers should be trained to detect, broadcast, interpret and elevate problems to the highest level necessary until resolved. ??Lessons learned from such problems must be articulated,?

documented and made part of institutional and Agency memory (see ?Lessons Learned Information System? on the World Wide Web athttp://llis.gsfc.nasa.gov). ??Acceptable risk must be defined and quantified, wherever possible,? and disseminated throughout the team and the organization to guide all activities in the context of Mission Success First.

Institutional Involvement
All successful spacecraft projects require strong engagement and participation of the project management team, the spacecraft discipline team, the systems engineering team, the operations team, the science team and the organization’s institutional management. For the Mars Climate Orbiter and the Mars Polar Lander, there clearly appeared to be little or no ownership of these missions within the Jet Propulsion Laboratory’s institutional organization until after the Mars Climate Orbiter mission failure occurred. In an effort to reduce costs, the project management team elected not to fully involve the Jet Propulsion Laboratory’s technical divisions in spacecraft design and development activities. They also did not appear to properly engage the safety and mission assurance group during the operations phase. Unfortunately, key oversight in a few critical discipline areas — propulsion, attitude control, navigation, flight software and systems — could have identified problems and brought issues to the attention of institutional management at the Jet Propulsion Laboratory as well as to project management. Because the Jet Propulsion Laboratory’s technical divisions were disengaged from the Mars Page 20
- 20 -

Climate Orbiter mission, there was little or no ownership of the mission beyond the flight project and a few organizational managers. The lack of institutional involvement resulted in a project team culture that was isolated from institutional experts at the Jet Propulsion Laboratory. The project team did not adequately engage these experts when problems arose, they did not elevate concerns to the highest levels within the contractor and they did not receive the proper coaching and mentoring during the project life cycle to ensure mission success. In short, there was lack of institutional involvement to help bridge the transition as old, proven ways of project management were discontinued and new, unproven ways were implemented.

Lessons Learned

??In the era of ?Faster, Better, Cheaper,? projects and line? organizations need to be extremely vigilant to ensure that a Mission Success First attitude propagates through all levels of the organization. A proper balance of contractor and project oversight by technical divisions at NASA field centers is required to ensure mission success and to develop a sense of ownership of the project by the institution. ??The Agency, field centers and projects need to convey to project? team members and line organizations that they are responsible for the success of each mission. NASA needs to instill a culture that encourages all internal and external team members to forcefully

and vigorously elevate concerns as far as necessary to get attention — either vertically or horizontally within the organization. ??Organizations should provide robust mechanisms for training,? mentoring and oversight of project managers and other leaders of project teams. An aggressive mentoring and certification program should be instituted to nurture competent project managers, systems engineers and mission assurance engineers to support future programs. ? Line managers at the field centers should be held accountable for all missions at their centers. As such, they should be held accountable for getting the right people to reviews and ensuring the right questions are asked at meetings and reviews to uncover mission-critical issues and concerns. They also must be accountable to ensure adequate answers are provided in response to their questions. This factor was missing on the Mars Climate Orbiter project. Let us be clear that we do not advocate returning to the old approach, wherein NASA civil servants performed oversight on every task performed by the system contractor. The need, rather, is for NASA to conduct rigorous reviews of the Page 21
- 21 -

contractor’s and the team’s work — something that was not done on Mars Climate Orbiter.

Communications Among Project Elements
The Mars Climate Orbiter project exhibited inadequate communications between project elements during its development and operations phases. This was identified as a contributing cause to the mission failure in the Board’s Phase I report (see Appendix B). A summary of specific inadequacies follows: ??Inadequate communications between project elements led to a lack of cross-? discipline knowledge among team members. Example: the operations navigation team’s lack of knowledge regarding the designed spacecraft’s characteristics, such as the impact of solar pressure on torque. ??There was a lack of early and constant involvement of all project elements? throughout the project life cycle. Example: inadequate communications between the development and operations teams. ??Project management did not develop an environment of open communications? within the operations team. Example: inadequate communications between operations navigation staff and the rest of the Mars Surveyor Operations team supporting the Mars Climate Orbiter. ??There was inadequate communication between the project system elements and? the institutional technical line divisions at the Jet Propulsion Laboratory. Example: lack of knowledge by the Jet Propulsion Laboratory’s navigation section regarding analyses and assumptions made by Mars Climate Orbiter operations navigators.

Lessons Learned
A successful project is a result of many factors: a good design, a good implementation strategy, a good understanding of how the project will function during the operations phase and project members with good technical skills. A project can have all these elements and still fail, however, because of a lack of good communications within the project team. Good communications within a project — including the contractors and science team elements — is fostered when the following environment is put into place by project management at the beginning of project formulation and maintained until the end of the mission: ??Project managers lead by example. They must be constant? communicators, proactively promoting and creating opportunities for communication. ??Communications meetings must be regular and frequent, and? attendance must be open to the entire project team, including Page 22
- 22 -

contractors and science elements — thus ensuring ample opportunity for anyone to speak up. During critical periods, daily meetings should be held to facilitate dissemination of fast-breaking news and rapid problem solving. ??An open atmosphere must be created, where anyone can raise an? issue or voice an opinion without being rejected out of hand. There must also be a constant and routine flow of information up, down and sideways, through formal and informal channels, making information available to all parties. ??If an issue is raised — no matter by whom — resolution must be? pursued in an open fashion with all involved parties. ??Government, industry and academia must work together as a? cohesive team to resolve issues. A project philosophy must be established to communicate any problem or concern raised by these participants to the NASA project office. That is, there must be no filtering of concerns or issues. This allows proper resources to be applied quickly for effective issue resolution. It requires an environment of trust to be created between the government, industry and academic components involved in the mission. ??Key project team members must be co-located during critical? periods, such as project design trade studies and critical problem solving. Co-location makes it easier for communication to occur across systems and organizations.

Mission Assurance
The Mars Climate Orbiter program did not incorporate a project-level mission assurance function during the operations phase. The Board observed lapses in the mission assurance function, such as the absence of an Incident, Surprise, Anomaly submittal

documenting anomalies impacting the Angular Momentum Desaturation module. The root cause of the mission failure may have been eliminated had there been a rigorous approach to the definition of mission-critical software — thereby allowing the aforementioned module to receive the appropriate level of review. In addition, software verification and validation at the module level and of the navigation algorithms at the subsequent system level did not detect the error, though there was evidence of the anomaly. A rigorous application of internal and external discipline engineering support in the review cycle, with participation from knowledgeable independent reviewers, also might have uncovered the discrepancy.

Lessons Learned

??A strong mission assurance function should be present in all? project phases. In addition to advising and assisting projects in implementing lower level, detailed mission assurance activities such as system safety and reliability analyses, it should also take Page 23
- 23 -

on the higher level, oversight function of ensuring that robust assurance processes are at work in the project. Example: mission assurance should ensure the proper and effective functioning of a problem-reporting process such as the Incident, Surprise, Anomaly process that failed to work effectively in the operational phase of the Mars Climate Orbiter mission. ??Rigorous discipline must be enforced in the review process. Key? reviews should have the proper skill mix of personnel for all disciplines involved in the subject matter under review. Independent reviewers or peers with significant relevant knowledge and experience are mandatory participants. ??From the simplest component or module to the most complex? system, end-to-end verification and validation conducted via simulation or testing of hardware/software must be structured to permit traceability and compliance with mission and derived requirements. Integrated hardware/software testing is a must to validate the system in a flight-like environment. Independent verification and validation of software is essential, particularly for mission-critical software functions. ??Final end–to-end verification and validation of all mission-critical? operational procedures (Trajectory Correction Maneuver 5, for example) must be performed. ??The definition of mission-critical software for both ground and? flight must be rigorous to allow the software development process to provide a check-and-balance system. Page 24
- 24 -

3. A New Vision for NASA Programs and Projects
In the future, NASA’s culture must be one driven by improved mission success within the context of a continued adherence to the ?Faster, Better, Cheaper? paradigm. We propose to establish Mission Success First as the highest priority within all levels of NASA. To do so, NASA’s culture — and current techniques for program and project management — must evolve. This new vision relies on implementing specific recommendations to improve mission success in the future. Reflecting on recent mishaps, a return to long, expensive projects is simply not warranted. However, the ?Faster, Better, Cheaper? mantra cannot become an excuse for reduced attention to quality or to mission success. In this section, a vision of NASA’s new culture and suggested methods of managing its projects are described.

Cultural Vision
NASA’s culture in the 21
st

century reemphasizes the need for overall mission success. At all levels in the organization, mission success is the highest priority. Every person in the Agency and its contractor organizations is focused on providing quality products and services. This includes searching for errors and potential failure modes and correcting them as early in the process as possible. Their confidence in their own individual capabilities is tempered with plenty of healthy skepticism. They are invigorated by the basic scientific method of thinking. They review and test, and ask others to independently review and test. They realize their jobs require scrupulous attention to details. All individuals feel ownership and accountability for their work. Mission success and good process discipline are emphasized daily, both in words and in actions. As they develop specific products (hardware components, software components or processes), they maintain their ownership over the full life cycle of that product, understanding how the product is being used, validating the interfaces and verifying that its end use is consistent with its intended use. They develop, understand, manage and communicate their risk assessments. Keeping a lookout for problems internal and external to their area, these responsible engineers look beyond their product needs and support wider systems engineering efforts to ensure a successful, robust system design. They feel responsible for the overall system in addition to their unique part, allowing more system-level issues to be identified and resolved early in the project. These individuals understand that the only real success is overall mission success. Page 25
- 25 -

NASA management at all levels promotes open communications (including bad news) and encourages inter-center cooperation and joint development efforts at the system and subsystem levels. Management provides strong leadership of badgeless teams, with civil servants and contractors alike involved in design, development, testing and early mission operations. Management ensures sufficient resources to promote continuous interaction between all elements of NASA, understanding that the sum is truly greater than the parts.

NASA Project Management
Our vision of an ideal project team builds on the foundation established in NASA Procedures and Guidelines (NPG 7120.5a) and includes some new insight into how projects should be executed.

Mission Success Criteria
In concert with NASA Headquarters and center-level senior management, program managers negotiate multi-mission objectives and associated top-level mission success criteria for the program. Subsequently, at the inception of each project, the project manager works with the program management to flow these needs into the project, thereby establishing specific project-level mission objectives and mission success criteria. This information is then flowed down through the project, resulting in system-level and subsystem-level requirements and associated mission success criteria, which will be baselined at the beginning of the project and managed throughout its life cycle. The project team strives for quantifiable, measurable mission success criteria whenever possible. Status reports on mission success criteria are delivered to program and senior level management throughout the project life cycle to ensure that mission success is not being eroded. A coordinated understanding of expected mission success levels is communicated throughout the organization and to the American public. Adequate resources are provided during all phases of the mission to assure that mission success criteria are met. A test of resources versus mission success criteria is constantly made during the development and operational phases. If there is an indication of inadequate resources, a decision is made to reduce the mission success criteria to match resources. If the mission success criteria drops below a minimum acceptable scientific and/or technical level, and no added resources are available, project cancellation is considered.

Technology Needs
Technology is the better part of the ?Faster, Better, Cheaper? paradigm. Technology advancements can lead to improved spacecraft systems, science components, spacecraft autonomous operations, ground systems or mission operations processes. Some generic spacecraft technology improvements (propulsion and guidance, navigation and control hardware, for example) are continuously in development at various NASA centers, and Page 26
- 26 -

serve multiple programs. Other technology improvements are initiated to solve specific mission needs. In our vision, NASA invests significantly more of its annual budget in both evolutionary and revolutionary technologies to improve future mission success. Evolutionary technologies represent continual improvement in systems design and operations. Revolutionary technologies — sometimes called breakthrough technologies — represent quantum leaps in capability and generally have high development risks, but may result in large payoffs. Good project definition requires early, detailed program-level engineering. At the program level, a strong, robust strategy spanning multiple missions is developed to achieve program objectives. This work results in the identification of specific technology

needs required for individual missions and projects, and becomes a driving factor in the infusion of technology into projects. These technology roadmaps are embraced by Agency personnel and provide strategic direction for technology development. Proper long-range planning and scheduling is required to begin development of these technologies well in advance of project ?need? dates. In our vision, the efforts to develop these technologies are underway in a timeframe such that the technologies can be matured to high technology readiness levels prior to being baselined into a project. Regardless of the development risk, these technologies are matured before project baselining in order that they may result in the lowest possible deployment risks — thereby allowing the projects to reap the benefits without incurring the risks.

Forming the Team: Project Staffing
Project success is strongly correlated to project team dynamics. This requires that projects and institutional elements interact continuously throughout the life cycle of the project. Senior management must define clear roles and responsibilities between projects and other elements of the organization. To maximize success, senior management assures selection of experienced project managers, based on previous project management training and field experience. Prospective candidates have the ability to select, motivate and lead a close-knit project team. They also possess the ability to interact well across organizational elements (centers, enterprises and contractor/ academic lines). A junior assistant project manager is also assigned to the project to receive mentoring and on-the-job training — thus becoming an investment for future Agency needs. Project team formation is based on team members with a good track record for technical, cost and schedule performance, along with the ability to take ownership and continually assess risk, as well as manage and communicate status. Team members are committed to the project and provide continuity throughout the life cycle of the project or mission. Page 27
- 27 -

One of NASA’s greatest assets is its people, many of whom are truly world-class experts. Yet utilization of these people across centers is inadequate due to lack of awareness of individual abilities and performance outside their center or discipline group. In our vision, there is more inter-center participation in these projects, using discipline specialists across the Agency for direct project support and staffing of review teams.

Project Management
In concert with senior management at the centers and NASA Headquarters, program managers establish mission success criteria at the beginning of each project. Project management works with program-level management to develop top-level requirements consistent with these success criteria. The project manages the flowdown of mission success criteria and associated top-level project requirements to all levels of the project, thus ensuring that mission success is not being compromised. Under project management leadership, Mission Success First is practiced and preached continually throughout the project. The project manager removes barriers and disconnects within the project between development and operations groups; between subsystem developers and system integration groups; and between government, contractors and the science community.

The project manager further ensures continuity of key personnel throughout the life cycle of the project. In the proposal stage, project plans are usually defined only in sufficient detail to allow for a reasonable assessment of cost and schedule, permitting contractor selection and overall project establishment. In our vision, to prepare for the subsequent baselining of the project, a thorough review of the project plan is performed. This is the first opportunity to think the project through from start to completion, based on contractor selection and proposed costs and schedule targets. It is also the first opportunity to avoid pitfalls. Adequate cost and schedule reserves are baselined into the project to protect against future delays and overruns. Disciplined planning, organization and staffing of project tasks is reviewed from the top down to ensure a ?good start.? When a project plan is baselined, cost, schedule and content plans are traditionally frozen and subsequent project efforts are measured with respect to this baseline. In our vision, at this early stage in the project, risks are identified at all levels as well and controlled in a similar manner, becoming the fourth dimension to the project. These risks are quantified and communicated throughout the project team as well as to senior management, much the way cost, schedule and content are assessed and communicated. During project evolution, risk management may entail trading risk on a system-by-system basis to ensure overall mission objectives are still being satisfied. Additionally, the baselined project plan contains sufficient flexibility to make adjustments to the plan, based on unanticipated issues that may surface at major design reviews. Without this flexibility, these project ?challenges? present additional risks downstream in the project Page 28
- 28 -

life cycle. In our vision, project management is prepared to request necessary cost or schedule relief when the situation warrants, thereby controlling risk and satisfying mission success criteria. Finally, the project manager promotes continuous capture of knowledge throughout the project. Data collection and ?document as you go? behavior are typical of routine project execution, allowing for smooth personnel transitions within the project and development of lessons-learned for possible use in later phases of the project and in future projects Agency-wide.

Science as an Integral Part of the Team
The ultimate objective of most NASA missions is to accomplish scientific and/or technical research (e.g., the New Millennium Program) and study. True mission success requires that scientists be intimately involved in the entire mission — from project initiation through mission completion. As part of mission definition/concept teams, scientists define science requirements, develop an understanding of expected spacecraft capabilities and limitations, conduct trade studies and influence spacecraft design to ensure adequate science return within project limitations. Scientists participate in project-level decisions, in systems engineering studies, in spacecraft development and in mission planning and operations. Participating throughout the project life cycle, scientists recognize and concur on the proper balance between engineering needs and science needs, in order to maximize the ability of the mission to accomplish the desired scientific objectives. For example, in a planetary mission

involving landing, safely landing on the planet must take precedence over science when spacecraft resources are allocated.

Systems Engineering
Systems engineering ensures that all top-level project requirements are directly derived from the identified and controlled mission success criteria, and that these top-level project requirements and mission success criteria are appropriately flowed down to lower levels. Configuration management of these requirements — and development of a traceability matrix linking requirements to implementation — occurs within systems engineering. Requirements are baselined early. Disciplined, documented change control processes are used to manage changes. Validation and verification plans are developed to ensure current work plans address and implement all ground and onboard requirements. This linking of mission success criteria, requirements, implementation and verification plans is reviewed at all major project design reviews and flight readiness reviews. Systems engineering ties the systems together and validates end-to-end supportability. Resource allocations between systems (power and telemetry, for example) are performed and controlled. All interfaces are tested and verified within and across subsystems. Systems engineers engage all disciplines to support integrated mission analyses using nominal and dispersed conditions. ?Out of family,? or anomalous, scenarios are also Page 29
- 29 -

identified, analyzed and simulated to determine mission robustness. Results of these studies include identification of disconnects and weak links, and validation of mission risk assessments. Trade studies are conducted throughout the project to continuously address risk. These studies are performed repetitively as spacecraft systems and mission operations plans evolve during the development phase. During the operational phase, systems engineering continues in this role, assessing mission risks and behavior under actual conditions. Attention to integrated risk management on the project is a key responsibility of systems engineering. For all mission phases, projects use Fault Tree Analyses, Failure Modes and Effects Analyses and Probabilistic Risk Assessments to identify what could go wrong. Each risk has an associated ?risk owner,? who is responsible for managing that risk. Like other ?earned value? concepts in project management, risk is continuously addressed throughout the project. The traditional ?earned value? approach enables management to objectively measure how much work has been accomplished on a project and compare that statistic with planned-work objectives determined at project startup. The process requires the project manager to plan, budget and schedule the work in the baseline plan, which contained the ?planned value.? As work is accomplished, it becomes ?earned? and is reflected as a completed task in the project. We envision something analogous for documenting and mitigating risks. Finally, risks are reported and risk mitigation techniques are rebaselined at all major project reviews, thereby ensuring mission success is not compromised.

Mission Assurance
The mission assurance function advises and assists projects in implementing a variety of lower-level, detailed, technical mission assurance activities, such as system safety and reliability analysis. It also conducts a higher-level oversight function, guaranteeing that

robust assurance processes — such as the problem reporting and corrective action process — are at work in the project. On one hand, mission assurance works shoulder-to-shoulder with the project. On the other, it maintains its independence, serving as a separate set of eyes that continuously oversee project developmental and operational efforts to ensure that mission success is not compromised. Mission assurance works with and reports to project management, yet maintains a separate reporting chain to center and even Agency senior management, should such measures become necessary to assure safety or mission success.

System and Subsystem Development Teams
At the core of the project are the development engineers, who are responsible for designing ground and flight system and subsystem components, including hardware, software or procedures. At the beginning of the project, the development teams learn how their product fits into the bigger picture and how end users intend to use their Page 30
- 30 -

product. They understand requirements and develop robust components that meet or exceed customer expectations. During development, they identify and manage risks. They take ownership. They understand, document and communicate limitations of their system, and they advocate solid reviews — internally, externally and continuously. Catching errors early and correcting them is a high priority for these teams. During project planning, they advocate development of prototype versions and early testing to uncover design errors, especially for higher-risk components. They perform comprehensive unit testing and are intimately involved with systems integration testing. Their philosophy is, ?Test, test and test some more.? Their motto is:

“Know what you build. Test what you build. Test what you fly. Test like you fly.”
Whether developing onboard spacecraft components or ground support components, these teams take particular care to identify mission-critical components and handle these with special focus. When a component is anticipated to be derived from a heritage component (as in the instance of software or hardware reuse), careful evaluation and testing is performed to ensure applicability and reusability within the new mission framework, once again considering robust mission scenarios.

Project Review Teams
In our vision, all review teams are established early in the project. The continuity of these teams is managed over the full life cycle of the project, utilizing key personnel. The review teams make commitments to the project to provide resources as specified in the project plan. Project management makes commitments to the review teams by establishing adequate project scheduling for supporting reviews and by implementing review team recommendations as needed. The specific objectives and scope of each review team are established up front and agreed upon by the project manager and senior

management. Establishing proper review teams is a top priority of project management and senior management in the line organization. Participation by the best experts inside and outside the Agency should be sought. ?Peer review teams? are established to provide a second set of eyes to review design, development, testing and operations. These teams are composed of people inside and outside the project who posses significant technical expertise in the relevant field. Peer team membership is balanced between peers on the project, line organizational personnel within the center, and external support from other centers, industry, other government organizations and/or academic institutions. Peer review results are reported to higherlevel review boards. Page 31
- 31 -

A ?red team? is established to study mission scenarios, to ensure operational readiness and to validate risks. Team membership is formed from personnel outside the project and generally external to the lead center. The team is composed of experienced veterans as well as ?newer? individuals with fresh, innovative ideas. This team provides an independent, aggressive, almost adversarial — yet helpful — role, addressing all levels of the project from high-level requirements down through subsystem design. Key review items include: ensuring system success and reliability; reviewing overall system design and design decisions; reviewing system safety and reliability analyses and risk assessments; reviewing planned and completed testing; and reviewing operational processes, procedures and team preparation. Red team review results and recommendations are reported to the project manager and the project team, as well as senior level management at the centers.

Mission Operations: Preparation and Execution
The role of the operations personnel in the project begins with the initial formation of the project team. A deputy project manager for operations is assigned and a small team is created to consider mission operations from the outset. Rigorous robust operations scenarios are conceived and assessed as part of formulating system design requirements. Operations plays an important role in the formulation phase of the project, prior to project approval. The core operations team provides a mechanism for capturing and improving knowledge as systems are developed and tested, and brings additional team members up to speed as launch approaches. Together with a core team of development personnel, operations performs high fidelity, pre-launch, end-to-end simulations to validate procedures, system performance and mission preparedness, as well as to solidify team cohesion. These end-to-end simulations exercise all nominal and contingency procedures under a variety of dispersed initial conditions, using flight plans and procedures already under strict configuration control. Mission rules are developed using the engineering team’s expertise. These rules are exercised during simulations to train the operations team in real time decision processes and discipline. Use of standardized procedures and forms for anomaly reporting is exercised. Following launch, the full flight team participates in frequent routine discussions addressing current mission status, upcoming events and plans and near-term decisions to be made. A poll of team members is conducted during these meetings to discuss

individual status, anomalies and discrepancies in their areas. For critical events, colocation of personnel is strongly encouraged in order to promote quick, effective decision-making and contingency replanning. Page 32
- 32 -

Vision Summary
Our emerging Mission Success First vision focuses on mission success by utilizing every individual in the organization to continuously employ solid engineering discipline, to take personal ownership for their product development efforts, and to continuously manage risk in order to design, develop and deliver robust systems capable of supporting nominal and contingency mission scenarios. Program-level and project-level planning address and champion technology infusion. This requires long-range planning and technology investments, resulting in delivery of low-risk products for project incorporation. Program and project mission success criteria and requirements are established at the outset to enable early, thorough project staffing and formulation. Systems engineering, flight operations personnel, mission assurance personnel and scientists are integrated into the project throughout its life cycle. Peer reviews and red teams are formed at the beginning of the project. They are knowledgeable of the project’s activities without becoming part of the project team itself, in order to maintain their independence. Finally, they support sustained involvement of key personnel. Spanning the full life cycle of the project, our vision includes testing, testing and more testing, conducted as early as possible in the work plans. Future projects increase attention to early and ongoing systems analysis and integration. Risks are identified early in the project and continuously managed in a quantifiable manner much the way cost, schedule and content are managed. These risk quantities are frequently reported to senior management, and a coordinated understanding of expected mission success levels is communicated throughout the organization and to the American pu



doc_118830235.docx