text:
This action might not be possible to undo. Are you sure you want to continue?
Part Two
Why The Accident Occurred
Many accident investigations do not go far enough. They identify the technical cause of the accident, and then connect it to a variant of "operator error" – the line worker who forgot to insert the bolt, the engineer who miscalculated the stress, or the manager who made the wrong decision. But this is seldom the entire issue. When the determinations of the causal chain are limited to the technical flaw and individual failure, typically the actions taken to prevent a similar event in the future are also limited: fix the technical problem and replace or retrain the individual responsible. Putting these corrections in place leads to another mistake – the belief that the problem is solved. The Board did not want to make these errors. Attempting to manage high-risk technologies while minimizing failures is an extraordinary challenge. By their nature, these complex technologies are intricate, with many interrelated parts. Standing alone, the components may be well understood and have failure modes that can be anticipated. Yet when these components are integrated into a larger system, unanticipated interactions can occur that lead to catastrophic outcomes. The risk of these complex systems is increased when they are produced and operated by complex organizations that also break down in unanticipated ways. In our view, the NASA organizational culture had as much to do with this accident as the foam. Organizational culture refers to the basic values, norms, beliefs, and practices that characterize the functioning of an institution. At the most basic level, organizational culture defines the assumptions that employees make as they carry out their work. It is a powerful force that can persist through reorganizations and the change of key personnel. It can be a positive or a negative force. In a report dealing with nuclear wastes, the National Research Council quoted Alvin Weinberg's classic statement about the "Faustian bargain" that nuclear scientists made with society. "The price that we demand of society for this magical energy source is both a vigilance and a longevity of our social institutions that we are quite unaccustomed to." This is also true of the space program. At NASA's urging, the nation committed to building an amazing, if compromised,
Report Volume I
vehicle called the Space Shuttle. When the agency did this, it accepted the bargain to operate and maintain the vehicle in the safest possible way. The Board is not convinced that NASA has completely lived up to the bargain, or that Congress and the Administration has provided the funding and support necessary for NASA to do so. This situation needs to be addressed – if the nation intends to keep conducting human space flight, it needs to live up to its part of the bargain. Part Two of this report examines NASA's organizational, historical, and cultural factors, as well as how these factors contributed to the accident. As in Part One, this part begins with history. Chapter 5 examines the post-Challenger history of NASA and its Human Space Flight Program. This includes reviewing the budget as well as organizational and management history, such as shifting management systems and locations. Chapter 6 documents management performance related to Columbia to establish events analyzed in later chapters. The chapter reviews the foam strikes, intense schedule pressure driven by an artificial requirement to deliver Node 2 to the International Space Station by a certain date, and NASA management's handling of concerns regarding Columbia during the STS-107 mission. In Chapter 7, the Board presents its views of how high-risk activities should be managed, and lists the characteristics of institutions that emphasize high-reliability results over economic efficiency or strict adherence to a schedule. This chapter measures the Space Shuttle Program's organizational and management practices against these principles and finds them wanting. Chapter 7 defines the organizational cause and offers recommendations. Chapter 8 draws from the previous chapters on history, budgets, culture, organization, and safety practices, and analyzes how all these factors contributed to this accident. This chapter captures the Board's views of the need to adjust management to enhance safety margins in Shuttle operations, and reaffirms the Board's position that without these changes, we have no confidence that other "corrective actions" will improve the safety of Shuttle operations. The changes we recommend will be difficult to accomplish – and will be internally resisted.
August 2003
Report Volume I
August 2003
From Challenger to Columbia
The Board is convinced that the factors that led to the Columbia accident go well beyond the physical mechanisms discussed in Chapter 3. The causal roots of the accident can also be traced, in part, to the turbulent post-Cold War policy environment in which NASA functioned during most of the years between the destruction of Challenger and the loss of Columbia. The end of the Cold War in the late 1980s meant that the most important political underpinning of NASA's Human Space Flight Program – U.S.-Soviet space competition – was lost, with no equally strong political objective to replace it. No longer able to justify its projects with the kind of urgency that the superpower struggle had provided, the agency could not obtain budget increases through the 1990s. Rather than adjust its ambitions to this new state of affairs, NASA continued to push an ambitious agenda of space science and exploration, including a costly Space Station Program. If NASA wanted to carry out that agenda, its only recourse, given its budget allocation, was to become more efficient, accomplishing more at less cost. The search for cost reductions led top NASA leaders over the past decade to downsize the Shuttle workforce, outsource various Shuttle Program responsibilities – including safety oversight – and consider eventual privatization of the Space Shuttle Program. The program's budget was reduced by 40 percent in purchasing power over the past decade and repeatedly raided to make up for Space Station cost overruns, even as the Program maintained a launch schedule in which the Shuttle, a developmental vehicle, was used in an operational mode. In addition, the uncertainty of top policymakers in the White House, Congress, and NASA as to how long the Shuttle would fly before being replaced resulted in the delay of upgrades needed to make the Shuttle safer and to extend its service life. The Space Shuttle Program has been transformed since the late 1980s implementation of post-Challenger management changes in ways that raise questions, addressed here and in later chapters of Part Two, about NASA's ability to safely
Report Volume I
operate the Space Shuttle. While it would be inaccurate to say that NASA managed the Space Shuttle Program at the time of the Columbia accident in the same manner it did prior to Challenger, there are unfortunate similarities between the agency's performance and safety practices in both periods.
The inherently vulnerable design of the Space Shuttle, described in Chapter 1, was a product of policy and technological compromises made at the time of its approval in 1972. That approval process also produced unreasonable expectations, even myths, about the Shuttle's future performance that NASA tried futilely to fulfill as the Shuttle became "operational" in 1982. At first, NASA was able to maintain the image of the Shuttle as an operational vehicle. During its early years of operation, the Shuttle launched satellites, performed on-orbit research, and even took members of Congress into orbit. At the beginning of 1986, the goal of "routine access to space" established by President Ronald Reagan in 1982 was ostensibly being achieved. That appearance soon proved illusory. On the cold morning of January 28, 1986, the Shuttle Challenger broke apart 73 seconds into its climb towards orbit. On board were Francis R. Scobee, Michael J. Smith, Ellison S. Onizuka, Judith A. Resnick, Ronald E. McNair, Sharon Christa McAuliffe, and Gregory B. Jarvis. All perished. Rogers Commission On February 3, 1986, President Reagan created the Presidential Commission on the Space Shuttle Challenger Accident, which soon became known as the Rogers Commission after its chairman, former Secretary of State William Rogers. The Commission's report, issued on June 6, 1986, concluded that the loss of Challenger was caused by a failure of the joint and seal between the two lower segments of the right Solid Rocket Booster. Hot gases blew past a rubber O-ring in the joint, leading to a structural failure and the explosive burnAugust 2003
ing of the Shuttle's hydrogen fuel. While the Rogers Commission identified the failure of the Solid Rocket Booster joint and seal as the physical cause of the accident, it also noted a number of NASA management failures that contributed to the catastrophe. The Rogers Commission concluded "the decision to launch the Challenger was flawed." Communication failures, incomplete and misleading information, and poor management judgments all figured in a decision-making process that permitted, in the words of the Commission, "internal flight safety problems to bypass key Shuttle managers." As a result, if those making the launch decision "had known all the facts, it is highly unlikely that they would have decided to launch." Far from meticulously guarding against potential problems, the Commission found that NASA had required "a contractor to prove that it was not safe to launch, rather than proving it was safe."1 The Commission also found that NASA had missed warning signs of the impending accident. When the joint began behaving in unexpected ways, neither NASA nor the Solid Rocket Motor manufacturer Morton-Thiokol adequately tested the joint to determine the source of the deviations from specifications or developed a solution to them, even though the problems frequently recurred. Nor did they respond to internal warnings about the faulty seal. Instead, Morton-Thiokol and NASA management came to see the problems as an acceptable flight risk – a violation of a design requirement that could be tolerated.2 During this period of increasing uncertainty about the joint's performance, the Commission found that NASA's safety system had been "silent." Of the management, organizational, and communication failures that contributed to the accident, four related to faults within the safety system, including "a lack of problem reporting requirements, inadequate trend analysis, misrepresentation of criticality, and lack of involvement in critical discussions."3 The checks and balances the safety system was meant to provide were not working. Still another factor influenced the decisions that led to the accident. The Rogers Commission noted that the Shuttle's increasing flight rate in the mid-1980s created schedule pressure, including the compression of training schedules, a shortage of spare parts, and the focusing of resources on near-term problems. NASA managers "may have forgotten–partly because of past success, partly because of their own well-nurtured image of the program–that the Shuttle was still in a research and development phase."4 The Challenger accident had profound effects on the U.S. space program. On August 15, 1986, President Reagan announced that "NASA will no longer be in the business of launching private satellites." The accident ended Air Force and intelligence community reliance on the Shuttle to launch national security payloads, prompted the decision to abandon the yet-to-be-opened Shuttle launch site at Vandenberg Air Force Base, and forced the development of improved expendable launch vehicles.6 A 1992 White House advisory committee concluded that the recovery from the Challenger
Report Volume I
• "The faulty Solid Rocket Motor joint and seal must be changed. This could be a new design eliminating the joint or a redesign of the current joint and seal. No design options should be prematurely precluded because of schedule, cost or reliance on existing hardware. All Solid Rocket Motor joints should satisfy the following: • "The joints should be fully understood, tested and verified." • "The certification of the new design should include: • Tests which duplicate the actual launch configuration as closely as possible. • Tests over the full range of operating conditions, including temperature." • "Full consideration should be given to conducting static firings of the exact flight configuration in a vertical attitude." • "The Shuttle Program Structure should be reviewed. The project managers for the various elements of the Shuttle program felt more accountable to their center management than to the Shuttle program organization." • "NASA should encourage the transition of qualified astronauts into agency management positions." • "NASA should establish an Office of Safety, Reliability and Quality Assurance to be headed by an Associate Administrator, reporting directly to the NASA Administrator. It would have direct authority for safety, reliability, and quality assurance throughout the agency. The office should be assigned the work force to ensure adequate oversight of its functions and should be independent of other NASA functional and program responsibilities." • "NASA should establish an STS Safety Advisory Panel reporting to the STS Program Manager. The charter of this panel should include Shuttle operational issues, launch commit criteria, flight rules, flight readiness and risk management." • "The Commission found that Marshall Space Flight Center project managers, because of a tendency at Marshall to management isolation, failed to provide full and timely information bearing on the safety of flight 51-L [the Challenger mission] to other vital elements of Shuttle program management … NASA should take energetic steps to eliminate this tendency at Marshall Space Flight Center, whether by changes of personnel, organization, indoctrination or all three." • "The nation's reliance on the Shuttle as its principal space launch capability created a relentless pressure on NASA to increase the flight rate … NASA must establish a flight rate that is consistent with its resources."5
disaster cost the country $12 billion, which included the cost of building the replacement Orbiter Endeavour.7 It took NASA 32 months after the Challenger accident to redesign and requalify the Solid Rocket Booster and to return the Shuttle to flight. The first post-accident flight was launched on September 29, 1988. As the Shuttle returned to flight, NASA Associate Administrator for Space Flight
August 2003
Richard Truly commented, "We will always have to treat it [the Shuttle] like an R&D test program, even many years into the future. I don't think calling it operational fooled anybody within the program … It was a signal to the public that shouldn't have been sent."8 The Shuttle Program After Return to Flight After the Rogers Commission report was issued, NASA made many of the organizational changes the Commission recommended. The space agency moved management of the Space Shuttle Program from the Johnson Space Center to NASA Headquarters in Washington, D.C. The intent of this change was to create a management structure "resembling that of the Apollo program, with the aim of preventing communication deficiencies that contributed to the Challenger accident."9 NASA also established an Office of Safety, Reliability, and Quality Assurance at its Headquarters, though that office was not given the "direct authority" over all of NASA's safety operations as the Rogers Commission had recommended. Rather, NASA human space flight centers each retained their own safety organization reporting to the Center Director. In the almost 15 years between the return to flight and the loss of Columbia, the Shuttle was again being used on a regular basis to conduct space-based research, and, in line with NASA's original 1969 vision, to build and service a space station. The Shuttle flew 87 missions during this period, compared to 24 before Challenger. Highlights from these missions include the 1990 launch, 1993 repair, and 1999 and 2002 servicing of the Hubble Space Telescope; the launch of several major planetary probes; a number of Shuttle-Spacelab missions devoted to scientific research; nine missions to rendezvous with the Russian space station Mir; the return of former Mercury astronaut Senator John Glenn to orbit in October 1998; and the launch of the first U.S. elements of the International Space Station. After the Challenger accident, the Shuttle was no longer described as "operational" in the same sense as commercial aircraft. Nevertheless, NASA continued planning as if the Shuttle could be readied for launch at or near whatever date was set. Tying the Shuttle closely to International Space Station needs, such as crew rotation, added to the urgency of maintaining a predictable launch schedule. The Shuttle is currently the only means to launch the already-built European, Japanese, and remaining U.S. modules needed to complete Station assembly and to carry and return most experiments and on-orbit supplies.10 Even after three occasions when technical problems grounded the Shuttle fleet for a month or more, NASA continued to assume that the Shuttle could regularly and predictably service the Station. In recent years, this coupling between the Station and Shuttle has become the primary driver of the Shuttle launch schedule. Whenever a Shuttle launch is delayed, it impacts Station assembly and operations. In September 2001, testimony on the Shuttle's achievements during the preceding decade by NASA's then-Deputy Associate Administrator for Space Flight William Readdy indicated the assumptions under which NASA was operating during that period:
Report Volume I
The Space Shuttle has made dramatic improvements in the capabilities, operations and safety of the system. The payload-to-orbit performance of the Space Shuttle has been significantly improved – by over 70 percent to the Space Station. The safety of the Space Shuttle has also been dramatically improved by reducing risk by more than a factor of five. In addition, the operability of the system has been significantly improved, with five minute launch windows – which would not have been attempted a decade ago – now becoming routine. This record of success is a testament to the quality and dedication of the Space Shuttle management team and workforce, both civil servants and contractors.11
Though NASA underwent many management reforms in the wake of the Challenger accident and appointed new directors at the Johnson, Marshall, and Kennedy centers, the agency's powerful human space flight culture remained intact, as did many institutional practices, even if in a modified form. As a close observer of NASA's organizational culture has observed, "Cultural norms tend to be fairly resilient … The norms bounce back into shape after being stretched or bent. Beliefs held in common throughout the organization resist alteration."12 This culture, as will become clear across the chapters of Part Two of this report, acted over time to resist externally imposed change. By the eve of the Columbia accident, institutional practices that were in effect at the time of the Challenger accident – such as inadequate concern over deviations from expected performance, a silent safety program, and schedule pressure – had returned to NASA.
Organizational culture refers to the basic values, norms, beliefs, and practices that characterize the functioning of a particular institution. At the most basic level, organizational culture defines the assumptions that employees make as they carry out their work; it defines "the way we do things here." An organization's culture is a powerful force that persists through reorganizations and the departure of key personnel.
The human space flight culture within NASA originated in the Cold War environment. The space agency itself was created in 1958 as a response to the Soviet launch of Sputnik, the first artificial Earth satellite. In 1961, President John F. Kennedy charged the new space agency with the task of reaching the moon before the end of the decade, and asked Congress and the American people to commit the immense resources for doing so, even though at the time NASA had only accumulated 15 minutes of human space flight experience. With its efforts linked to U.S.-Soviet competition for global leadership, there was a sense in the NASA workforce that the agency was engaged in a historic struggle central to the nation's agenda. The Apollo era created at NASA an exceptional "can-do" culture marked by tenacity in the face of seemingly impossible challenges. This culture valued the interaction among
August 2003
research and testing, hands-on engineering experience, and a dependence on the exceptional quality of the its workforce and leadership that provided in-house technical capability to oversee the work of contractors. The culture also accepted risk and failure as inevitable aspects of operating in space, even as it held as its highest value attention to detail in order to lower the chances of failure. The dramatic Apollo 11 lunar landing in July 1969 fixed NASA's achievements in the national consciousness, and in history. However, the numerous accolades in the wake of the moon landing also helped reinforce the NASA staff's faith in their organizational culture. Apollo successes created the powerful image of the space agency as a "perfect place," as "the best organization that human beings could create to accomplish selected goals."13 During Apollo, NASA was in many respects a highly successful organization capable of achieving seemingly impossible feats. The continuing image of NASA as a "perfect place" in the years after Apollo left NASA employees unable to recognize that NASA never had been, and still was not, perfect, nor was it as symbolically important in the continuing Cold War struggle as it had been for its first decade of existence. NASA personnel maintained a vision of their agency that was rooted in the glories of an earlier time, even as the world, and thus the context within which the space agency operated, changed around them. As a result, NASA's human space flight culture never fully adapted to the Space Shuttle Program, with its goal of routine access to space rather than further exploration beyond low-Earth orbit. The Apollo-era organizational culture came to be in tension with the more bureaucratic space agency of the 1970s, whose focus turned from designing new spacecraft at any expense to repetitively flying a reusable vehicle on an ever-tightening budget. This trend toward bureaucracy and the associated increased reliance on contracting necessitated more effective communications and more extensive safety oversight processes than had been in place during the Apollo era, but the Rogers Commission found that such features were lacking. In the aftermath of the Challenger accident, these contradictory forces prompted a resistance to externally imposed changes and an attempt to maintain the internal belief that NASA was still a "perfect place," alone in its ability to execute a program of human space flight. Within NASA centers, as Human Space Flight Program managers strove to maintain their view of the organization, they lost their ability to accept criticism, leading them to reject the recommendations of many boards and blue-ribbon panels, the Rogers Commission among them. External criticism and doubt, rather than spurring NASA to change for the better, instead reinforced the will to "impose the party line vision on the environment, not to reconsider it," according to one authority on organizational behavior. This in turn led to "flawed decision making, self deception, introversion and a diminished curiosity about the world outside the perfect place."14 The NASA human space flight culture the Board found during its investigation manifested many of these characteristics, in particular a self-confidence about NASA possessing unique knowledge about how to
Report Volume I
safely launch people into space.15 As will be discussed later in this chapter, as well as in Chapters 6, 7, and 8, the Board views this cultural resistance as a fundamental impediment to NASA's effective organizational performance.
A strong indicator of the priority the national political leadership assigns to a federally funded activity is its budget. By that criterion, NASA's space activities have not been high on the list of national priorities over the past three decades (see Figure 5.3-1). After a peak during the Apollo program, when NASA's budget was almost four percent of the federal budget, NASA's budget since the early 1970s has hovered at one percent of federal spending or less.
4.0 3.5
Percent of Federal Budget
3.0 2.5 2.0 1.5 1.0 0.5 0.0
Figure 5.3-1. NASA budget as a percentage of the Federal budget. (Source: NASA History Office)
Particularly in recent years, as the national leadership has confronted the challenging task of allocating scarce public resources across many competing demands, NASA has had difficulty obtaining a budget allocation adequate to its continuing ambitions. In 1990, the White House chartered a blue-ribbon committee chaired by aerospace executive Norman Augustine to conduct a sweeping review of NASA and its programs in response to Shuttle problems and the flawed mirror on the Hubble Space Telescope.16 The review found that NASA's budget was inadequate for all the programs the agency was executing, saying that "NASA is currently over committed in terms of program obligations relative to resources available–in short, it is trying to do too much, and allowing too little margin for the unexpected."17 "A reinvigorated space program," the Augustine committee went on to say, "will require real growth in the NASA budget of approximately 10 percent per year (through the year 2000) reaching a peak spending level of about $30 billion per year (in constant 1990 dollars) by about the year 2000." Translated into the actual dollars of Fiscal Year 2000, that recommendation would have meant a NASA budget of over $40 billion; the actual NASA budget for that year was $13.6 billion.18 During the past decade, neither the White House nor Congress has been interested in "a reinvigorated space program." Instead, the goal has been a program that would continue to
August 2003
produce valuable scientific and symbolic payoffs for the nation without a need for increased budgets. Recent budget allocations reflect this continuing policy reality. Between 1993 and 2002, the government's discretionary spending grew in purchasing power by more than 25 percent, defense spending by 15 percent, and non-defense spending by 40 percent (see Figure 5.3-2). NASA's budget, in comparison, showed little change, going from $14.31 billion in Fiscal Year 1993 to a low of $13.6 billion in Fiscal Year 2000, and increasing to $14.87 billion in Fiscal Year 2002. This represented a loss of 13 percent in purchasing power over the decade (see Figure 5.3-3).19
The lack of top-level interest in the space program led a 2002 review of the U.S. aerospace sector to observe that "a sense of lethargy has affected the space industry and community. Instead of the excitement and exuberance that dominated our early ventures into space, we at times seem almost apologetic about our continued investments in the space program."20
Warnings of a Shuttle Accident "Shuttle reliability is uncertain, but has been estimated to range between 97 and 99 percent. If the Shuttle reliability is 98 percent, there would be a 50-50 chance of losing an Orbiter within 34 flights … The probability of maintaining at least three Orbiters in the Shuttle fleet declines to less than 50 percent after flight 113."21
-The Office of Technology Assessment, 1989
Change from Base Year 1993
1.40 1.30
1.20 1.10
Total Discretionary
Figure 5.3-2. Changes in Federal spending from 1993 through 2002. (Source: NASA Office of Legislative Affairs)
"And although it is a subject that meets with reluctance to open discussion, and has therefore too often been relegated to silence, the statistical evidence indicates that we are likely to lose another Space Shuttle in the next several years … probably before the planned Space Station is completely established on orbit. This would seem to be the weak link of the civil space program – unpleasant to recognize, involving all the uncertainties of statistics, and difficult to resolve."
-The Augustine Committee, 1990
Fiscal Year 1965 1975 1985 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004
Real Dollars (in millions) 5,250 3,229 7,573 14,310 14,570 13,854 13,884 13,709 13,648 13,653 13,601 14,230 14,868 15,335 (requested) 15,255
Constant Dollars (in FY 2002 millions) 24,696 10,079 11,643 17,060 16,965 15,790 15,489 14,994 14,641 14,443 14,202 14,559 14,868 NA NA
Shuttle as Developmental Vehicle "Shuttle is also a complex system that has yet to demonstrate an ability to adhere to a fixed schedule" NASA Human Space Flight Culture "NASA has not been sufficiently responsive to valid criticism and the need for change."22
-The Augustine Committee, 1990
-The Augustine Committee, 1990
Faced with this budget situation, NASA had the choice of either eliminating major programs or achieving greater efficiencies while maintaining its existing agenda. Agency leaders chose to attempt the latter. They continued to develop the space station, continued robotic planetary and scientific missions, and continued Shuttle-based missions for both scientific and symbolic purposes. In 1994 they took on the responsibility for developing an advanced technology launch vehicle in partnership with the private sector. They tried to do this by becoming more efficient. "Faster, better, cheaper" became the NASA slogan of the 1990s.23 The flat budget at NASA particularly affected the human space flight enterprise. During the decade before the Columbia accident, NASA rebalanced the share of its budget allocated to human space flight from 48 percent of agency funding in Fiscal Year 1991 to 38 percent in Fiscal Year 1999, with the remainder going mainly to other science and technology efforts. On NASA's fixed budget, that meant
August 2003
Figure 5.3-3. NASA Budget. (Source: NASA and Office of Management and Budget)
Report Volume I
Pressure on NASA's budget has come not only from the White House, but also from the Congress. In recent years there has been an increasing tendency for the Congress to add "earmarks" – congressional additions to the NASA budget request that reflect targeted Members' interests. These earmarks come out of already-appropriated funds, reducing the amounts available for the original tasks. For example, as Congress considered NASA's Fiscal Year 2002 appropriation, the NASA Administrator told the House Appropriations subcommittee with jurisdiction over the NASA budget that the agency was "extremely concerned regarding the magnitude and number of congressional earmarks" in the House and Senate versions of the NASA appropriations bill.24 He noted "the total number of House and Senate earmarks … is approximately 140 separate items, an increase of nearly 50 percent over FY 2001." These earmarks reflected "an increasing fraction of items that circumvent the peer review process, or involve construction or other objectives that have no relation to NASA mission objectives." The potential Fiscal Year 2002 earmarks represented "a net total of $540 million in reductions to ongoing NASA programs to fund this extremely large number of earmarks."25
tion of Boris Yeltsin and halting the proliferation of nuclear weapons and the means to deliver them. Space Shuttle Program Budget Patterns For the past 30 years, the Space Shuttle Program has been NASA's single most expensive activity, and of all NASA's efforts, that program has been hardest hit by the budget constraints of the past decade. Given the high priority assigned after 1993 to completing the costly International Space Station, NASA managers have had little choice but to attempt to reduce the costs of operating the Space Shuttle. This left little funding for Shuttle improvements. The squeeze on the Shuttle budget was even more severe after the Office of Management and Budget in 1994 insisted that any cost overruns in the International Space Station budget be made up from within the budget allocation for human space flight, rather than from the agency's budget as a whole. The Shuttle was the only other large program within that budget category. Figures 5.3-4 and 5.3-5 show the trajectory of the Shuttle budget over the past decade. In Fiscal Year 1993, the outgoing Bush administration requested $4.128 billion for the Space Shuttle Program; five years later, the Clinton Administration request was for $2.977 billion, a 27 percent reduction. By Fiscal Year 2003, the budget request had increased to $3.208 billion, still a 22 percent reduction from a decade earlier. With inflation taken into account, over the past decade, there has been a reduction of approximately 40 percent in the purchasing power of the program's budget, compared to a reduction of 13 percent in the NASA budget overall.
Change –50.0 –417.4** –168.9 –53.0 0.0 –50.0 –31.0 +25.0 –40.0 –5.0 +44.8 NASA Operating Plan* 4,052.9 3,772.3 3,155.1 3,143.8 2,960.9 2,912.8 2,998.3 2,984.4 3,118.8 3,270.0 Change –25.1 –6.4 0.0 –35.0 –190.0 –15.0 –29.7 –26.8 –6.9 –8.9
the Space Shuttle and the International Space Station were competing for decreasing resources. In addition, at least $650 million of NASA's human space flight budget was used to purchase Russian hardware and services related to U.S.-Russian space cooperation. This initiative was largely driven by the Clinton Administration's foreign policy and national security objectives of supporting the administraFiscal Year 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 President's Request to Congress 4,128.0 4,196.1 3,324.0 3,231.8 3,150.9 2,977.8 3,059.0 2,986.2 3,165.7 3,283.8 3,208.0 Congressional Appropriation 4,078.0 3,778.7 3,155.1 3,178.8 3,150.9 2,927.8 3,028.0 3,011.2 3,125.7 3,278.8 3,252.8
Figure 5.3-4. Space Shuttle Program Budget (in millions of dollars). (Source: NASA Office of Space Flight) * NASA's operating plan is the means for adjusting congressional appropriations among various activities during the fiscal year as changing circumstances dictate. These changes must be approved by NASA's appropriation subcommittees before they can be put into effect. **This reduction primarily reflects the congressional cancellation of the Advanced Solid Rocket Motor Program
Report Volume I
August 2003
6000 5500
40% Purchasing Power
45% Purchasing Power
In most years, Congress appropriates slightly less for the Space Shuttle Program than the President requested; in some cases, these reductions have been requested by NASA during the final stages of budget deliberations. After its budget was passed by Congress, NASA further reduced the Shuttle budget in the agency's operating plan–the plan by which NASA actually allocates its appropriated budget during the fiscal year to react to changing program needs. These released funds were allocated to other activities, both within the human space flight program and in other parts of the agency. Changes in recent years include: Fiscal Year 1997 • NASA transferred $190 million to International Space Station (ISS). Fiscal Year 1998 • At NASA's request, Congress transferred $50 million to ISS. • NASA transferred $15 million to ISS. Fiscal Year 1999 • At NASA's request, Congress reduced Shuttle $31 million so NASA could fund other requirements. • NASA reduced Shuttle $32 million by deferring two flights; funds transferred to ISS. • NASA added $2.3 million from ISS to previous NASA request. Fiscal Year 2000 • Congress added $25 million to Shuttle budget for upgrades and transferred $25 million from operations to upgrades. • NASA reduced Shuttle $11.5 million per governmentwide rescission requirement and transferred $15.3 million to ISS. Fiscal Year 2001 • At NASA's request, Congress reduced Shuttle budget by $40 million to fund Mars initiative. • NASA reduced Shuttle $6.9 million per rescission requirement. Fiscal Year 2002 • Congress reduced Shuttle budget $50 million to reflect cancellation of electric Auxiliary Power Unit and added $20 million for Shuttle upgrades and $25 million for Vehicle Assembly Building repairs. • NASA transferred $7.6 million to fund Headquarters requirements and cut $1.2 million per rescission requirement. [Source: Marcia Smith, Congressional Research Service, Presentation at CAIB Public Hearing, June 12, 2003]
Constant FY 2002 Dollars in Millions
5000 4500 4000 3500 3000 2500 2000 1500
Flight Rate
"Freeze Design" Policy (Kraft Report) Space Flight Operations Contract
Initial Funding for High Priority Safety Upgrades
Initiated Space Shuttle Upgrades Prgrm
Operating Plan Actuals
7 8 6 8 8 4
1st Flight to ISS
FY 2004 President's Budget
7 4 6 5 5 5 5 5
Figure 5.3-5. NASA budget as a percentage of the Federal budget from 1991 to 2008. (Source: NASA Office of Space Flight)
This budget squeeze also came at a time when the Space Shuttle Program exhibited a trait common to most aging systems: increased costs due to greater maintenance requirements, a declining second- and third-tier contractor support base, and deteriorating infrastructure. Maintaining the Shuttle was becoming more expensive at a time when Shuttle budgets were decreasing or being held constant. Only in the last few years have those budgets begun a gradual increase. As Figure 5.3-5 indicates, most of the steep reductions in the Shuttle budget date back to the first half of the 1990s. In the second half of the decade, the White House Office of Management and Budget and NASA Headquarters held the Shuttle budget relatively level by deferring substantial funding for Shuttle upgrades and infrastructure improvements, while keeping pressure on NASA to limit increases in operating costs.
In 1992 the White House replaced NASA Administrator Richard Truly with aerospace executive Daniel S. Goldin, a self-proclaimed "agent of change" who held office from April 1, 1992, to November 17, 2001 (in the process becoming the longest-serving NASA Administrator). Seeing "space exploration (manned and unmanned) as NASA's principal purpose with Mars as a destiny," as one management scholar observed, and favoring "administrative transformation" of NASA, Goldin engineered "not one or two policy changes, but a torrent of changes. This was not evolutionary change, but radical or discontinuous change."26 His tenure at NASA was one of continuous turmoil, to which the Space Shuttle Program was not immune. Of course, turbulence does not necessarily degrade organizational performance. In some cases, it accompanies productive change, and that is what Goldin hoped to achieve. He believed in the management approach advocated by W. Edwards Deming, who had developed a series of widely acclaimed management principles based on his work in Japan during the "economic miracle" of the 1980s. Goldin attempted to apply some of those principles to NASA, including the notion that a corporate headquarters should
Report Volume I
not attempt to exert bureaucratic control over a complex organization, but rather set strategic directions and provide operating units with the authority and resources needed to pursue those directions. Another Deming principle was that checks and balances in an organization were unnecessary
August 2003
and sometimes counterproductive, and those carrying out the work should bear primary responsibility for its quality. It is arguable whether these business principles can readily be applied to a government agency operating under civil service rules and in a politicized environment. Nevertheless, Goldin sought to implement them throughout his tenure.27 Goldin made many positive changes in his decade at NASA. By bringing Russia into the Space Station partnership in 1993, Goldin developed a new post-Cold War rationale for the agency while managing to save a program that was politically faltering. The International Space Station became NASA's premier program, with the Shuttle serving in a supporting role. Goldin was also instrumental in gaining acceptance of the "faster, better, cheaper"28 approach to the planning of robotic missions and downsizing "an agency that was considered bloated and bureaucratic when he took it over."29 Goldin described himself as "sharp-edged" and could often be blunt. He rejected the criticism that he was sacrificing safety in the name of efficiency. In 1994 he told an audience at the Jet Propulsion Laboratory, "When I ask for the budget to be cut, I'm told it's going to impact safety on the Space Shuttle … I think that's a bunch of crap."30 One of Goldin's high-priority objectives was to decrease involvement of the NASA engineering workforce with the
1993 Total Workforce Total Civil Service Workforce JSC KSC MSFC Stennis/Dryden Headquarters Total Contractor Workforce JSC KSC MSFC Stennis/Dryden Headquarters 30,091 3,781 1,330 1,373 874 84 120 26,310 7,487 9,173 9,298 267 85 1994 27,538 3,324 1,304 1,104 791 64 61 24,214 6,805 8,177 8,635 523 74 1995 25,346 2,959 1,248 1,018 576 55 62 22,387 5,887 7,691 8,210 529 70 1996 23,625 2,596 1,076 932 523 32 32 21,029 5,442 7,208 7,837 505 37
Space Shuttle Program and thereby free up those skills for finishing the space station and beginning work on his preferred objective–human exploration of Mars. Such a shift would return NASA to its exploratory mission. He was often at odds with those who continued to focus on the centrality of the Shuttle to NASA's future. Initial Shuttle Workforce Reductions With NASA leadership choosing to maintain existing programs within a no-growth budget, Goldin's "faster, better, cheaper" motto became the agency's slogan of the 1990s.31 NASA leaders, however, had little maneuvering room in which to achieve efficiency gains. Attempts by NASA Headquarters to shift functions or to close one of the three human space flight centers were met with strong resistance from the Centers themselves, the aerospace firms they used as contractors, and the congressional delegations of the states in which the Centers were located. This alliance resembles the classic "iron triangle" of bureaucratic politics, a conservative coalition of bureaucrats, interest groups, and congressional subcommittees working together to promote their common interests.32 With Center infrastructure off-limits, this left the Space Shuttle budget as an obvious target for cuts. Because the Shuttle required a large "standing army" of workers to
1997 19,476 2,195 958 788 401 29 20 17,281 *10,556 539 5,650 536 0 1998 18,654 1,954 841 691 379 27 16 16,700 10,525 511 5,312 453 0 1999 18,068 1,777 800 613 328 26 10 16,291 10,733 430 4,799 329 0 2000 17,851 1,786 798 626 336 16 10 16,065 10,854 436 4,444 331 0 2001 18,012 1,759 794 614 327 14 10 16,253 11,414 439 4,197 203 0 2002 17,462 1,718 738 615 337 16 12 15,744 11,445 408 3,695 196 0
Figure 5.4-1. Space Shuttle Program workforce. [Source: NASA Office of Space Flight] * Because Johnson Space Center manages the Space Flight Operations Contract, all United Space Alliance employees are counted as working for Johnson.
Report Volume I
August 2003
keep it flying, reducing the size of the Shuttle workforce became the primary means by which top leaders lowered the Shuttle's operating costs. These personnel reduction efforts started early in the decade and continued through most of the 1990s. They created substantial uncertainty and tension within the Shuttle workforce, as well as the transitional difficulties inherent in any large-scale workforce reassignment. In early 1991, even before Goldin assumed office and less than three years after the Shuttle had returned to flight after the Challenger accident, NASA announced a goal of saving three to five percent per year in the Shuttle budget over five years. This move was in reaction to a perception that the agency had overreacted to the Rogers Commission recommendations – for example, the notion that the many layers of safety inspections involved in preparing a Shuttle for flight had created a bloated and costly safety program. From 1991 to 1994, NASA was able to cut Shuttle operating costs by 21 percent. Contractor personnel working on the Shuttle declined from 28,394 to 22,387 in these three years, and NASA Shuttle staff decreased from 4,031 to 2,959.33 Figure 5.4-1 shows the changes in Space Shuttle workforce over the past decade. A 1994 National Academy of Public Administration review found that these cuts were achieved primarily through "operational and organizational efficiencies and consolidations, with resultant reductions in staffing levels and other actions which do not significantly impact basic program content or capabilities."34 NASA considered additional staff cuts in late 1994 and early 1995 as a way of further reducing the Space Shuttle Program budget. In early 1995, as the national leadership focused its attention on balancing the federal budget, the projected five-year Shuttle budget requirements exceeded by $2.5 billion the budget that was likely to be approved by the White House Office of Management and Budget.35 Despite its already significant progress in reducing costs, NASA had to make further workforce cuts. Anticipating this impending need, a 1994-1995 NASA "Functional Workforce Review" concluded that removing an additional 5,900 people from the NASA and contractor Shuttle workforce – just under 13 percent of the total – could be done without compromising safety.36 These personnel cuts were made in Fiscal Years 1996 and 1997. By the end of 1997, the NASA Shuttle civilian workforce numbered 2,195, and the contractor workforce 17,281. Shifting Shuttle Management Arrangements Workforce reductions were not the only modifications to the Shuttle Program in the middle of the decade. In keeping with Goldin's philosophy that Headquarters should concern itself primarily with strategic issues, in February 1996 Johnson Space Center was designated as "lead center" for the Space Shuttle Program, a role it held prior to the Challenger accident. This shift was part of a general move of all program management responsibilities from NASA Headquarters to the agency's field centers. Among other things, this change meant that Johnson Space Center managers would have authority over the funding and management of Shuttle activiReport Volume I
ties at the Marshall and Kennedy Centers. Johnson and Marshall had been rivals since the days of Apollo, and long-term Marshall employees and managers did not easily accept the return of Johnson to this lead role. The shift of Space Shuttle Program management to Johnson was worrisome to some. The head of the Space Shuttle Program at NASA Headquarters, Bryan O'Connor, argued that transfer of the management function to the Johnson Space Center would return the Shuttle Program management to the flawed structure that was in place before the Challenger accident. "It is a safety issue," he said, "we ran it that way [with program management at Headquarters, as recommended by the Rogers Commission] for 10 years without a mishap and I didn't see any reason why we should go back to the way we operated in the pre-Challenger days."37 Goldin gave O'Connor several opportunities to present his arguments against a transfer of management responsibility, but ultimately decided to proceed. O'Connor felt he had no choice but to resign.38 (O'Connor returned to NASA in 2002 as Associate Administrator for Safety and Mission Assurance.) In January 1996, Goldin appointed as Johnson's director his close advisor, George W.S. Abbey. Abbey, a space program veteran, was a firm believer in the values of the original human space flight culture, and as he assumed the directorship, he set about recreating as many of the positive features of that culture as possible. For example, he and Goldin initiated, as a way for young engineers to get hands-on experience, an in-house X-38 development program as a prototype for a space station crew rescue vehicle. Abbey was a powerful leader, who through the rest of the decade exerted substantial control over all aspects of Johnson Space Center operations, including the Space Shuttle Program. Space Flight Operations Contract By the middle of the decade, spurred on by Vice President Al Gore's "reinventing government" initiative, the goal of balancing the federal budget, and the views of a Republican-led House of Representatives, managers throughout the government sought new ways of making public sector programs more efficient and less costly. One method considered was transferring significant government operations and responsibilities to the private sector, or "privatization." NASA led the way toward privatization, serving as an example to other government agencies. In keeping with his philosophy that NASA should focus on its research-and-development role, Goldin wanted to remove NASA employees from the repetitive operations of various systems, including the Space Shuttle. Giving primary responsibility for Space Shuttle operations to the private sector was therefore consistent with White House and congressional priorities and attractive to Goldin on its own terms. Beginning in 1994, NASA considered the feasibility of consolidating many of the numerous Shuttle operations contracts under a single prime contractor. At that time, the Space Shuttle Program was managing 86 separate contracts held by 56 different firms. Top NASA managers thought that consolidating these contracts could reduce the amount of redundant overhead, both for NASA and for the contractors
August 2003
themselves. They also wanted to explore whether there were functions being carried out by NASA that could be more effectively and inexpensively carried out by the private sector. An advisory committee headed by early space flight veteran Christopher Kraft recommended such a step in its March 1995 report, which became known as the "Kraft Report."39 (The report characterized the Space Shuttle in a way that the Board judges to be at odds with the realities of the Shuttle Program). The report made the following findings and recommendations: • "The Shuttle has become a mature and reliable system … about as safe as today's technology will provide." • "Given the maturity of the vehicle, a change to a new mode of management with considerably less NASA oversight is possible at this time." • "Many inefficiencies and difficulties in the current Shuttle Program can be attributed to the diffuse and fragmented NASA and contractor structure. Numerous contractors exist supporting various program elements, resulting in ambiguous lines of communication and diffused responsibilities." • NASA should "consolidate operations under a singlebusiness entity." • "The program remains in a quasi-development mode and yearly costs remain higher than required," and NASA should "freeze the current vehicle configuration, minimizing future modifications, with such modifications delivered in block updates. Future block updates should implement modifications required to make the vehicle more re-usable and operational." • NASA should "restructure and reduce the overall Safety, Reliability, and Quality Assurance elements – without reducing safety."40 When he released his committee's report, Kraft said that "if NASA wants to make more substantive gains in terms of efficiency, cost savings and better service to its customers, we think it's imperative they act on these recommendations … And we believe that these savings are real, achievable, and can be accomplished with no impact to the safe and successful operation of the Shuttle system."41 Although the Kraft Report stressed that the dramatic changes it recommended could be made without compromising safety, there was considerable dissent about this claim. NASA's Aerospace Safety Advisory Panel – independent, but often not very influential – was particularly critical. In May 1995, the Panel noted that "the assumption [in the Kraft Report] that the Space Shuttle systems are now ʻmature' smacks of a complacency which may lead to serious mishaps. The fact is that the Space Shuttle may never be mature enough to totally freeze the design." The Panel also noted that "the report dismisses the concerns of many credible sources by labeling honest reservations and the people who have made them as being partners in an unneeded ʻsafety shield' conspiracy. Since only one more accident would kill the program and destroy far more than the spacecraft, it is extremely callous" to make such an accusation.42
Report Volume I
The notion that NASA would further reduce the number of civil servants working on the Shuttle Program prompted senior Kennedy Space Center engineer José Garcia to send to President Bill Clinton on August 25, 1995, a letter that stated, "The biggest threat to the safety of the crew since the Challenger disaster is presently underway at NASA." Garcia's particular concern was NASA's "efforts to delete the ʻchecks and balances' system of processing Shuttles as a way of saving money … Historically NASA has employed two engineering teams at KSC, one contractor and one government, to cross check each other and prevent catastrophic errors … although this technique is expensive, it is effective, and it is the single most important factor that sets the Shuttle's success above that of any other launch vehicle … Anyone who doesn't have a hidden agenda or fear of losing his job would admit that you can't delete NASA's checks and balances system of Shuttle processing without affecting the safety of the Shuttle and crew."43 NASA leaders accepted the advice of the Kraft Report and in August 1995 solicited industry bids for the assignment of Shuttle prime contractor. In response, Lockheed Martin and Rockwell, the two major Space Shuttle operations contractors, formed a limited liability corporation, with each firm a 50 percent owner, to compete for what was called the Space Flight Operations Contract. The new corporation would be known as United Space Alliance. In November 1995, NASA awarded the operations contract to United Space Alliance on a sole source basis. (When Boeing bought Rockwell's aerospace group in December 1996, it also took over Rockwell's 50 percent ownership of United Space Alliance.) The company was responsible for 61 percent of the Shuttle operations contracts. Some in Congress were skeptical that safety could be maintained under the new arrangement, which transferred significant NASA responsibilities to the private sector. Despite these concerns, Congress ultimately accepted the reasoning behind the contract.44 NASA then spent much of 1996 negotiating the contract's terms and conditions with United Space Alliance. The Space Flight Operations Contract was designed to reward United Space Alliance for performance successes and penalize its performance failures. Before being eligible for any performance fees, United Space Alliance would have to meet a series of safety "gates," which were intended to ensure that safety remained the top priority in Shuttle operations. The contract also rewarded any cost reductions that United Space Alliance was able to achieve, with NASA taking 65 percent of any savings and United Space Alliance 35 percent.45 NASA and United Space Alliance formally signed the Space Flight Operations Contract on October 1, 1996. Initially, only the major Lockheed Martin and Rockwell Shuttle contracts and a smaller Allied Signal Unisys contract were transferred to United Space Alliance. The initial contractual period was six years, from October 1996 to September 2002. NASA exercised an option for a two-year extension in 2002, and another two-year option exists. The total value of the contract through the current extension is estimated at $12.8 billion. United Space Alliance currently has approximately 10,000 employees.
August 2003
The Space Flight Operations Contract has two major areas of innovation: • It replaced the previous "cost-plus" contracts (in which a firm was paid for the costs of its activity plus a negotiated profit) with a complex contract structure that included performance-based and cost reduction incentives. Performance measures include safety, launch readiness, on-time launch, Solid Rocket Booster recovery, proper orbital insertion, and successful landing. • It gave additional responsibilities for Shuttle operation, including safety and other inspections and integration of the various elements of the Shuttle system, to United Space Alliance. Many of those responsibilities were previously within the purview of NASA employees. Under the Space Flight Operations Contract, United Space Alliance had overall responsibility for processing selected Shuttle hardware, including: • Inspecting and modifying the Orbiters • Installing the Space Shuttle Main Engines on the Orbiters • Assembling the sections that make up the Solid Rocket Boosters • Attaching the External Tank to the Solid Rocket Boosters, and then the Orbiter to the External Tank • Recovering expended Solid Rocket boosters In addition to processing Shuttle hardware, United Space Alliance is responsible for mission design and planning, astronaut and flight controller training, design and integration of flight software, payload integration, flight operations, launch and recovery operations, vehicle-sustaining engineering, flight crew equipment processing, and operation and maintenance of Shuttle-specific facilities such as the Vehicle Assembly Building, the Orbiter Processing Facility, and the launch pads. United Space Alliance also provides spare parts for the Orbiters, maintains Shuttle flight simulators, and provides tools and supplies, including consumables such as food, for Shuttle missions. Under the Space Flight Operations Contract, NASA has the following responsibilities and roles: • Maintaining ownership of the Shuttles and all other assets of the Shuttle program • Providing to United Space Alliance the Space Shuttle Main Engines, the External Tanks, and the Redesigned Solid Rocket Motor segments for assembly into the Solid Rocket Boosters • Managing the overall process of ensuring Shuttle safety • Developing requirements for major upgrades to all assets • Participating in the planning of Shuttle missions, the directing of launches, and the execution of flights • Performing surveillance and audits and obtaining technical insight into contractor activities • Deciding if and when to "commit to flight" for each mission46
The contract provided for additional consolidation and then privatization, when all remaining Shuttle operations would be transferred from NASA. Phase 2, scheduled for 19982000, called for the transfer of Johnson Space Center-managed flight software and flight crew equipment contracts and the Marshall Space Center-managed contracts for the External Tank, Space Shuttle Main Engine, Reusable Solid Rocket Motor, and Solid Rocket Booster. However, Marshall and its contractors, with the concurrence of the Space Shuttle Program Office at Johnson Space Center, successfully resisted the transfer of its contracts. Therefore, the Space Flight Operations Contract's initial efficiency and integrated management goals have not been achieved. The major annual savings resulting from the Space Flight Operations Contract, which in 1996 were touted to be some $500 million to $1 billion per year by the early 2000s, have not materialized. These projections assumed that by 2002, NASA would have put all Shuttle contracts under the auspices of United Space Alliance, and would be moving toward Shuttle privatization. Although the Space Flight Operations Contract has not been as successful in achieving cost efficiencies as its proponents hoped, it has reduced some Shuttle operating costs and other expenses. By one estimate, in its first six years the contract has saved NASA a total of more than $1 billion.47 Privatizing the Space Shuttle To its proponents, the Space Flight Operations Contract was only a beginning. In October 1997, United Space Alliance submitted to the Space Shuttle Program Office a contractually required plan for privatizing the Shuttle, which the program did not accept. But the notion of Shuttle privatization lingered at NASA Headquarters and in Congress, where some members advocated a greater private sector role in the space program. Congress passed the Commercial Space Act of 1998, which directed the NASA Administrator to "plan for the eventual privatization of the Space Shuttle Program."48 By August 2001, NASA Headquarters prepared for White House consideration a "Privatization White Paper" that called for transferring all Shuttle hardware, pilot and commander astronauts, and launch and operations teams to a private operator.49 In September 2001, Space Shuttle Program Manager Ron Dittemore released his report on a "Concept of Privatization of the Space Shuttle Program,"50 which argued that for the Space Shuttle "to remain safe and viable, it is necessary to merge the required NASA and contractor skill bases" into a single private organization that would manage human space flight. This perspective reflected Dittemore's belief that the split of responsibilities between NASA and United Space Alliance was not optimal, and that it was unlikely that NASA would ever recapture the Shuttle responsibilities that were transferred in the Space Flight Operations Contract. Dittemore's plan recommended transferring 700 to 900 NASA employees to the private organization, including: • Astronauts, including the flight crew members who operate the Shuttle
August 2003
Report Volume I
Full Time Persons Employment
• Program and project management, including Space Shuttle Main Engine, External Tank, Redesigned Solid Rocket Booster, and Extravehicular Activity • Mission operations, including flight directors and flight controllers • Ground operations and processing, including launch director, process engineering, and flow management • Responsibility for safety and mission assurance After such a shift occurred, according to the Dittemore plan, "the primary role for NASA in Space Shuttle operations … will be to provide an SMA [Safety and Mission Assurance] independent assessment … utilizing audit and surveillance techniques."51 With a change in NASA Administrators at the end of 2001 and the new Bush Administration's emphasis on "competitive sourcing" of government operations, the notion of wholesale privatization of the Space Shuttle was replaced with an examination of the feasibility of both public- and private-sector Program management. This competitive sourcing was under examination at the time of the Columbia accident. Workforce Transformation and the End of Downsizing Workforce reductions instituted by Administrator Goldin as he attempted to redefine the agency's mission and its overall organization also added to the turbulence of his reign. In the 1990s, the overall NASA workforce was reduced by 25 percent through normal attrition, early retirements, and buyouts – cash bonuses for leaving NASA employment. NASA operated under a hiring freeze for most of the decade, making it difficult to bring in new or younger people. Figure 5.4-2 shows the downsizing of the overall NASA workforce during this period as well as the associated shrinkage in NASA's technical workforce. NASA Headquarters was particularly affected by workforce reductions. More than half its employees left or were transferred in parallel with the 1996 transfer of program management responsibilities back to the NASA centers. The Space Shuttle Program bore more than its share of Headquarters personnel cuts. Headquarters civil service staff working on the Space Shuttle Program went from 120 in 1993 to 12 in 2003. While the overall workforce at the NASA Centers involved in human space flight was not as radically reduced, the combination of the general workforce reduction and the introduction of the Space Flight Operations Contract significantly impacted the Centers' Space Shuttle Program civil service staff. Johnson Space Center went from 1,330 in 1993 to 738 in 2002; Marshall Space Flight Center, from 874 to 337; and Kennedy Space Center from 1,373 to 615. Kennedy Director Roy Bridges argued that personnel cuts were too deep, and threatened to resign unless the downsizing of his civil service workforce, particularly those involved with safety issues, was reversed.52 By the end of the decade, NASA realized that staff reductions had gone too far. By early 2000, internal and external
Report Volume I
26,000 24,000 22,000 20,000 18,000 16,000
14,000 13,000 12,000 11,000 10,000 9,000 1993
Total Workforce
Technical Workforce
Figure 5.4-2. Downsizing of the overall NASA workforce and the NASA technical workforce.
studies convinced NASA leaders that the workforce needed to be revitalized. These studies noted that "five years of buyouts and downsizing have led to serious skill imbalances and an overtaxed core workforce. As more employees have departed, the workload and stress [on those] remaining have increased, with a corresponding increase in the potential for impacts to operational capacity and safety." 53 NASA announced that NASA workforce downsizing would stop short of the 17,500 target, and that its human space flight centers would immediately hire several hundred workers.
In addition to budget pressures, workforce reductions, management changes, and the transfer of government functions to the private sector, the Space Shuttle Program was beset during the past decade by uncertainty about when the Shuttle might be replaced. National policy has vacillated between treating the Shuttle as a "going out of business" program and anticipating two or more decades of Shuttle use. As a result, limited and inconsistent investments have been made in Shuttle upgrades and in revitalizing the infrastructure to support the continued use of the Shuttle. Even before the 1986 Challenger accident, when and how to replace the Space Shuttle with a second generation reusable launch vehicle was a topic of discussion among space policy leaders. In January 1986, the congressionally chartered National Commission on Space expressed the need for a Shuttle replacement, suggesting that "the Shuttle fleet will become obsolescent by the turn of the century."54 Shortly after the Challenger accident (but not as a reaction to it), President Reagan announced his approval of "the new Orient Express" (see Figure 5.5-1). This reusable launch vehicle, later known as the National Aerospace Plane, "could, by the end of the decade, take off from Dulles Airport, accelerate up to 25 times the speed of sound attaining low-Earth orbit, or fly to Tokyo within two hours."55 This goal proved too ambitious, particularly without substantial
August 2003
funding. In 1992, after a $1.7 billion government investment, the National Aerospace Plane project was cancelled. This pattern – optimistic pronouncements about a revolutionary Shuttle replacement followed by insufficient government investment, and then program cancellation due to technical difficulties – was repeated again in the 1990s.
Reflecting its leadership's preference for bold initiatives, NASA chose the third alternative. With White House support,57 NASA began the X-33 project in 1996 as a joint effort with Lockheed Martin. NASA also initiated the less ambitious X-34 project with Orbital Sciences Corporation. At the time, the future of commercial space launches was bright, and political sentiment in the White House and Congress encouraged an increasing reliance on private-sector solutions for limiting government expenditures. In this context, these unprecedented joint projects appeared less risky than they actually were. The hope was that NASA could replace the Shuttle through private investments, without significant government spending. Both the X-33 and X-34 incorporated new technologies. The X-33 was to demonstrate the feasibility of an aerospike engine, new Thermal Protection Systems, and composite rather than metal propellant tanks. These radically new technologies were in turn to become the basis for a new orbital vehicle called VentureStar™ that could replace the Space Shuttle by 2006 (see Figure 5.5-2). The X-33 and X-34 ran into technical problems and never flew. In 2001, after spending $1.3 billion, NASA abandoned both projects. In all three projects – National Aerospace Plane, X-33, and X-34 – national leaders had set ambitious goals in response to NASA's ambitious proposals. These programs relied on the invention of revolutionary technology, had run into major technical problems, and had been denied the funds needed to overcome these problems – assuming they could be solved. NASA had spent nearly 15 years and several billion dollars, and yet had made no meaningful progress toward a Space Shuttle replacement. In 2000, as the agency ran into increasing problems with the X-33, NASA initiated the Space Launch Initiative, a $4.5 billion multi-year effort to develop new space launch technologies. By 2002, after spending nearly $800 million, NASA again changed course. The Space Launch Initiative failed to find technologies that could revolutionize space launch, forcing NASA to shift its focus to an Orbital Space Plane, developed with existing technology, that would complement the Shuttle by carrying crew, but not cargo, to and from orbit. Under a new Integrated Space Transportation Plan, the Shuttle might continue to fly until 2020 or beyond. (See Section 5.6 for a discussion of this plan.) As a result of the haphazard policy process that created these still-born developmental programs, the uncertainty over Shuttle replacement persisted. Between 1986 and 2002, the planned replacement date for the Space Shuttle was consistent only in its inconsistency: it changed from 2002 to 2006 to 2012, and before the Columbia accident, to 2020 or later. Safety Concerns and Upgrading the Space Shuttle This shifting date for Shuttle replacement has severely complicated decisions on how to invest in Shuttle Program upgrades. More often than not, investments in upgrades were delayed or deferred on the assumption they would be a waste of money if the Shuttle were to be retired in the near future (see Figure 5.5-3).
August 2003
Figure 5.5-1. A 1986 artist's conception of the National Aerospace Plane on a mission to the Space Station.
In 1994, NASA listed alternatives for access to space through 2030. • Upgrade the Space Shuttle to enable flights through 2030 • Develop a new expendable launcher • Replace the Space Shuttle with a "leapfrog" next-generation advanced technology system that would achieve order-of-magnitude improvements in the cost effectiveness of space transportation.56
Figure 5.5-2. The VentureStar was intended to replace the Space Shuttle based on technology developed for the X-33.
Report Volume I
During the course of the investigation, more than 50 past reports regarding NASA and the Space Shuttle Program were reviewed. The principal purpose of these reviews was to note what factors that reports examined, what findings were made, and what response, if any, NASA may have made to the findings. Board members then used these findings and responses as a benchmark during their investigation to compare to NASA's current programs. In addition to an extensive 300-page examination of every Aerospace Safety Advisory Panel report (see Appendix D.18), the reports listed on the accompanying chart were examined for specific factors related to the investigation. A complete listing of those past reports' findings, plus the full text of the reports, is contained in Appendix D.18.
Report Reviewed Infrastructure Communications Contracts
Topic Examined Risk Management Maintenance Workforce Issues • • • • • • • • • • • • • • • • • • • • • • • • • • Quality Assurance Safety Programs Security
Rogers Commission Report – 1986 STS-29R Prelaunch Assessment – 1989 "Augustine Report" – 1990 Paté-Cornell Report – 1990 "Aldridge Report" – 1992 GAO: NASA Infrastructure – 1996 GAO: NASA Workforce Reductions – 1996 Super Light Weight Tank Independent Assessment – 1997 Process Readiness Review – 1998 S&MA Ground Operations Report – 1998 GAO: NASA Management Challenges – 1999 Independent Assessment JS-9047 – 1999 Independent Assessment JS-9059 – 1999 Independent Assessment JS-9078 – 1999 Independent Assessment JS-9083 – 1999 S&MA Ground Operations Report – 1999 Space Shuttle Independent Assessment Team – 1999 Space Shuttle Ground Operations Report – 1999 Space Shuttle Program (SSP) Annual Report – 1999
• •
• • • •
• •
Report Volume I
August 2003
Risk Management
GAO: Human Capital & Safety – 2000 Independent Assessment JS-0032 – 2000 Independent Assessment JS-0034 – 2000 Independent Assessment JS-0045 – 2000 IG Audit Report 00-039 – 2000 NASA Independent Assessment Team – 2000 Space Shuttle Program Annual Report – 2000 ASAP Report – 2001 GAO: NASA Critical Areas – 2001 GAO: Space Shuttle Safety – 2001 Independent Assessment JS-1014 – 2001 Independent Assessment JS-1024 – 2001 Independent Assessment KS-0003 – 2001 Independent Assessment KS-1001 – 2001 Workforce Survey-KSC – 2001 Space Shuttle Program Annual Report – 2001 SSP Processing Independent Assessment – 2001 ASAP Report – 2002 GAO: Lessons Learned Process – 2002 Independent Assessment KS-1002 – 2002 Selected NASA Lessons Learned – 1992-2002 NASA/Navy Benchmarking Exchange – 2002 Space Shuttle Program Annual Report – 2002 ASAP Leading Indicators -- 2003 NASA Quality Management System – 2003 QAS Tiger Team Report – 2003 Shuttle Business Environment – 2003 • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •
• • • • •
• • • •
• •
• • • •
Report Volume I
August 2003
Workforce Issues • • • • • • • •
Quality Assurance
Safety Programs
Fiscal Year 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003
Upgrades $454.5 $247.2 $224.5 $215.9 $206.7 $175.2 $239.1 $289.3 $379.5 $347.5
• "The size and complexity of the Shuttle system and of NASA/contractor relationships place extreme importance on understanding, communication, and information handling … Communication of problems and concerns upward to the SSP from the ʻfloor' also appeared to leave room for improvement."62 The Shuttle Independent Assessment Team report also stated that the Shuttle "clearly cannot be thought of as ʻoperational' in the usual sense. Extensive maintenance, major amounts of ʻtouch labor' and a high degree of skill and expertise will always be required." However, "the workforce has received a conflicting message due to the emphasis on achieving cost and staff reductions, and the pressures placed on increasing scheduled flights as a result of the Space Station."63 Responding to NASA's concern that the Shuttle required safety-related upgrades, the President's proposed NASA budget for Fiscal Year 2001 proposed a "safety upgrades initiative." That initiative had a short life span. In its Fiscal Year 2002 budget request, NASA proposed to spend $1.836 billion on Shuttle upgrades over five years. A year later, the Fiscal Year 2003 request contained a plan to spend $1.220 billion – a 34 percent reduction. The reductions were primarily a response to rising Shuttle operating costs and the need to stay within a fixed Shuttle budget. Cost growth in Shuttle operations forced NASA to "use funds intended for Space Shuttle safety upgrades to address operational, supportability, obsolescence, and infrastructure needs." 64 At its March 2001 meeting, NASA's Space Flight Advisory Committee advised that "the Space Shuttle Program must make larger, more substantial safety upgrades than currently planned … a budget on the order of three times the budget currently allotted for improving the Shuttle systems" was needed.65 Later that year, five Senators complained that "the Shuttle program is being penalized, despite its outstanding performance, in order to conform to a budget strategy that is dangerously inadequate to ensure safety in America's human space flight program."66 (See Chapter 7 for additional discussion of Shuttle safety upgrades.) Deteriorating Shuttle Infrastructure The same ambiguity about investing in Shuttle upgrades has also affected the maintenance of Shuttle Program ground infrastructure, much of which dates to Project Apollo and 1970s Shuttle Program construction. Figure 5.5-4 depicts the age of the Shuttle's infrastructure as of 2000. Most ground infrastructure was not built for such a protracted lifespan. Maintaining infrastructure has been particularly difficult at Kennedy Space Center, where it is constantly exposed to a salt water environment. Board investigators have identified deteriorating infrastructure associated with the launch pads, Vehicle Assembly Building, and the crawler transporter. Figures 5.5-5 and 5.5-6 depict some of this deterioration. For example, NASA has installed nets, and even an entire sub-roof, inside the Vehicle Assembly Building to prevent concrete from the building's ceiling from hitting the Orbiter and Shuttle stack. In addition, the corrosion-control challenge results in zinc primer
August 2003
Figure 5.5-3. Shuttle Upgrade Budgets (in millions of dollars). (Source: NASA)
In 1995, for instance, the Kraft Report embraced the principle that NASA should "freeze the design" of the Shuttle and defer upgrades due to the vehicle's "mature" status and the need for NASA to "concentrate scarce resources on developing potential replacements for the Shuttle."58 NASA subsequently halted a number of planned upgrades, only to reverse course a year later to "take advantage of technologies to improve Shuttle safety and the need for a robust Space Shuttle to assemble the ISS."59 In a June 1999 letter to the White House, NASA Administrator Daniel Goldin declared that the nation faced a "Space Launch Crisis." He reported on a NASA review of Shuttle safety that indicated the budget for Shuttle upgrades in Fiscal year 2000 was "inadequate to accommodate upgrades necessary to yield significant safety improvements."60 After two "close calls" during STS-93 in July 1999 Goldin also chartered a Shuttle Independent Assessment Team (SIAT) chaired by Harry McDonald, Director of NASA Ames Research Center. Among the team's findings, reported in March 2000:61 • "Over the course of the Shuttle Program … processes, procedures and training have continuously been improved and implemented to make the system safer. The SIAT has a major concern … that this critical feature of the Shuttle Program is being eroded." The major factor leading to this concern "is the reduction in allocated resources and appropriate staff … There are important technical areas that are ʻone-deep.' " Also, "the SIAT feels strongly that workforce augmentation must be realized principally with NASA personnel rather than with contractor personnel." • The SIAT was concerned with "success-engendered safety optimism … The SSP must rigorously guard against the tendency to accept risk solely because of prior success." • "The SIAT was very concerned with what it perceived as Risk Management process erosion created by the desire to reduce costs … The SIAT feels strongly that NASA Safety and Mission Assurance should be restored to its previous role of an independent oversight body, and not be simply a ʻsafety auditor.' "
Report Volume I
Daniel Goldin left NASA in November 2001 after more than nine years as Administrator. The White House chose Sean O'Keefe, the Deputy Director of the White House Office of Management and Budget, as his replacement. O'Keefe stated as he took office that he was not a "rocket scientist," but rather that his expertise was in the management of large government programs. His appointment was an explicit acknowledgement by the new Bush administration that NASA's primary problems were managerial and financial. By the time O'Keefe arrived, NASA managers had come to recognize that 1990s funding reductions for the Space Shuttle Program had resulted in an excessively fragile program, and also realized that a Space Shuttle replacement was not on the horizon. In 2002, with these issues in mind, O'Keefe made a number of changes to the Space Shuttle Program. He transferred management of both the Space Shuttle Program and the International Space Station from Johnson Space Center to NASA Headquarters. O'Keefe also began considering whether to expand the Space Flight Operations Contract to cover additional Space Shuttle elements, or to pursue "competitive sourcing," a Bush administration initiative that encouraged government agencies to compete with the private sector for management responsibilities of publicly funded activities. To research whether competitive sourcing would be a viable approach for the Space Shuttle Program, NASA chartered the Space Shuttle Competitive Sourcing Task Force through the RAND Corporation, a federally funded think tank. In its report, the Task Force recognized the many obstacles to transferring the Space Shuttle to non-NASA management, primarily NASA's reticence to relinquish control, but concluded that "NASA must pursue competitive sourcing in one form or another."68 NASA began a "Strategic Management of Human Capital" initiative to ensure the quality of the future NASA workforce. The goal is to address the various external and internal challenges that NASA faces as it tries to ensure an appropriate mix and depth of skills for future program requirements. A number of aspects to its Strategic Human Capital Plan require legislative approval and are currently before the Congress.
Figure 5.5-4. Age of the Space Shuttle infrastructure. (Source: Connie Milton to Space Flight Advisory Council, 2000.
on certain launch pad areas being exposed to the elements. When rain falls on these areas, it carries away zinc, runs onto the leading edge of the Orbiter's wings, and causes pinholes in the Reinforced Carbon-Carbon panels (see Chapter 3). In 2000, NASA identified 100 infrastructure items that demanded immediate attention. NASA briefed the Space Flight Advisory Committee on this "Infrastructure Revitalization" initiative in November of that year. The Committee concluded that "deteriorating infrastructure is a serious, major problem," and, upon touring several Kennedy Space Center facilities, declared them "in deplorable condition."67 NASA subsequently submitted a request to the White House Office of Management and Budget during Fiscal Year 2002 budget deliberations for $600 million to fund the infrastructure initiative. No funding was approved. In Fiscal Year 2002, Congress added $25 million to NASA's budget for Vehicle Assembly Building repairs. NASA has reallocated limited funds from the Shuttle budget to pressing infrastructure repairs, and intends to take an integrated look at infrastructure as part of its new Shuttle Service Life Extension Program. Nonetheless, like Space Shuttle upgrades, infrastructure revitalization has been mired by the uncertainty surrounding the Shuttle Program's lifetime. Considering that the Shuttle will likely be flying for many years to come, NASA, the White House, and Congress alike now face the specter of having to deal with years of infrastructure neglect.
• Roof • Siding • Doors
Boxcar Offices
Figure 5.5-5 and 5.5-6. Examples of the seriously deteriorating infrastructure used to support the Space Shuttle Program. At left is Launch Complex 39A, and at right is the Vehicle Assembly building, both at the Kennedy Space Center.
Report Volume I August 2003
The new NASA leadership also began to compare Space Shuttle program practices with the practices of similar high-technology, high-risk enterprises. The Navy nuclear submarine program was the first enterprise selected for comparative analysis. An interim report on this "benchmarking" effort was presented to NASA in December 2002.69 In November 2002, NASA made a fundamental change in strategy. In what was called the Integrated Space Transportation Plan (see Figure 5.6-1), NASA shifted money from the Space Launch Initiative to the Space Shuttle and International Space Station programs. The plan also introduced the Orbital Space Plane as a complement to the Shuttle for the immediate future. Under this strategy, the Shuttle is to fly through at least 2010, when a decision will be made on how long to extend Shuttle operations – possibly through 2020 or even beyond. As a step in implementing the plan, NASA included $281.4 million in its Fiscal Year 2004 budget submission to begin a Shuttle Service Life Extension Program,70 which NASA describes as a "strategic and proactive program designed to keep the Space Shuttle flying safely and efficiently." The program includes "high priority projects for safety, supportability, and infrastructure" in order to "combat obsolescence of vehicle, ground systems, and facilities."71
When the Bush Administration came to the White House in January 2001, the International Space Station program was $4 billion over its projected budget. The Administration's Fiscal Year 2002 budget, released in February 2001, declared that the International Space Station would be limited to a "U.S Core Complete" configuration, a reduced design that could accommodate only three crew members. The last step in completing the U.S. portion of this configuration would be the addition of the Italian-supplied but U.S.owned "Node 2," which would allow Europe and Japan to connect their laboratory modules to the Station. Launching Node 2 and thereby finishing "core complete" configuration became an important political and programmatic milestone (see Figure 5.7-1).
Node 2
Figure 5.7-1. The "Core Complete" configuration of the International Space Station.
Figure 5.6-1. The Integrated Space Transportation Plan.
During congressional testimony in May of 2001, Sean O'Keefe, who was then Deputy Director of the White House Office of Management and Budget, presented the Administration's plan to bring International Space Station costs under control. The plan outlined a reduction in assembly and logistics flights to reach "core complete" configuration from 36 to 30. It also recommended redirecting about $1 billion in funding by canceling U.S. elements not yet completed, such as the habitation module and the X-38 Crew Return Vehicle. The X-38 would have allowed emergency evacuation and landing capability for a seven-member station crew. Without it, the crew was limited to three, the number that could fit into a Russian Soyuz crew rescue vehicle. In his remarks, O'Keefe stated: NASA's degree of success in gaining control of cost growth on Space Station will not only dictate the capabilities that the Station will provide, but will send a strong signal about the ability of NASA's Human Space Flight program to effectively manage large development programs. NASA's credibility with the Administration and the Congress for delivering on what is promised and the longer-term implications that such credibility may have on the future of Human Space Flight hang in the balance.72 At the request of the White House Office of Management and Budget, in July 2001 NASA Administrator Dan Goldin
August 2003
The International Space Station has been the centerpiece of NASA's human space flight program in the 1990s. In several instances, funds for the Shuttle Program have paid for various International Space Station items. The Space Station has also affected the Space Shuttle Program schedule. By the time the functional cargo block Zarya, the Space Station's first element, was launched from the Baikonur Cosmodrome in Kazakhstan in November 1998, the Space Station was two years behind schedule. The launch of STS-88, the first of many Shuttle missions assigned to station assembly, followed a month later. Another four assembly missions in 1999 and 2000 readied the station for its first permanent crew, Expedition 1, which arrived in late 2000.
Report Volume I
formed an International Space Station Management and Cost Evaluation Task Force. The International Space Station Management and Cost Evaluation Task Force was to assist NASA in identifying the reforms needed to restore the Station Program's fiscal and management credibility. While the primary focus of the Task Force was on the Space Station Program management, its November 2001 report issued a general condemnation of how NASA, and particularly Johnson Space Center, had managed the International Space Station, and by implication, NASA's overall human space flight effort. 73 The report noted "existing deficiencies in management structure, institutional culture, cost estimating, and program control," and that "the institutional needs of the [human space flight] Centers are driving the Program, rather than Program requirements being served by the Centers." The Task Force suggested that as a cost control measure, the Space Shuttle be limited to four flights per year and that NASA revise the station crew rotation period to six months. The cost savings that would result from eliminating flights could be used to offset cost overruns. NASA accepted a reduced flight rate. The Space Shuttle Program office concluded that, based on a rate of four flights a year, Node 2 could be launched by February 19, 2004. In testimony before the House Committee on Science on November 7, 2001, Task Force Chairman Thomas Young identified what became known as a "performance gate." He suggested that over the next two years, NASA should plan and implement a credible "core complete" program. In Fall 2003, "an assessment would be made concerning the ISS program performance and NASA's credibility. If satisfactory, resource needs would be assessed and an [ISS] ʻend state' that realized the science potential would become the baseline. If unsatisfactory, the core complete program would become the ʻend state.' "74 Testifying the same day, Office of Management and Budget Deputy Director Sean O'Keefe indicated the Administration's agreement with the planned performance gate: The concept presented by the task force of a decision gate in two years that could lead to an end state other than the U.S. core complete Station is an innovative approach, and one the Administration will adopt. It calls for NASA to make the necessary management reforms to successfully build the core complete Station and operate it within the $8.3 billion available through FY 2006 plus other human space flight resources … If NASA fails to meet the standards, then an end-state beyond core complete is not an option. The strategy places the burden of proof on NASA performance to ensure that NASA fully implements the needed reforms.75 Mr. O'Keefe added in closing: A most important next step – one on which the success of all these reforms hinges – is to provide new leadership for NASA and its Human Space Flight activities. NASA has been well-served by Dan Goldin. New leadership is now necessary to continue moving the ball down the
Report Volume I
field with the goal line in sight. The Administration recognizes the importance of getting the right leaders in place as soon as possible, and I am personally engaged in making sure that this happens. A week later, Sean O'Keefe was nominated by President Bush as the new NASA Administrator. To meet the new flight schedule, in 2002 NASA revised its Shuttle manifest, calling for a docking adaptor to be installed in Columbia after the STS-107 mission so that it could make an October 2003 flight to the International Space Station. Columbia was not optimal for Station flights – the Orbiter could not carry enough payload – but it was assigned to this flight because Discovery was scheduled for 18 months of major maintenance. To ensure adequate Shuttle availability for the February 2004 Node 2 launch date, Columbia would fly an International Space Station resupply mission. The White House and Congress had put the International Space Station Program, the Space Shuttle Program, and indeed NASA on probation. NASA had to prove it could meet schedules within cost, or risk halting Space Station construction at core complete – a configuration far short of what NASA anticipated. The new NASA management viewed the achievement of an on-schedule Node 2 launch as an endorsement of its successful approach to Shuttle and Station Programs. Any suggestions that it would be difficult to meet that launch date were brushed aside. This insistence on a fixed launch schedule was worrisome. The International Space Station Management and Cost Evaluation Task Force, in particular, was concerned with the emphasis on a specific launch date. It noted in its 2002 review of progress toward meeting its recommendations that "significant progress has been made in nearly all aspects of the ISS Program," but that there was "significant risk with the Node 2 (February '04) schedule."76 By November 2002, NASA had flown 16 Space Shuttle missions dedicated to Station assembly and crew rotation. Five crews had lived onboard the Station, the last four of them delivered via Space Shuttles. As the Station had grown, so had the complexity of the missions required to complete it. With the International Space Station assembly more than half complete, the Station and Shuttle programs had become irreversibly linked. Any problems with or perturbations to the planned schedule of one program reverberated through both programs. For the Shuttle program, this meant that the conduct of all missions, even non-Station missions like STS-107, would have an impact on the Node 2 launch date. In 2002, this reality, and the events of the months that would follow, began to place additional schedule pressures on the Space Shuttle Program. Those pressures are discussed in Section 6.2.
Over the last decade, the Space Shuttle Program has operated in a challenging and often turbulent environment. As
August 2003
discussed in this chapter, there were at least three major contributing factors to that environment: • Throughout the decade, the Shuttle Program has had to function within an increasingly constrained budget. Both the Shuttle budget and workforce have been reduced by over 40 percent during the past decade. The White House, Congress, and NASA leadership exerted constant pressure to reduce or at least freeze operating costs. As a result, there was little margin in the budget to deal with unexpected technical problems or make Shuttle improvements. • The Shuttle was mischaracterized by the 1995 Kraft Report as "a mature and reliable system … about as safe as today's technology will provide." Based on this mischaracterization, NASA believed that it could turn increased responsibilities for Shuttle operations over to a single prime contractor and reduce its direct involvement in ensuring safe Shuttle operations, instead monitoring contractor performance from a more detached position. NASA also believed that it could use the "mature" Shuttle to carry out operational missions without continually focusing engineering attention on understanding the mission-by-mission anomalies inherent in a developmental vehicle. • In the 1990s, the planned date for replacing the Shuttle shifted from 2006 to 2012 and then to 2015 or later. Given the uncertainty regarding the Shuttle's service life, there has been policy and budgetary ambivalence on investing in the vehicle. Only in the past year has NASA begun to provide the resources needed to sustain extended Shuttle operations. Previously, safety and support upgrades were delayed or deferred, and Shuttle infrastructure was allowed to deteriorate. The Board observes that this is hardly an environment in which those responsible for safe operation of the Shuttle can function without being influenced by external pressures. It is to the credit of Space Shuttle managers and the Shuttle workforce that the vehicle was able to achieve its program objectives for as long as it did. An examination of the Shuttle Program's history from Challenger to Columbia raises the question: Did the Space Shuttle Program budgets constrained by the White House and Congress threaten safe Shuttle operations? There is no straightforward answer. In 1994, an analysis of the Shuttle budget concluded that reductions made in the early 1990s represented a "healthy tightening up" of the program.77 Certainly those in the Office of Management and Budget and in NASA's congressional authorization and appropriations subcommittees thought they were providing enough resources to operate the Shuttle safely, while also taking into account the expected Shuttle lifetime and the many other demands on the Federal budget. NASA Headquarters agreed, at least until Administrator Goldin declared a "space launch crisis" in June 1999 and asked that additional resources for safety upgrades be added to the NASA budget. By 2001, however, one experienced observer of the space program described the Shuttle workforce as "The Few, the Tired,"
Report Volume I
and suggested that "a decade of downsizing and budget tightening has left NASA exploring the universe with a less experienced staff and older equipment."78 It is the Board's view that this latter statement is an accurate depiction of the Space Shuttle Program at the time of STS107. The Program was operating too close to too many margins. The Board also finds that recent modest increases in the Shuttle Program's budget are necessary and overdue steps toward providing the resources to sustain the program for its now-extended lifetime. Similarly, NASA has recently recognized that providing an adequately sized and appropriately trained workforce is critical to the agency's future success. An examination of the Program's management changes also leads to the question: Did turmoil in the management structure contribute to the accident? The Board found no evidence that the transition from many Space Shuttle contractors to a partial consolidation of contracts under a single firm has by itself introduced additional technical risk into the Space Shuttle Program. The transfer of responsibilities that has accompanied the Space Flight Operations Contract has, however, complicated an already complex Program structure and created barriers to effective communication. Designating the Johnson Space Center as the "lead center" for the Space Shuttle Program did resurrect some of the Center rivalries and communication difficulties that existed before the Challenger accident. The specific ways in which this complexity and lack of an integrated approach to Shuttle management impinged on NASA's performance during and before the flight of STS-107 are discussed in Chapters 6 and 7. As the 21st century began, NASA's deeply ingrained human space flight culture – one that has evolved over 30 years as the basis for a more conservative, less technically and organizationally capable organization than the Apollo-era NASA – remained strong enough to resist external pressures for adaptation and change. At the time of the launch of STS-107, NASA retained too many negative (and also many positive) aspects of its traditional culture: "flawed decision making, self deception, introversion and a diminished curiosity about the world outside the perfect place."79 These characteristics were reflected in NASA's less than stellar performance before and during the STS-107 mission, which is described in the following chapters.
August 2003
The citations that contain a reference to "CAIB document" with CAB or CTF followed by seven to eleven digits, such as CAB001-0010, refer to a document in the Columbia Accident Investigation Board database maintained by the Department of Justice and archived at the National Archives.
1 16
Bush administration space policy is discussed in Dan Quayle, Standing Firm: A Vice-Presidential Memoir (New York: Harper Collins, 1994), pp. 185-190. Report of the Advisory Committee on the Future of the U.S. Space Program, December 1990. The quotes are from p. 2 of the report's executive summary. Report of the Advisory Committee on the Future of the U.S. Space Program. Measured in terms of total national spending, the report's recommendations would have returned NASA spending to 0.38 percent of U.S. Gross Domestic Product – a level of investment not seen since 1969. For Fiscal Years 1965-2002 in Real and Constant Dollars, see NASA, "Space Activities of the U.S. Government – in Millions of Real Year Dollars," and "Space Activities of the U.S. Government – Adjusted for Inflation," in Aeronautics and Space Report of the President – Fiscal Year 2002 Activity, forthcoming. For Fiscal Years 2003-2004 in Real Dollars, see Office of Management and Budget, "Outlays By Agency: 19622008," in Historical Budget of the United States Government, Fiscal Year 2004, (Washington: Government Printing Office, 2003), pp. 70-75. Commission on the Future of the U.S. Aerospace Industry, Final Report, November 18, 2002, p. 3-1. U.S. Congress, Office of Technology Assessment, "Shuttle Fleet Attrition if Orbiter Recovery Reliability is 98 Percent," August 1989, p. 6. From: Round Trip to Orbit: Human Space Flight Alternatives: Special Report, OTS-ISC-419. Report of the Advisory Committee on the Future of the U.S. Space Program. Howard E. McCurdy, Faster, Better, Cheaper: Low-Cost Innovation in the U.S. Space Program (Baltimore: The Johns Hopkins University Press, 2001). Letter from Daniel Goldin to Representative James T. Walsh, October 4, 2001. CAIB document CAB065-01630169. Ibid. W. Henry Lambright, Transforming Government: Dan Goldin and the Remaking of NASA (Washington: Price Waterhouse Coopers Endowment for the Business of Government, March 2001), pp. 12; 27-29. Deming's management philosophy was not the only new notion that Goldin attempted to apply to NASA. He was also an advocate of the "Total Quality Management" approach and other modern management schemes. Trying to adapt to these various management theories was a source of some stress. For a discussion of Goldin's approach, see Howard McCurdy, Faster, Better, Cheaper: Low-Cost Innovation in the U.S. Space Program (Baltimore: The Johns Hopkins University Press, 2001). It is worth noting that while the "faster, better, cheaper" approach led to many more NASA robotic missions being launched after 1992, not all of those missions were successful. In particular, there were two embarrassing failures of Mars missions in 1999. Lambright, Transforming Government, provides an early but comprehensive evaluation of the Goldin record. The quote is from p. 28. Goldin is quoted in Bill Harwood, "Pace of Cuts Fuels Concerns About Shuttle," Space News, December 19-25, 1994, p. 1. McCurdy, Faster, Better, Cheaper.
Report of the Presidential Commission on the Space Shuttle Challenger Accident, June 6, 1986, (Washington: Government Printing Office, 1986), Vol. I, p. 82, 118. Report of the Presidential Commission, Vol. I, p. 48. Report of the Presidential Commission, Vol. I, p. 52. Report of the Presidential Commission, Vol. I, pp. 164-165. Report of the Presidential Commission, Vol. I, pp. 198-201. Report of The National Commission for the Review of the National Reconnaissance Office: The NRO at the Crossroads, November 2000, p. 66. Roger Guillemette, "Vandenberg: Space Shuttle Launch and Landing Site, Part 1," Spaceflight, October 1994, pp. 354-357, and Roger Guillemette, "Vandenberg: Space Shuttle Launch and Landing Site, Part 2," Spaceflight, November 1994, pp. 378-381; Dennis R. Jenkins, Space Shuttle: The History of the National Space Transportation System – The First 100 Missions (Cape Canaveral, FL, Specialty Press, 2001), pp. 467476. Vice President's Space Policy Advisory Board, A Post Cold War Assessment of U.S. Space Policy, December 1992, p. 6. Quoted in John M. Logsdon, "Return to Flight: Richard H. Truly and the Recovery from the Challenger Accident," in Pamela E. Mack, editor, From Engineering to Big Science: The NACA and NASA Collier Trophy Research Project Winners, NASA SP-4219 (Washington: Government Printing Office, 1998), p. 363. Aviation Week & Space Technology, November 10, 1986, p. 30. There are proposals for using other U.S. systems, in development but not yet ready for flight, to provide an alternate U.S. means of station access. These "Alternate Access to Space" proposals have not been evaluated by the Board. Testimony of William F. Readdy to the Subcommittee on Science, Technology and Space, U.S. Senate, September 6, 2001. Howard E. McCurdy, Inside NASA: High Technology and Organizational Change in the U.S. Space Program (Baltimore: The Johns Hopkins University Press, 1993), p. 24. Garry D. Brewer, "Perfect Places: NASA as an Idealized Institution," in Radford Byerly, Jr., ed., Space Policy Reconsidered (Boulder, CO: Westview Press, 1989), p. 158. Brewer, when he wrote these words, was a professor of organizational behavior at Yale University with no prior exposure to NASA. For first-hand discussions of NASA's Apollo-era organizational culture, see Christopher Kraft, Flight: My Life in Mission Control (New York: E.P. Dutton, 2001); Gene Kranz, Failure is Not an Option: Mission Control from Mercury to Apollo 13 (New York: Simon & Schuster, 2000); and Thomas J. Kelly, Moon Lander: How We Developed the Apollo Lunar Module (Washington: Smithsonian Institution Press, 2001). Brewer, "Perfect Places," pp. 159-165. As NASA human space flight personnel began to become closely involved with their counterparts in the Russian space program after 1992, there was grudging acceptance that Russian human space flight personnel were also skilled in their work, although they carried it out rather differently than did NASA. Report Volume I
2 3 4 5 6
20 21
7 8
22 23
9 10
24 25 26
11 12
14 15
30 31
August 2003
For two recent works that apply the "Iron Triangle" concept to other policy areas, see Randall B. Ripley and Grace A. Franklin, Congress, the Bureaucracy and Public Policy, 5th Edition, (Pacific Grove, CA: Brooks/ Cole Publishing Company, 1991); and Paul C. Light, Forging Legislation: The Politics of Veterans Reform, (New York: W. W. Norton, 1992). Information obtained from Anna Henderson, NASA Office of Space Flight, to e-mail to John Logsdon, June 13, 2003. National Academy of Public Administration, A Review of the Space Shuttle Costs, Reduction Goals, and Procedures, December 1994, pp. 3-5. CAIB document CAB026-0313. Presentation to NASA Advisory Council by Stephen Oswald, Acting Director, Space Shuttle Requirements, "Space Flight Operations Contract (SFOC) Acquisition Status," April 23, 1996. CAIB document CTF0641369. Bryan D. O'Connor, Status Briefing to NASA Administrator, "Space Shuttle Functional Workforce Review," February 14, 1995. CAIB document CAB015-0400. Ralph Vartabedian, "Ex-NASA Chief Hits Flight Safety," Houston Chronicle, March 7, 1996. Kathy Sawyer, "NASA Space Shuttle Director Resigns," Washington Post, February 3, 1996, p. A3. See also "Take this Job and Shuttle It: Why NASA's Space Shuttle Chief Quit," Final Frontier, July/August 1996, pp. 16-17; "NASA Alters Its Management, Philosophy," Space News, February 12-18, 1996, p. 3. Report of the Space Shuttle Management Independent Review Team, February 1995. Ibid, pp. 3-18. NASA News Release 95-27, "Shuttle Management Team Issues Final Report," March 15, 1995. Aerospace Safety Advisory Panel, "Review of the Space Shuttle Management Independent Review Program," May 1995. CAIB document CAB015-04120413. Jose Garcia to President William Jefferson Clinton, August 25, 1995. See, for instance: "Determinations and Findings for the Space Shuttle Program," United States House of Representatives, Subcommittee on Space, of the Committee on Science, 104 Cong., 1 Sess., November 30, 1995. See remarks by Daniel S. Goldin, Opening Remarks at the September 30, 1996, ceremony commemorating the signing of the Space Flight Operations Contract, Houston, Texas. (Videotape recording.) Congressional Budget Office, "NASA's Space Flight Operations Contract and Other Technologically Complex Government Activities Conducted by Contractors," July 29, 2003. Russell Turner, testimony at public hearing before the Columbia Accident Investigation Board, June 12, 2003. See Section 204 of Public Law 105-303, October 28, 1999. Joe Rothenberg to Dan Goldin, August 17, 2001, CAIB document CAB015-1134; "Space Shuttle Privatization," CAIB document CAB0151135; "Space Shuttle Privatization: Options and Issues," Rev: 8/14/01, CAIB document CAB015-1147. Ron Dittemore, "Concept of Privatization of the Space Shuttle Program," September 2001. CAIB document CTF005-0283. Ibid. Roy Bridges, Testimony before the Columbia Accident Investigation Board, March 25, 2003. The quotes are taken from NASA-submitted material appended to the statement of NASA Administrator Daniel Goldin to the Senate Subcommittee on Science, Technology and Space, March 22, 2000, p. 7. National Commission on Space, Pioneering the Space Frontier: An Exciting Vision of Our Next Fifty Years in Space, Report of the National Commission on Space (Bantam Books, 1986). President Ronald Reagan, "Message to the Congress on America's Agenda for the Future," February 6, 1986, Public Papers of the Presidents of the United States: Ronald Reagan: Book I-January 1 to June 27, 1986 (Washington, DC: U.S. Government Printing Office, 19821991), p. 159. Office of Space Systems Development, NASA Headquarters, "Access to Space Study—Summary Report," January 1994, reproduced in John M. Logsdon, et al. eds., Exploring the Unknown, Volume IV: Accessing Space NASA SP-4407 (Government Printing Office, 1999), pp. 584-604. Report Volume I
The White House, Office of Science and Technology Policy, "Fact Sheet--National Space Transportation Policy," August 5, 1994, pp. 1-2, reprinted in Logsdon et al., Exploring the Unknown, Volume IV, pp. 626631. Report of the Space Shuttle Management Independent Review Team, pp. 3-18. "Statement of William F. Readdy, Deputy Associate Administrator, Office of Space Flight, National Aeronautics and Space Administration before the Subcommittee on Space and Aeronautics Committee on Science, House of Representatives," October 21, 1999. CAIB document CAB0260146. Letter from Daniel Goldin to Jacob Lew, Director, Office of Management and Budget, July 6, 1999. NASA, Space Shuttle Independent Assessment Team, "Report to the Associate Administrator, Office of Space Flight, October-December 1999," March 7, 2000. CAIB document CTF017-0169. Ibid. Ibid. Dr. Richard Beck, Director, Resources Analysis Division, NASA, "Agency Budget Overview, FY 2003 Budget," February 6, 2002, p. 20. CAIB document CAB070-0001. Space Flight Advisory Committee, NASA Office of Space Flight, Meeting Report, May 1-2, 2001, p. 7. CAIB document CTF017-0034. Senators Bill Nelson, Bob Graham, Mary Landrieu, John Breaux, and Orrin Hatch to Senator Barbara Mikulski, September 18, 2001. Space Flight Advisory Committee, NASA Office of Space Flight, Meeting Report, May 1-2, 2001, p. 7. CAIB document CTF017-0034. Task Force on Space Shuttle Competitive Sourcing, Alternate Trajectories: Options for Competitive Sourcing of the Space Shuttle Program, Executive Summary, The RAND Corporation, 2002. CAIB document CAB003-1614. NNBE Benchmarking Team, NASA Office of Safety & Mission Assurance and NAVSEA 92Q Submarine Safety & Quality Assurance Division, "NASA/Navy Benchmarking Exchange (NNBE)," Interim Report, December 20, 2002. CAIB document CAB030-0392. The team's final report was issued in July 2003. NASA FY 2004 Congressional Budget, "Theme: Space Shuttle." [Excerpt from NASA FY 2004 budget briefing book also known as the "IBPD Narrative"]. CAIB document CAB065-04190440. NASA, "Theme: Space Shuttle." CAIB document CAB065-04190440. Testimony of Sean O'Keefe, Deputy Director, Office of Management and Budget, to the Subcommittee of the Committee on Appropriations, "Part 1, National Aeronautics and Space Administration," Hearings Before a Subcommittee of the Committee on Appropriations, United States House of Representatives, 107th Congress, 1st Sess., May 2001, p. 32. "Report by the International Space Station (ISS) Management and Cost Evaluation (IMCE) Task Force to the NASA Advisory Council," November 1, 2001, pp. 1-5. CAIB document CTF044-6016. Testimony of Tom Young, Chairman, ISS Management and Cost Evaluation (IMCE) Task Force, to the Committee on Science, U.S. House of Representatives, "The Space Station Task Force Report," Hearing Before the Committee on Science, United States House of Representatives, 107th Congress, 1st Sess., November, 2001, p. 23. Testimony of Sean O'Keefe, Deputy Director, Office of Management and Budget, to the Committee on Science, U.S. House of Representatives, "The Space Station Task Force Report," Hearing Before the Committee on Science, United States House of Representatives, 107th Congress, 1st Sess., November, 2001, p. 28. Thomas Young, IMCE Chair, "International Space Station (ISS) Management and Cost Evaluation (IMCE) Task Force Status Report to the NASA Advisory Council," (Viewgraphs) December 11, 2002, p. 11. CAIB document CAB065-0189. General Research Corporation, Space Shuttle Budget Allocation Review, Volume 1, July 1994, p. 7. CAIB document CAIB015-0161. Beth Dickey, "The Few, the Tired," Government Executive, April 2001, p. 71. Brewer, "Perfect Places," pp. 159.
58 59
33 34
60 61
62 63 64
37 38
65 66 67 68
39 40 41 42
43 44
71 72
47 48 49
50 51 52 53
77 78 79
August 2003
Decision Making at NASA
The dwindling post-Cold War Shuttle budget that launched NASA leadership on a crusade for efficiency in the decade before Columbia's final flight powerfully shaped the environment in which Shuttle managers worked. The increased organizational complexity, transitioning authority structures, and ambiguous working relationships that defined the restructured Space Shuttle Program in the 1990s created turbulence that repeatedly influenced decisions made before and during STS-107. This chapter connects Chapter 5's analysis of NASA's broader policy environment to a focused scrutiny of Space Shuttle Program decisions that led to the STS-107 accident. Section 6.1 illustrates how foam debris losses that violated design requirements came to be defined by NASA management as an acceptable aspect of Shuttle missions, one that posed merely a maintenance "turnaround" problem rather than a safety-of-flight concern. Section 6.2 shows how, at a pivotal juncture just months before the Columbia accident, the management goal of completing Node 2 of the International Space Station on time encouraged Shuttle managers to continue flying, even after a significant bipod-foam debris strike on STS-112. Section 6.3 notes the decisions made during STS-107 in response to the bipod foam strike, and reveals how engineers' concerns about risk and safety were competing with – and were defeated by – management's belief that foam could not hurt the Orbiter, as well as the need to keep on schedule. In relating a rescue and repair scenario that might have enabled the crew's safe return, Section 6.4 grapples with yet another latent assumption held by Shuttle managers during and after STS-107: that even if the foam strike had been discovered, nothing could have been done. flying the Shuttle with a known problem that violated design requirements? It would seem that the longer the Shuttle Program allowed debris to continue striking the Orbiters, the more opportunity existed to detect the serious threat it posed. But this is not what happened. Although engineers have made numerous changes in foam design and application in the 25 years that the External Tank has been in production, the problem of foam-shedding has not been solved, nor has the Orbiter's ability to tolerate impacts from foam or other debris been significantly improved. The Need for Foam Insulation The External Tank contains liquid oxygen and hydrogen propellants stored at minus 297 and minus 423 degrees Fahrenheit. Were the super-cold External Tank not sufficiently insulated from the warm air, its liquid propellants would boil, and atmospheric nitrogen and water vapor would condense and form thick layers of ice on its surface. Upon launch, the ice could break off and damage the Orbiter. (See Chapter 3.) To prevent this from happening, large areas of the External Tank are machine-sprayed with one or two inches of foam, while specific fixtures, such as the bipod ramps, are hand-sculpted with thicker coats. Most of these insulating materials fall into a general category of "foam," and are outwardly similar to hardware store-sprayable foam insulation. The problem is that foam does not always stay where the External Tank manufacturer Lockheed Martin installs it. During flight, popcorn- to briefcase-size chunks detach from the External Tank. Original Design Requirements Early in the Space Shuttle Program, foam loss was considered a dangerous problem. Design engineers were extremely concerned about potential damage to the Orbiter and its fragile Thermal Protection System, parts of which are so vulnerable to impacts that lightly pressing a thumbnail into them leaves a mark. Because of these concerns, the baseline
August 2003
The shedding of External Tank foam – the physical cause of the Columbia accident – had a long history. Damage caused by debris has occurred on every Space Shuttle flight, and most missions have had insulating foam shed during ascent. This raises an obvious question: Why did NASA continue
Report Volume I
design requirements in the Shuttle's "Flight and Ground System Specification-Book 1, Requirements," precluded foam-shedding by the External Tank. Specifically: 220.127.116.11.14 Debris Prevention: The Space Shuttle System, including the ground systems, shall be designed to preclude the shedding of ice and/or other debris from the Shuttle elements during prelaunch and flight operations that would jeopardize the flight crew, vehicle, mission success, or would adversely impact turnaround operations.1 18.104.22.168.17 External Tank Debris Limits: No debris shall emanate from the critical zone of the External Tank on the launch pad or during ascent except for such material which may result from normal thermal protection system recession due to ascent heating.2 The assumption that only tiny pieces of debris would strike the Orbiter was also built into original design requirements, which specified that the Thermal Protection System (the tiles and Reinforced Carbon-Carbon, or RCC, panels) would be built to withstand impacts with a kinetic energy less than 0.006 foot-pounds. Such a small tolerance leaves the Orbiter vulnerable to strikes from birds, ice, launch pad debris, and pieces of foam. Despite the design requirement that the External Tank shed no debris, and that the Orbiter not be subjected to any significant debris hits, Columbia sustained damage from debris strikes on its inaugural 1981 flight. More than 300 tiles had to be replaced.3 Engineers stated that had they known in advance that the External Tank "was going to produce the debris shower that occurred" during launch, "they would have had a difficult time clearing Columbia for flight."4 Discussion of Foam Strikes Prior to the Rogers Commission Foam strikes were a topic of management concern at the time of the Challenger accident. In fact, during the Rogers Commission accident investigation, Shuttle Program Manager Arnold Aldrich cited a contractor's concerns about foam shedding to illustrate how well the Shuttle Program manages risk: On a series of four or five external tanks, the thermal insulation around the inner tank … had large divots of insulation coming off and impacting the Orbiter. We found significant amount of damage to one Orbiter after a flight and … on the subsequent flight we had a camera in the equivalent of the wheel well, which took a picture of the tank after separation, and we determined that this was in fact the cause of the damage. At that time, we wanted to be able to proceed with the launch program if it was acceptable … so we undertook discussions of what would be acceptable in terms of potential field repairs, and during those discussions, Rockwell was very conservative because, rightly, damage to the Orbiter TPS [Thermal Protection System] is damage to the Orbiter system, and it has a very stringent environment to experience during the re-entry phase.
Report Volume I
Aldrich described the pieces of foam as "… half a foot square or a foot by half a foot, and some of them much smaller and localized to a specific area, but fairly high up on the tank. So they had a good shot at the Orbiter underbelly, and this is where we had the damage."5 Continuing Foam Loss Despite the high level of concern after STS-1 and through the Challenger accident, foam continued to separate from the External Tank. Photographic evidence of foam shedding exists for 65 of the 79 missions for which imagery is available. Of the 34 missions for which there are no imagery, 8 missions where foam loss is not seen in the imagery, and 6 missions where imagery is inconclusive, foam loss can be inferred from the number of divots on the Orbiter's lower surfaces. Over the life of the Space Shuttle Program, Orbiters have returned with an average of 143 divots in the upper and lower surfaces of the Thermal Protection System tiles, with 31 divots averaging over an inch in one dimension.6 (The Orbiters' lower surfaces have an average of 101 hits, 23 of which are larger than an inch in diameter.) Though the Orbiter is also struck by ice and pieces of launch-pad hardware during launch, by micrometeoroids and orbital debris in space, and by runway debris during landing, the Board concludes that foam is likely responsible for most debris hits. With each successful landing, it appears that NASA engineers and managers increasingly regarded the foam-shedding as inevitable, and as either unlikely to jeopardize safety or simply an acceptable risk. The distinction between foam loss and debris events also appears to have become blurred. NASA and contractor personnel came to view foam strikes not as a safety of flight issue, but rather a simple maintenance, or "turnaround" issue. In Flight Readiness Review documentation, Mission Management Team minutes, InFlight Anomaly disposition reports, and elsewhere, what was originally considered a serious threat to the Orbiter
In Family: A reportable problem that was previously experienced, analyzed, and understood. Out of limits performance or discrepancies that have been previously experienced may be considered as in-family when specifically approved by the Space Shuttle Program or design project.8 Out of Family: Operation or performance outside the expected performance range for a given parameter or which has not previously been experienced.9 Accepted Risk: The threat associated with a specific circumstance is known and understood, cannot be completely eliminated, and the circumstance(s) producing that threat is considered unlikely to reoccur. Hence, the circumstance is fully known and is considered a tolerable threat to the conduct of a Shuttle mission. No Safety-of-Flight-Issue: The threat associated with a specific circumstance is known and understood and does not pose a threat to the crew and/or vehicle.
August 2003
Flight ET # ET Type Orbiter Inclination Launch Date Launch Time (Local)
STS-7 06 SWT Challenger 28.45 deg 06/18/83 07:33:00 AM EDT
STS-32R 25 LWT Columbia 28.45 deg 01/09/90 07:35:00 AM EST
STS-50 45 LWT Columbia 28.45 deg 06/25/92 12:12:23 PM EDT
STS-52 55 LWT Columbia 28.45 deg 10/22/92 1:09:39 PM EDT
STS-62 62 LWT Columbia 39.0 deg 03/04/94 08:53:00 AM EST
STS-112 115 SLWT Atlantis 51.6 deg 10/07/02 3:46:00 PM EDT
STS-107 93 LWT Columbia 39.0 deg 01/16/03 10:39:00 AM EDT
Figure 6.1-1. There have been seven known cases where the left External Tank bipod ramp foam has come off in flight.
came to be treated as "in-family," 7 a reportable problem that was within the known experience base, was believed to be understood, and was not regarded as a safety-of-flight issue. Bipod Ramp Foam Loss Events Chunks of foam from the External Tank's forward bipod attachment, which connects the Orbiter to the External Tank, are some of the largest pieces of debris that have struck the Orbiter. To place the foam loss from STS-107 in a broader context, the Board examined every known instance of foam-shedding from this area. Foam loss from the left bipod ramp (called the –Y ramp in NASA parlance) has been confirmed by imagery on 7 of the 113 missions flown. However, only on 72 of these missions was available imagery of sufficient quality to determine left bipod ramp foam loss. Therefore, foam loss from the left bipod area occurred on approximately 10 percent of flights (seven events out of 72 imaged flights). On the 66 flights that imagery was available for the right bipod area, foam loss was never observed. NASA could not explain why only the left bipod experienced foam loss. (See Figure 6.1-1.)
The first known bipod ramp foam loss occurred during STS-7, Challenger's second mission (see Figure 6.1-2). Images taken after External Tank separation revealed that a 19- by 12-inch piece of the left bipod ramp was missing, and that the External Tank had some 25 shallow divots in the foam just forward of the bipod struts and another 40 divots in the foam covering the lower External Tank. After the mission was completed, the Program Requirements Control Board cited the foam loss as an In-Flight Anomaly. Citing an event as an In-Flight Anomaly means that before the next launch, a specific NASA organization must resolve the problem or prove that it does not threaten the safety of the vehicle or crew.11 At the Flight Readiness Review for the next mission, Orbiter Project management reported that, based on the completion of repairs to the Orbiter Thermal Protection System, the bipod ramp foam loss In-Flight Anomaly was resolved, or "closed." However, although the closure documents detailed the repairs made to the Orbiter, neither the Certificate of Flight Readiness documentation nor the Flight Readiness Review documentation referenced correcting the cause of the damage – the shedding of foam.
Figure 6.1-2. The first known instance of bipod ramp shedding occurred on STS-7 which was launched on June 18, 1983.
Report Volume I
Figure 6.1-3. Only three months before the final launch of Columbia, the bipod ramp foam had come off during STS-112.
August 2003
Over the course of the 113 Space Shuttle missions, the left bipod ramp has shed significant pieces of foam at least seven times. (Foam-shedding from the right bipod ramp has never been confirmed. The right bipod ramp may be less subject to foam shedding because it is partially shielded from aerodynamic forces by the External Tank's liquid oxygen line.) The fact that five of these left bipod shedding events occurred on missions flown by Columbia sparked considerable Board debate. Although initially this appeared to be a improbable coincidence that would have caused the Board to fault NASA for improper trend analysis and lack of engineering curiosity, on closer inspection, the Board concluded that this "coincidence" is probably the result of a bias in the sample of known bipod foam-shedding. Before the Challenger accident, only Challenger and Columbia carried umbilical well cameras that imaged the External Tank after separation, so there are more images of Columbia than of the other Orbiters.10 The bipod was imaged 26 of 28 of Columbia's missions; in contrast, Challenger had 7 of 10, Discovery had only 14 of 30, Atlantis only 14 of 26, and Endeavour 12 of 19.
impact damage was shallow, the tile loss was not a result of excessive aerodynamic loads, and the External Tank Thermal Protection System failure was the result of "inadequate venting."13 The External Tank Project closed out its In-Flight Anomaly with the rationale that foam loss during ascent was "not considered a flight or safety issue."14 Note the difference in how the each program addressed the foamshedding problem: While the Integration Office deemed it an "accepted risk," the External Tank Project considered it "not a safety-of-flight issue." Hazard Report 37 would figure in the STS-113 Flight Readiness Review, where the crucial decision was made to continue flying with the foam-loss problem. This inconsistency would reappear 10 years later, after bipod foam-shedding during STS-112. The fourth and fifth bipod ramp foam loss events went undetected until the Board directed NASA to review all available imagery for other instances of bipod foam-shedding. This review of imagery from tracking cameras, the umbilical well camera, and video and still images from flight crew hand held cameras revealed bipod foam loss on STS-52 and STS-62, both of which were flown by Columbia. STS-52, launched on October 22, 1992, lost an 8- by 4-inch corner of the left bipod ramp as well as portions of foam covering the left jackpad, a piece of External Tank hardware that facilitates the Orbiter attachment process. The STS-52 post-mission inspection noted a higher-than-average 290 hits on upper and lower Thermal Protection System tiles, 16 of which were greater than one inch in one dimension. External Tank separation videos of STS-62, launched on March 4, 1994, revealed that a 1- by 3-inch piece of foam in the rear face of the left bipod ramp was missing, as were small pieces of foam around the bipod ramp. Because these incidents of missing bipod foam were not detected until after the STS-107 accident, no In-Flight Anomalies had been written. The Board concludes that NASA's failure to identify these bipod foam losses at the time they occurred means the agency must examine the adequacy of its film review, post-flight inspection, and Program Requirements Control Board processes. The sixth and final bipod ramp event before STS-107 occurred during STS-112 on October 7, 2002 (see Figure 6.13). At 33 seconds after launch, when Atlantis was at 12,500 feet and traveling at Mach 0.75, ground cameras observed an object traveling from the External Tank that subsequently impacted the Solid Rocket Booster/External Tank Attachment ring (see Figure 6.1-4). After impact, the debris broke into multiple pieces that fell along the Solid Rocket Booster exhaust plume.15 Post-mission inspection of the Solid Rocket Booster confirmed damage to foam on the forward face of the External Tank Attachment ring. The impact was approximately 4 inches wide and 3 inches deep. Post-External Tank separation photography by the crew showed that a 4- by 5by 12-inch (240 cubic-inch) corner section of the left bipod ramp was missing, which exposed the super lightweight ablator coating on the bipod housing. This missing chunk of foam was believed to be the debris that impacted the External Tank Attachment ring during ascent. The post-launch review of photos and video identified these debris events, but the Mission Evaluation Room logs and Mission Management Team minutes do not reflect any discussions of them.
August 2003
The second bipod ramp foam loss occurred during STS-32R, Columbia's ninth flight, on January 9, 1990. A post-mission review of STS-32R photography revealed five divots in the intertank foam ranging from 6 to 28 inches in diameter, the largest of which extended into the left bipod ramp foam. A post-mission inspection of the lower surface of the Orbiter revealed 111 hits, 13 of which were one inch or greater in one dimension. An In-Flight Anomaly assigned to the External Tank Project was closed out at the Flight Readiness Review for the next mission, STS-36, on the basis that there may have been local voids in the foam bipod ramp where it attached to the metal skin of the External Tank. To address the foam loss, NASA engineers poked small "vent holes" through the intertank foam to allow trapped gases to escape voids in the foam where they otherwise might build up pressure and cause the foam to pop off. However, NASA is still studying this hypothesized mechanism of foam loss. Experiments conducted under the Board's purview indicate that other mechanisms may be at work. (See "Foam Fracture Under Hydrostatic Pressure" in Chapter 3.) As discussed in Chapter 3, the Board notes that the persistent uncertainty about the causes of foam loss and potential Orbiter damage results from a lack of thorough hazard analysis and engineering attention. The third bipod foam loss occurred on June 25, 1992, during the launch of Columbia on STS-50, when an approximately 26- by 10-inch piece separated from the left bipod ramp area. Post-mission inspection revealed a 9-inch by 4.5-inch by 0.5-inch divot in the tile, the largest area of tile damage in Shuttle history. The External Tank Project at Marshall Space Flight Center and the Integration Office at Johnson Space Center cited separate In-Flight Anomalies. The Integration Office closed out its In-Flight Anomaly two days before the next flight, STS-46, by deeming damage to the Thermal Protection System an "accepted flight risk."12 In Integration Hazard Report 37, the Integration Office noted that the
Report Volume I
Figure 6.1-4. On STS-112, the foam impacted the External Tank Attach ring on the Solid Rocket Booster, causing this tear in the insulation on the ring.
STS-113 Flight Readiness Review: A Pivotal Decision Because the bipod ramp shedding on STS-112 was significant, both in size and in the damage it caused, and because it occurred only two flights before STS-107, the Board investigated NASA's rationale to continue flying. This decision made by the Program Requirements Control Board at the STS-113 Flight Readiness Review is among those most directly linked to the STS-107 accident. Had the foam loss during STS-112 been classified as a more serious threat, managers might have responded differently when they heard about the foam strike on STS-107. Alternately, in the face of the increased risk, STS-107 might not have flown at all. However, at STS-113's Flight Readiness Review, managers formally accepted a flight rationale that stated it was safe to fly with foam losses. This decision enabled, and perhaps even encouraged, Mission Management Team members to use similar reasoning when evaluating whether the foam strike on STS-107 posed a safety-of-flight issue. At the Program Requirements Control Board meeting following the return of STS-112, the Intercenter Photo Working Group recommended that the loss of bipod foam be classified as an In-Flight Anomaly. In a meeting chaired by
NASA Marshall Space Flight Center, Huntsville, Alabama
Shuttle Program Manager Ron Dittemore and attended by many of the managers who would be actively involved with STS-107, including Linda Ham, the Program Requirements Control Board ultimately decided against such classification. Instead, after discussions with the Integration Office and the External Tank Project, the Program Requirements Control Board Chairman assigned an "action" to the External Tank Project to determine the root cause of the foam loss and to propose corrective action. This was inconsistent with previous practice, in which all other known bipod foam-shedding was designated as In-Flight Anomalies. The Program Requirements Control Board initially set December 5, 2002, as the date to report back on this action, even though STS-113 was scheduled to launch on November 10. The due date subsequently slipped until after the planned launch and return of STS-107. The Space Shuttle Program decided to fly not one but two missions before resolving the STS-112 foam loss. The Board wondered why NASA would treat the STS-112 foam loss differently than all others. What drove managers to reject the recommendation that the foam loss be deemed an In-Flight Anomaly? Why did they take the unprecedented step of scheduling not one but eventually two missions to fly before the External Tank Project was to report back on foam losses? It seems that Shuttle managers had become conditioned over time to not regard foam loss or debris as a safetyof-flight concern. As will be discussed in Section 6.2, the need to adhere to the Node 2 launch schedule also appears to have influenced their decision. Had the STS-113 mission been delayed beyond early December 2002, the Expedition 5 crew on board the Space Station would have exceeded its 180-day on-orbit limit, and the Node 2 launch date, a major management goal, would not be met. Even though the results of the External Tank Project engineering analysis were not due until after STS-113, the foam-shedding was reported, or "briefed," at STS-113's Flight Readiness Review on October 31, 2002, a meeting that Dittemore and Ham attended. Two slides from this brief (Figure 6.1-5) explain the disposition of bipod ramp foam loss on STS-112.
NASA Marshall Space Flight Center, Huntsville, Alabama
Space Shuttle Projects Office (MSFC)
Space Shuttle Projects Office (MSFC)
Presenter Date
STS-112/ET-115 Bipod Ramp Foam Loss
Jerry Smelser, NASA/MP31
Page 3
October 31, 2002
STS-112/ET-115 Bipod Ramp Foam Loss
Presenter Jerry Smelser, NASA/MP31 Date October 31, 2002 Page 4
• Issue
• Foam was lost on the STS-112/ET-115 –Y bipod ramp (~ 4" X 5" X 12") exposing the bipod housing SLA closeout
• Rationale for Flight
• Current bipod ramp closeout has not been changed since STS-54 (ET-51) • The Orbiter has not yet experienced "Safety of Flight" damage from loss of foam in 112 flights (including 3 known flights with bipod ramp foam loss) • There have been no design / process / equipment changes over the last 60 ETs (flights) • All ramp closeout work (including ET-115 and ET-116) was performed by experienced practitioners (all over 20 years experience each) • Ramp foam application involves craftmanship in the use of validated application processess • No change in Inspection / Process control / Post application handling, etc • Probability of loss of ramp TPS is no higher/no lower than previous flights • The ET is safe to fly with no new concerns (and no added risk)
Bipod Attach Fitting
Prior to Foam Closeout After Final Foam Trim
• Background
• ET TPS Foam loss over the life of the Shuttle Program has never been a "Safety of Flight" issue • More than 100 External Tanks have flown with only 3 documented instances of significant foam loss on a bipod ramp
Missing Foam on –Y Bipod Ramp
Figure 6.1-5. These two briefing slides are from the STS-113 Flight Readiness Review. The first and third bullets on the right-hand slide are incorrect since the design of the bipod ramp had changed several times since the flights listed on the slide.
Report Volume I August 2003
This rationale is seriously flawed. The first and third statements listed under "Rationale for Flight" are incorrect. Contrary to the chart, which was presented by Jerry Smelser, the Program Manager for the External Tank Project, the bipod ramp design had changed, as of External Tank-76. This casts doubt on the implied argument that because the design had not changed, future bipod foam events were unlikely to occur. Although the other points may be factually correct, they provide an exceptionally weak rationale for safe flight. The fact that ramp closeout work was "performed by experienced practitioners" or that "application involves craftsmanship in the use of validated application processes" in no way decreases the chances of recurrent foam loss. The statement that the "probability of loss of ramp Thermal Protection System is no higher/no lower than previous flights" could be just as accurately stated "the probability of bipod foam loss on the next flight is just as high as it was on previous flights." With no engineering analysis, Shuttle managers used past success as a justification for future flights, and made no change to the External Tank configurations planned for STS-113, and, subsequently, for STS-107. Along with this chart, the NASA Headquarters Safety Office presented a report that estimated a 99 percent probability of foam not being shed from the same area, even though no corrective action had been taken following the STS-112 foam-shedding.16 The ostensible justification for the 99 percent figure was a calculation of the actual rate of bipod loss over 61 flights. This calculation was a sleightof-hand effort to make the probability of bipod foam loss appear low rather than a serious grappling with the probability of bipod ramp foam separating. For one thing, the calculation equates the probability of left and right bipod loss, when right bipod loss has never been observed, and the amount of imagery available for left and right bipod events differs. The calculation also miscounts the actual number of bipod ramp losses in two ways. First, by restricting the sample size to flights between STS-112 and the last known bipod ramp loss, it excludes known bipod ramp losses from STS-7, STS-32R, and STS-50. Second, by failing to project the statistical rate of bipod loss across the many missions for which no bipod imagery is available, the calculation assumes a "what you don't see won't hurt you" mentality when in fact the reverse is true. When the statistical rate of bipod foam loss is projected across missions for which imagery is not available, and the sample size is extended to include every mission from STS-1 on, the probability of bipod loss increases dramatically. The Board's review after STS-107, which included the discovery of two additional bipod ramp losses that NASA had not previously noted, concluded that bipod foam loss occurred on approximately 10 percent of all missions. During the brief at STS-113's Flight Readiness Review, the Associate Administrator for Safety and Mission Assurance scrutinized the Integration Hazard Report 37 conclusion that debris-shedding was an accepted risk, as well as the External Tank Project's rationale for flight. After conferring, STS-113 Flight Readiness Review participants ultimately agreed that foam shedding should be characterized as an "accepted risk" rather than a "not a safety-of-flight" issue. Space Shuttle Program management accepted this
Report Volume I
rationale, and STS-113's Certificate of Flight Readiness was signed. The decision made at the STS-113 Flight Readiness Review seemingly acknowledged that the foam posed a threat to the Orbiter, although the continuing disagreement over whether foam was "not a safety of flight issue" versus an "accepted risk" demonstrates how the two terms became blurred over time, clouding the precise conditions under which an increase in risk would be permitted by Shuttle Program management. In retrospect, the bipod foam that caused a 4- by 3-inch gouge in the foam on one of Atlantis' Solid Rocket Boosters – just months before STS-107 – was a "strong signal" of potential future damage that Shuttle engineers ignored. Despite the significant bipod foam loss on STS-112, Shuttle Program engineers made no External Tank configuration changes, no moves to reduce the risk of bipod ramp shedding or potential damage to the Orbiter on either of the next two flights, STS-113 and STS-107, and did not update Integrated Hazard Report 37. The Board notes that although there is a process for conducting hazard analyses when the system is designed and a process for re-evaluating them when a design is changed or the component is replaced, no process addresses the need to update a hazard analysis when anomalies occur. A stronger Integration Office would likely have insisted that Integrated Hazard Analysis 37 be updated. In the course of that update, engineers would be forced to consider the cause of foam-shedding and the effects of shedding on other Shuttle elements, including the Orbiter Thermal Protection System. STS-113 launched at night, and although it is occasionally possible to image the Orbiter from light given off by the Solid Rocket Motor plume, in this instance no imagery was obtained and it is possible that foam could have been shed. The acceptance of the rationale to fly cleared the way for Columbia's launch and provided a method for Mission managers to classify the STS-107 foam strike as a maintenance and turnaround concern rather than a safety-of-flight issue. It is significant that in retrospect, several NASA managers identified their acceptance of this flight rationale as a serious error. The foam-loss issue was considered so insignificant by some Shuttle Program engineers and managers that the STS-107 Flight Readiness Review documents include no discussion of the still-unresolved STS-112 foam loss. According to Program rules, this discussion was not a requirement because the STS-112 incident was only identified as an "action," not an In-Flight Anomaly. However, because the action was still open, and the date of its resolution had slipped, the Board believes that Shuttle Program managers should have addressed it. Had the foam issue been discussed in STS-107 pre-launch meetings, Mission managers may have been more sensitive to the foam-shedding, and may have taken more aggressive steps to determine the extent of the damage. The seventh and final known bipod ramp foam loss occurred on January 16, 2003, during the launch of Columbia on STS-107. After the Columbia bipod loss, the Program Requirements Control Board deemed the foam loss an In-Flight Anomaly to be dealt with by the External Tank Project.
August 2003
Other Foam/Debris Events To better understand how NASA's treatment of debris strikes evolved over time, the Board investigated missions where debris was shed from locations other than the External Tank bipod ramp. The number of debris strikes to the Orbiters' lower surface Thermal Protection System that resulted in tile damage greater than one inch in diameter is shown in Figure 6.1-6.17 The number of debris strikes may be small, but a single strike could damage several tiles (see Figure 6.1-7). One debris strike in particular foreshadows the STS-107 event. When Atlantis was launched on STS-27R on December 2, 1988, the largest debris event up to that time significantly damaged the Orbiter. Post-launch analysis of tracking camera imagery by the Intercenter Photo Working Group identified a large piece of debris that struck the Thermal Protection System tile at approximately 85 seconds into the flight. On Flight Day Two, Mission Control asked the flight crew to inspect Atlantis with a camera mounted on the remote manipulator arm, a robotic device that was not installed on Columbia for STS-107. Mission Commander R.L. "Hoot" Gibson later stated that Atlantis "looked like it had been blasted by a shotgun."18 Concerned that the Orbiter's Thermal Protection System had been breached, Gibson ordered that the video be transferred to Mission Control so that NASA engineers could evaluate the damage. When Atlantis landed, engineers were surprised by the extent of the damage. Post-mission inspections deemed it "the most severe of any mission yet flown."19 The Orbiter had 707 dings, 298 of which were greater than an inch in one dimension. Damage was concentrated outboard of a line right of the bipod attachment to the liquid oxygen umbilical line. Even more worrisome, the debris had knocked off a tile, exposing the Orbiter's skin to the heat of re-entry. Post-flight analysis concluded that structural damage was confined to the exposed cavity left by the missing tile, which happened to be at the location of a thick aluminum plate covering an L-band navigation antenna. Were it not for the thick alumiLower surface damage dings >1 inch diameter
num plate, Gibson stated during a presentation to the Board that a burn-through may have occurred.20 The Board notes the distinctly different ways in which the STS-27R and STS-107 debris strike events were treated. After the discovery of the debris strike on Flight Day Two of STS-27R, the crew was immediately directed to inspect the vehicle. More severe thermal damage – perhaps even a burn-through – may have occurred were it not for the aluminum plate at the site of the tile loss. Fourteen years later, when a debris strike was discovered on Flight Day Two of STS-107, Shuttle Program management declined to have the crew inspect the Orbiter for damage, declined to request onorbit imaging, and ultimately discounted the possibility of a burn-through. In retrospect, the debris strike on STS-27R is a "strong signal" of the threat debris posed that should have been considered by Shuttle management when STS-107 suffered a similar debris strike. The Board views the failure to do so as an illustration of the lack of institutional memory in the Space Shuttle Program that supports the Board's claim, discussed in Chapter 7, that NASA is not functioning as a learning organization. After the STS-27R damage was evaluated during a postflight inspection, the Program Requirements Control Board assigned In-Flight Anomalies to the Orbiter and Solid Rocket Booster Projects. Marshall Sprayable Ablator (MSA-1) material found embedded in an insulation blanket on the right Orbital Maneuvering System pod confirmed that the ablator on the right Solid Rocket Booster nose cap was the most likely source of debris.21 Because an improved ablator material (MSA-2) would now be used on the Solid Rocket Booster nose cap, the issue was considered "closed" by the time of the next mission's Flight Readiness Review. The Orbiter Thermal Protection System review team concurred with the use of the improved ablator without reservation. An STS-27R investigation team notation mirrors a Columbia Accident Investigation Board finding. The STS-27R investigation noted: "it is observed that program emphasis
Bipod Ramp Foam Loss Event
STS-26R OV-103, Flight 7
STS-17 OV-099, Flight 6
STS-27R OV-104, Flight 3 Cause: SRB Ablative
STS-73 OV-102, Flight 18
STS-87 OV-102, Flight 24 Cause: ET Intertank Foam
Space Shuttle Mission Number
Figure 6.1-6. This chart shows the number of dings greater than one inch in diameter on the lower surface of the Orbiter after each mission from STS-6 through STS-113. Flights where the bipod ramp foam is known to have come off are marked with a red triangle.
Report Volume I August 2003
DATE April 12, 1981 June 18, 1983 December 2, 1988 January 9, 1990 December 2, 1990 January 22, 1992 March 24, 1992 June 25, 1992 October 22, 1992 April 8, 1993 October 4, 1994 November 19, 1997
COMMENTS Lots of debris damage. 300 tiles replaced. First known left bipod ramp foam shedding event. Debris knocks off tile; structural damage and near burn through results. Second known left bipod ramp foam event. First time NASA calls foam debris "safety of flight issue," and "re-use or turnaround issue." First mission after which the next mission (STS-45) launched without debris InFlight Anomaly closure/resolution. Damage to wing RCC Panel 10-right. Unexplained Anomaly, "most likely orbital debris." Third known bipod ramp foam event. Hazard Report 37: an "accepted risk." Undetected bipod ramp foam loss (Fourth bipod event). Acreage tile damage (large area). Called "within experience base" and considered "in family." Undetected bipod ramp foam loss (Fifth bipod event). Damage to Orbiter Thermal Protection System spurs NASA to begin 9 flight tests to resolve foam-shedding. Foam fix ineffective. In-Flight Anomaly eventually closed after STS-101 as "accepted risk." Sixth known left bipod ramp foam loss. First time major debris event not assigned an In-Flight Anomaly. External Tank Project was assigned an Action. Not closed out until after STS-113 and STS-107. Columbia launch. Seventh known left bipod ramp foam loss event.
October 7, 2002 January 16, 2003
Figure 6.1-7. The Board identified 14 flights that had significant Thermal Protection System damage or major foam loss. Two of the bipod foam loss events had not been detected by NASA prior to the Columbia Accident Investigation Board requesting a review of all launch images.
and attention to tile damage assessments varies with severity and that detailed records could be augmented to ease trend maintenance" (emphasis added).22 In other words, Space Shuttle Program personnel knew that the monitoring of tile damage was inadequate and that clear trends could be more readily identified if monitoring was improved, but no such improvements were made. The Board also noted that an STS-27R investigation team recommendation correlated to the Columbia accident 14 years later: "It is recommended that the program actively solicit design improvements directed toward eliminating debris sources or minimizing damage potential."23 Another instance of non-bipod foam damage occurred on STS-35. Post-flight inspections of Columbia after STS-35 in December 1990, showed a higher-than-average amount of damage on the Orbiter's lower surface. A review of External Tank separation film revealed approximately 10 areas of missing foam on the flange connecting the liquid hydrogen
Report Volume I
tank to the intertank. An In-Flight Anomaly was assigned to the External Tank Project, which closed it by stating that there was no increase in Orbiter Thermal Protection System damage and that it was "not a safety-of-flight concern."24 The Board notes that it was in a discussion at the STS-36 Flight Readiness Review that NASA first identified this problem as a turnaround issue.25 Per established procedures, NASA was still designating foam-loss events as In-Flight Anomalies and continued to make various corrective actions, such as drilling more vent holes and improving the foam application process. Discovery was launched on STS-42 on January 22, 1992. A total of 159 hits on the Orbiter Thermal Protection System were noted after landing. Two 8- to 12-inch-diameter divots in the External Tank intertank area were noted during post-External Tank separation photo evaluation, and these pieces of foam were identified as the most probable sources of the damage. The External Tank Project was assigned an
August 2003
In-Flight Anomaly, and the incident was later described as an unexplained or isolated event. However, at later Flight Readiness Reviews, the Marshall Space Flight Center briefed this as being "not a safety-of-flight" concern.26 The next flight, STS-45, would be the first mission launched before the foam-loss In-Flight Anomaly was closed. On March 24, 1992, Atlantis was launched on STS-45. Post-mission inspection revealed exposed substrate on the upper surface of right wing leading edge Reinforced Carbon-Carbon (RCC) panel 10 caused by two gouges, one 1.9 inches by 1.6 inches and the other 0.4 inches by 1 inch.27 Before the next flight, an In-Flight Anomaly assigned to the Orbiter Project was closed as "unexplained," but "most likely orbital debris."28 Despite this closure, the Safety and Mission Assurance Office expressed concern as late as the pre-launch Mission Management Team meeting two days before the launch of STS-49. Nevertheless, the mission was cleared for launch. Later laboratory tests identified pieces of man-made debris lodged in the RCC, including stainless steel, aluminum, and titanium, but no conclusion was made about the source of the debris. (The Board notes that this indicates there were transport mechanisms available to determine the path the debris took to impact the wing leading edge. See Section 3.4.) The Program Requirements Control Board also assigned the External Tank Project an In-Flight Anomaly after foam loss on STS-56 (Discovery) and STS-58 (Columbia), both of which were launched in 1993. These missions demonstrate the increasingly casual ways in which debris impacts were dispositioned by Shuttle Program managers. After postflight analysis determined that on both missions the foam had come from the intertank and bipod jackpad areas, the rationale for closing the In-Flight Anomalies included notations that the External Tank foam debris was "in-family," or within the experience base.29 During the launch of STS-87 (Columbia) on November 19, 1997, a debris event focused NASA's attention on debrisshedding and damage to the Orbiter. Post-External Tank separation photography revealed a significant loss of material from both thrust panels, which are fastened to the Solid Rocket Booster forward attachment points on the intertank structure. Post-landing inspection of the Orbiter noted 308 hits, with 244 on the lower surface and 109 larger than an inch. The foam loss from the External Tank thrust panels was suspected as the most probable cause of the Orbiter Thermal Protection System damage. Based on data from post-flight inspection reports, as well as comparisons with statistics from 71 similarly configured flights, the total number of damage sites, and the number of damage sites one inch or larger, were considered "out-of-family."30 An investigation was conducted to determine the cause of the material loss and the actions required to prevent a recurrence. The foam loss problem on STS-87 was described as "popcorning" because of the numerous popcorn-size foam particles that came off the thrust panels. Popcorning has always occurred, but it began earlier than usual in the launch of STS-87. The cause of the earlier-than-normal popcorning (but not the fundamental cause of popcorning) was traced
Report Volume I
back to a change in foam-blowing agents that caused pressure buildups and stress concentrations within the foam. In an effort to reduce its use of chlorofluorocarbons (CFCs), NASA had switched from a CFC-11 (chlorofluorocarbon) blowing agent to an HCFC-141b blowing agent beginning with External Tank-85, which was assigned to STS-84. (The change in blowing agent affected only mechanically applied foam. Foam that is hand sprayed, such as on the bipod ramp, is still applied using CFC-11.) The Program Requirements Control Board issued a Directive and the External Tank Project was assigned an In-Flight Anomaly to address the intertank thrust panel foam loss. Over the course of nine missions, the External Tank Project first reduced the thickness of the foam on the thrust panels to minimize the amount of foam that could be shed; and, due to a misunderstanding of what caused foam loss at that time, put vent holes in the thrust panel foam to relieve trapped gas pressure. The In-Flight Anomaly remained open during these changes, and foam shedding occurred on the nine missions that tested the corrective actions. Following STS-101, the 10th mission after STS-87, the Program Requirements Control Board concluded that foam-shedding from the thrust panel had been reduced to an "acceptable level" by sanding and venting, and the In-Flight Anomaly was closed.31 The Orbiter Project, External Tank Project, and Space Shuttle Program management all accepted this rationale without question. The Board notes that these interventions merely reduced foam-shedding to previously experienced levels, which have remained relatively constant over the Shuttle's lifetime. Making the Orbiter More Resistant To Debris Strikes If foam shedding could not be prevented entirely, what did NASA do to make the Thermal Protection System more resistant to debris strikes? A 1990 study by Dr. Elisabeth Paté-Cornell and Paul Fishback attempted to quantify the risk of a Thermal Protection System failure using probabilistic analysis.32 The data they used included (1) the probability that a tile would become debonded by either debris strikes or a poor bond, (2) the probability of then losing adjacent tiles, (3) depending on the final size of the failed area, the probability of burn-through, and (4) the probability of failure of a critical sub-system if burn-through occurs. The study concluded that the probability of losing an Orbiter on any given mission due to a failure of Thermal Protection System tiles was approximately one in 1,000. Debris-related problems accounted for approximately 40 percent of the probability, while 60 percent was attributable to tile debonding caused by other factors. An estimated 85 percent of the risk could be attributed to 15 percent of the "acreage," or larger areas of tile, meaning that the loss of any one of a relatively small number of tiles pose a relatively large amount of risk to the Orbiter. In other words, not all tiles are equal – losing certain tiles is more dangerous. While the actual risk may be different than that computed in the 1990 study due to the limited amount of data and the underlying simplified assumptions, this type of analysis offers insight that enables management to concentrate their resources on protecting the Orbiters' critical areas.
August 2003
Two years after the conclusion of that study, NASA wrote to Paté-Cornell and Fishback describing the importance of their work, and stated that it was developing a longterm effort to use probabilistic risk assessment and related disciplines to improve programmatic decisions.33 Though NASA has taken some measures to invest in probabilistic risk assessment as a tool, it is the Board's view that NASA has not fully exploited the insights that Paté-Cornell's and Fishback's work offered.34 Impact Resistant Tile NASA also evaluated the possibility of increasing Thermal Protection System tile resistance to debris hits, lowering the possibility of tile debonding, and reducing tile production and maintenance costs.35 Indeed, tiles with a "tough" coating are currently used on the Orbiters. This coating, known as Toughened Uni-piece Fibrous Insulation (TUFI), was patented in 1992 and developed for use on high-temperature rigid insulation.36 TUFI is used on a tile material known as Alumina Enhanced Thermal Barrier (AETB), and has a debris impact resistance that is greater than the current acreage tile's resistance by a factor of approximately 6-20.37 At least 772 of these advanced tiles have been installed on the Orbiters' base heat shields and upper body flaps.38 However, due to its higher thermal conductivity, TUFI-coated AETB cannot be used as a replacement for the larger areas of tile coverage. (Boeing, Lockheed Martin and NASA are developing a lightweight, impact-resistant, low-conductivity tile.39) Because the impact requirements for these next-generation tiles do not appear to be based on resistance to specific (and probable) damage sources, it is the Board's view that certification of the new tile will not adequately address the threat posed by debris. Conclusion Despite original design requirements that the External Tank not shed debris, and the corresponding design requirement that the Orbiter not receive debris hits exceeding a trivial amount of force, debris has impacted the Shuttle on each flight. Over the course of 113 missions, foam-shedding and other debris impacts came to be regarded more as a turnaround or maintenance issue, and less as a hazard to the vehicle and crew. Assessments of foam-shedding and strikes were not thoroughly substantiated by engineering analysis, and the process for closing In-Flight Anomalies is not well-documented and appears to vary. Shuttle Program managers appear to have confused the notion of foam posing an "accepted risk" with foam not being a "safety-of-flight issue." At times, the pressure to meet the flight schedule appeared to cut short engineering efforts to resolve the foam-shedding problem. NASA's lack of understanding of foam properties and behavior must also be questioned. Although tests were conducted to develop and qualify foam for use on the External Tank, it appears there were large gaps in NASA's knowledge about this complex and variable material. Recent testing conducted at Marshall Space Flight Center and under the auspices of the Board indicate that mechanisms previously
Report Volume I
considered a prime source of foam loss, cryopumping and cryoingestion, are not feasible in the conditions experienced during tanking, launch, and ascent. Also, dissections of foam bipod ramps on External Tanks yet to be launched reveal subsurface flaws and defects that only now are being discovered and identified as contributing to the loss of foam from the bipod ramps. While NASA properly designated key debris events as InFlight Anomalies in the past, more recent events indicate that NASA engineers and management did not appreciate the scope, or lack of scope, of the Hazard Reports involving foam shedding.40 Ultimately, NASA's hazard analyses, which were based on reducing or eliminating foam-shedding, were not succeeding. Shuttle Program management made no adjustments to the analyses to recognize this fact. The acceptance of events that are not supposed to happen has been described by sociologist Diane Vaughan as the "normalization of deviance."41 The history of foam-problem decisions shows how NASA first began and then continued flying with foam losses, so that flying with these deviations from design specifications was viewed as normal and acceptable. Dr. Richard Feynman, a member of the Presidential Commission on the Space Shuttle Challenger Accident, discusses this phenomena in the context of the Challenger accident. The parallels are striking: The phenomenon of accepting … flight seals that had shown erosion and blow-by in previous flights is very clear. The Challenger flight is an excellent example. There are several references to flights that had gone before. The acceptance and success of these flights is taken as evidence of safety. But erosions and blow-by are not what the design expected. They are warnings that something is wrong … The O-rings of the Solid Rocket Boosters were not designed to erode. Erosion was a clue that something was wrong. Erosion was not something from which safety can be inferred … If a reasonable launch schedule is to be maintained, engineering often cannot be done fast enough to keep up with the expectations of originally conservative certification criteria designed to guarantee a very safe vehicle. In these situations, subtly, and often with apparently logical arguments, the criteria are altered so that flights may still be certified in time. They therefore fly in a relatively unsafe condition, with a chance of failure of the order of a percent (it is difficult to be more accurate).42 Findings F6.1−1 NASA has not followed its own rules and requirements on foam-shedding. Although the agency continuously worked on the foam-shedding problem, the debris impact requirements have not been met on any mission. Foam-shedding, which had initially raised serious safety concerns, evolved into "in-family" or "no safety-of-flight" events or were deemed an "accepted risk." Five of the seven bipod ramp events occurred on missions flown by Columbia, a seemingly high number. This observation is likely due to
August 2003
Columbia having been equipped with umbilical cameras earlier than other Orbiters. There is lack of effective processes for feedback or integration among project elements in the resolution of In-Flight Anomalies. Foam bipod debris-shedding incidents on STS-52 and STS-62 were undetected at the time they occurred, and were not discovered until the Board directed NASA to examine External Tank separation images more closely. Foam bipod debris-shedding events were classified as In-Flight Anomalies up until STS-112, which was the first known bipod foam-shedding event not classified as an In-Flight Anomaly. The STS-112 assignment for the External Tank Project to "identify the cause and corrective action of the bipod ramp foam loss event" was not due until after the planned launch of STS-113, and then slipped to after the launch of STS-107. No External Tank configuration changes were made after the bipod foam loss on STS-112. Although it is sometimes possible to obtain imagery of night launches because of light provided by the Solid Rocket Motor plume, no imagery was obtained for STS-113. NASA failed to adequately perform trend analysis on foam losses. This greatly hampered the agency's ability to make informed decisions about foam losses. Despite the constant shedding of foam, the Shuttle Program did little to harden the Orbiter against foam impacts through upgrades to the Thermal Protection System. Without impact resistance and strength requirements that are calibrated to the energy of debris likely to impact the Orbiter, certification of new Thermal Protection System tile will not adequately address the threat posed by debris.
that the complexity and political mandates surrounding the International Space Station Program, as well as Shuttle Program management's responses to them, resulted in pressure to meet an increasingly ambitious launch schedule. In mid-2001, NASA adopted plans to make the over-budget and behind-schedule International Space Station credible to the White House and Congress. The Space Station Program and NASA were on probation, and had to prove they could meet schedules and budgets. The plan to regain credibility focused on the February 19, 2004, date for the launch of Node 2 and the resultant Core Complete status. If this goal was not met, NASA would risk losing support from the White House and Congress for subsequent Space Station growth. By the late summer of 2002, a variety of problems caused Space Station assembly work and Shuttle flights to slip beyond their target dates. With the Node 2 launch endpoint fixed, these delays caused the schedule to become ever more compressed. Meeting U.S. Core Complete by February 19, 2004, would require preparing and launching 10 flights in less than 16 months. With the focus on retaining support for the Space Station program, little attention was paid to the effects the aggressive Node 2 launch date would have on the Shuttle Program. After years of downsizing and budget cuts (Chapter 5), this mandate and events in the months leading up to STS107 introduced elements of risk to the Program. Columbia and the STS-107 crew, who had seen numerous launch slips due to missions that were deemed higher priorities, were further affected by the mandatory Core Complete date. The high-pressure environments created by NASA Headquarters unquestionably affected Columbia, even though it was not flying to the International Space Station. February 19, 2004 – "A Line in the Sand" Schedules are essential tools that help large organizations effectively manage their resources. Aggressive schedules by themselves are often a sign of a healthy institution. However, other institutional goals, such as safety, sometimes compete with schedules, so the effects of schedule pressure in an organization must be carefully monitored. The Board posed the question: Was there undue pressure to nail the Node 2 launch date to the February 19, 2004, signpost? The management and workforce of the Shuttle and Space Station programs each answered the question differently. Various members of NASA upper management gave a definite "no." In contrast, the workforce within both programs thought there was considerable management focus on Node 2 and resulting pressure to hold firm to that launch date, and individuals were becoming concerned that safety might be compromised. The weight of evidence supports the workforce view. Employees attributed the Node 2 launch date to the new Administrator, Sean O'Keefe, who was appointed to execute a Space Station management plan he had proposed as Deputy Director of the White House Office of Management and Budget. They understood the scrutiny that NASA, the new Administrator, and the Space Station Program were under,
August 2003
Recommendations: • None
Countdown to Space Station "Core Complete:" A Workforce Under Pressure During the course of this investigation, the Board received several unsolicited comments from NASA personnel regarding pressure to meet a schedule. These comments all concerned a date, more than a year after the launch of Columbia, that seemed etched in stone: February 19, 2004, the scheduled launch date of STS-120. This flight was a milestone in the minds of NASA management since it would carry a section of the International Space Station called "Node 2." This would configure the International Space Station to its "U.S. Core Complete" status. At first glance, the Core Complete configuration date seemed noteworthy but unrelated to the Columbia accident. However, as the investigation continued, it became apparent
Report Volume I
but now it seemed to some that budget and schedule were of paramount concern. As one employee reflected: I guess my frustration was … I know the importance of showing that you … manage your budget and that's an important impression to make to Congress so you can continue the future of the agency, but to a lot of people, February 19th just seemed like an arbitrary date … It doesn't make sense to me why at all costs we were marching to this date. The importance of this date was stressed from the very top. The Space Shuttle and Space Station Program Managers briefed the new NASA Administrator monthly on the status of their programs, and a significant part of those briefings was the days of margin remaining in the schedule to the launch of Node 2 – still well over a year away. The Node 2 schedule margin typically accounted for more than half of the briefing slides. Figure 6.2-1 is one of the charts presented by the Shuttle Program Manager to the NASA Administrator in December 2002. The chart shows how the days of margin in the existing schedule were being managed to meet the requirement
of a Node 2 launch on the prescribed date. The triangles are events that affected the schedule (such as the slip of a Russian Soyuz flight). The squares indicate action taken by management to regain the lost time (such as authorizing work over the 2002 winter holidays). Figure 6.2-2 shows a slide from the International Space Station Program Manager's portion of the briefing. It indicates that International Space Station Program management was also taking actions to regain margin. Over the months, the extent of some testing at Kennedy was reduced, the number of tasks done in parallel was increased, and a third shift of workers would be added in 2003 to accomplish the processing. These charts illustrate that both the Space Shuttle and Space Station Programs were being managed to a particular launch date – February 19, 2004. Days of margin in that schedule were one of the principle metrics by which both programs came to be judged. NASA Headquarters stressed the importance of this date in other ways. A screen saver (see Figure 6.2-3) was mailed to managers in NASA's human spaceflight program that depicted a clock counting down to February 19, 2004 – U.S. Core Complete.
SSP Schedule Reserve
SSP Core Complete
1 35
Schedule Margin - Past
1 OV-103) 2 Moved Node2 to OV-105
Late OMM start (Node 2 was on
Margin (in months)
3 Accommodate 4S slip 1 week
-14 -1
-21 10
4 ISS adding wrist joint on UF2
5 Moved OV-104 Str. Ins. to 9th flt
6 Engine Flowliner cracks
Reduced Structural Inspection Requirements
8 Accommodate 4S slip
9 O2 flexline leak/ SRMS damage
10 Defer reqmts; apply reserve
12.01 03.02 06.02 09.02 12.02
Management action Schedule impact event
SSP Core Complete Schedule Threats
STS-120/Node 2 launch subject to 45 days of schedule risk • HQ mitigate Range Cutout • HQ and ISS mitigate Soyuz • HQ mitigate Range Cutout • HQ and ISS mitigate Soyuz • HQ and ISS mitigate Soyuz
Management Options
• USA commit holiday/weekend reserves and apply additional resources (i.e., 3rd shift) to hold schedule (Note: 3rd shift not yet included) • HQ mitigate Range Cutout • HQ and ISS mitigate Soyuz conflict threat
Figure 6.2-1. This chart was presented by the Space Shuttle Program Manager to the NASA Administrator in December 2002. It illustrates how the schedule was being managed to meet the Node 2 launch date of February 19, 2004.
Report Volume I
August 2003
While employees found this amusing because they saw it as a date that could not be met, it also reinforced the message that NASA Headquarters was focused on and promoting the achievement of that date. This schedule was on the minds of the Shuttle managers in the months leading up to STS-107. The Background: Schedule Complexity and Compression In 2001, the International Space Station Cost and Management Evaluation Task Force report recommended, as a cost-saving measure, a limit of four Shuttle flights to the International Space Station per year. To meet this requirement, managers began adjusting the Shuttle and Station manifests to "get back in the budget box." They rearranged Station assembly sequences, moving some elements forward and taking others out. When all was said and done, the launch of STS-120, which would carry Node 2 to the International Space Station, fell on February 19, 2004. The Core Complete date simply emerged from this planning effort in 2001. By all accounts, it was a realistic and achievable date when first approved. At the time there was more concern that four Shuttle flights a year would limit the
Figure 6.2-3. NASA Headquarters distributed to NASA employees this computer screensaver counting down to February 19, 2004.
capability to carry supplies to and from the Space Station, to rotate its crew, and to transport remaining Space Station segments and equipment. Still, managers felt it was a rea-
ISS Schedule Reserve
ISS Core Complete Schedule Margin - Past
45 days
Time Now
Schedule margin decreased 0.75 month due to KSC Systems Test growth and closeouts growth
Margin (in months)
1.75 months slip to on dock (O/D) at KSC. Alenia build and subcontractor problems
Reduced KSC Systems Test Preps/Site Activation and Systems Test scope
22.5 days
22.5 days
0 days
0 days
3 months slip to O/D at KSC. Alenia assembly and financial problems
-30 days
-37.5 days
Reduced scope and testing; worked KSC tasks in parallel (e.g.: Closeouts & Leak Checks)
1.25 months slip to O/D at KSC Alenia work planning inefficiencies
-67.5 days
-3.0 6/01 9/01 2/02 4/02 11/02
Increased the number of KSC tasks in parallel, and adjusted powered-on testing to 3 shifts/day/5days/week
As of Date Schedule Time
ISS Core Complete Schedule Threat
• O/D KSC date will likely slip another 2 months • Alenia financial concerns • KSC test problems • Node ships on time but work or paper is not complete 0-1 month impact • Traveled work "as-built" reconciliation • Paper closure
ISS Management Options
• Hold ASI to delivery schedule • Management discussions with ASI and ESA • Reduce testing scope • Add Resources/Shifts/Weekends@KSC (Lose contingency on Node)
Figure 6.2-2. At the same December 2002 meeting, the International Space Station Program Manager presented this slide, showing the actions being taken to regain margin in the schedule. Note that the yellow triangles reflect zero days remaining margin.
Report Volume I August 2003
sonable goal and assumed that if circumstances warranted a slip of that date, it would be granted. Shuttle and Station managers worked diligently to meet the schedule. Events gradually ate away at the schedule margin. Unlike the "old days" before the Station, the Station/Shuttle partnership created problems that had a ripple effect on both programs' manifests. As one employee described it, "the serial nature" of having to fly Space Station assembly missions in a specific order made staying on schedule more challenging. Before the Space Station, if a Shuttle flight had to slip, it would; other missions that had originally followed it would be launched in the meantime. Missions could be flown in any sequence. Now the manifests were a delicate balancing act. Missions had to be flown in a certain order and were constrained by the availability of the launch site, the Russian Soyuz and Progress schedules, and a myriad of other processes. As a result, employees stated they were now experiencing a new kind of pressure. Any necessary change they made on one mission was now impacting future launch dates. They had a sense of being "under the gun." Shuttle and Station program personnel ended up with manifests that one employee described as "changing, changing, changing" all the time. One of the biggest issues they faced entering 2002 was "up mass," the amount of cargo the Shuttle can carry to the Station. Up mass was not a new problem, but when the Shuttle flight rate was reduced to four per year, up mass became critical. Working groups were actively evaluating options in the summer of 2002 and bartering to get each flight to function as expected. Sometimes the up mass being traded was actual Space Station crew members. A crew rotation planned for STS-118 was moved to a later flight because STS-118 was needed for other cargo. This resulted in an increase of crew duration on the Space Station, which was creeping past the 180-day limit agreed to by the astronaut office, flight surgeons, and Space Station international partners. A space station worker described how this one change created many other problems, and added: "… we had a train wreck coming …" Future onorbit crew time was being projected at 205 days or longer to maintain the assembly sequence and meet the schedule. By July 2002, the Shuttle and Space Station Programs were facing a schedule with very little margin. Two setbacks occurred when technical problems were found during routine maintenance on Discovery. STS-107 was four weeks away from launch at the time, but the problems grounded the entire Shuttle fleet. The longer the fleet was grounded, the more schedule margin was lost, which further compounded the complexity of the intertwined Shuttle and Station schedules. As one worker described the situation: … a one-week hit on a particular launch can start a steam roll effect including all [the] constraints and by the time you get out of here, that one-week slip has turned into a couple of months. In August 2002, the Shuttle Program realized it would be unable to meet the Space Station schedule with the available Shuttles. Columbia had never been outfitted to make
Report Volume I
a Space Station flight, so the other three Orbiters flew the Station missions. But Discovery was in its Orbiter Maintenance Down Period, and would not be available for another 17 months. All Space Station flights until then would have to be made by Atlantis and Endeavour. As managers looked ahead to 2003, they saw that after STS-107, these two Orbiters would have to alternate flying five consecutive missions, STS-114 through STS-118. To alleviate this pressure, and regain schedule margin, Shuttle Program managers elected to modify Columbia to enable it to fly Space Station missions. Those modifications were to take place immediately after STS-107 so that Columbia would be ready to fly its first Space Station mission eight months later. This decision put Columbia directly in the path of Core Complete. As the autumn of 2002 began, both the Space Shuttle and Space Station Programs began to use what some employees termed "tricks" to regain schedule margin. Employees expressed concern that their ability to gain schedule margin using existing measures was waning. In September 2002, it was clear to Space Shuttle and Space Station Program managers that they were not going to meet the schedule as it was laid out. The two Programs proposed a new set of launch dates, documented in an e-mail (right) that included moving STS-120, the Node 2 flight, to mid-March 2004. (Note that the first paragraph ends with "… the 10A [U.S. Core Complete, Node 2] launch remains 2/19/04.") These launch date changes made it possible to meet the early part of the schedule, but compressed the late 2003/ early 2004 schedule even further. This did not make sense to many in the program. One described the system as at "an uncomfortable point," noted having to go to great lengths to reduce vehicle-processing time at Kennedy, and added: … I don't know what Congress communicated to O'Keefe. I don't really understand the criticality of February 19th, that if we didn't make that date, did that mean the end of NASA? I don't know … I would like to think that the technical issues and safely resolving the technical issues can take priority over any budget issue or scheduling issue. When the Shuttle fleet was cleared to return to flight, attention turned to STS-112, STS-113, and STS-107, set for October, November, and January. Workers were uncomfortable with the rapid sequence of flights. The thing that was beginning to concern me … is I wasn't convinced that people were being given enough time to work the problems correctly. The problems that had grounded the fleet had been handled well, but the program nevertheless lost the rest of its margin. As the pressure to keep to the Node 2 schedule continued, some were concerned that this might influence the future handling of problems. One worker expressed the concern: … and I have to think that subconsciously that even though you don't want it to affect decision-making, it probably does.
August 2003
-----Original Message----From: THOMAS, DAWN A. (JSC-OC) (NASA) Sent: Friday, September 20, 2002 7:10 PM To: 'Flowers, David'; 'Horvath, Greg'; 'O'Fallon, Lee'; 'Van Scyoc, Neal'; 'Gouti, Tom'; 'Hagen, Ray'; 'Kennedy, John'; 'Thornburg, Richard'; 'Gari, Judy'; 'Dodds, Joel'; 'Janes, Lou Ann'; 'Breen, Brian'; 'Deheck-Stokes, Kristina'; 'Narita, Kaneaki (NASDA)'; 'Patrick, Penny O'; 'Michael Rasmussen (E-mail)'; DL FPWG; 'Hughes, Michael G'; 'Bennett, Patty'; 'Masazumi, Miyake'; 'Mayumi Matsuura'; NORIEGA, CARLOS I. (JSC-CB) (NASA); BARCLAY, DINA E. (JSC-DX) (NASA); MEARS, AARON (JSC-XA) (HS); BROWN, WILLIAM C. (JSC-DT) (NASA); DUMESNIL, DEANNA T. (JSC-OC) (USA); MOORE, NATHAN (JSC-REMOTE); MONTALBANO, JOEL R. (JSC-DA8) (NASA); MOORE, PATRICIA (PATTI) (JSC-DA8) (NASA); SANCHEZ, HUMBERTO (JSC-DA8) (NASA) Subject: FPWG status - 9/20/02 OA/MA mgrs mtg results
The ISS and SSP Program Managers have agreed to proceed with the crew rotation change and the following date changes: 12A launch to 5/23/03, 12A.1 launch to 7/24/03, 13A launch to 10/2/03, and 13A.1 launch to NET 11/13/03. Please note that 10A launch remains 2/19/04. The ISS SSCN that requests evaluation of these changes will be released Monday morning after the NASA/Russian bilateral Requirements and Increment Planning videocon. It will contain the following: • Increments 8 and 9 redefinition - this includes baseline of ULF2 into the tactical timeframe as the new return flight for Expedition 9 • Crew size changes for 7S, 13A.1, 15A, and 10A • Shuttle date changes as listed above • Russian date changes for CY2003 that were removed from SSCN 6872 (11P launch/10P undock and subsequent) • CY2004 Russian data if available Monday morning • Duration changes for 12A and 15A • Docking altitude update for 10A, along with "NET" TBR closure. The evaluation due date is 10/2/02. Board/meeting dates are as follows: MIOCB status - 10/3/02; comment dispositioning - 10/3/02 FPWG (meeting date/time under review); OA/MA Program Managers status - 10/4/02; SSPCB and JPRCB - 10/8/02; MMIOCB status (under review) and SSCB - 10/10/02. The 13A.1 date is indicated as "NET" (No Earlier Than) since SSP ability to meet that launch date is under review due to the processing flow requirements. There is no longer a backup option to move ULF2 to OV-105: due to vehicle processing requirements, there is no launch opportunity on OV-105 past May 2004 until after OMM. The Program Managers have asked for preparation of a backup plan in case of a schedule slip of ULF2. In order to accomplish this, the projected ISS upmass capability shortfall will be calculated as if ULF2 launch were 10/7/04, and a recommendation made for addressing the resulting shortfall and increment durations. Some methods to be assessed: manifest restructuring, fallback moves of rotation flight launch dates, LON (Launch on Need) flight on 4/29/04.
[ISS=International Space Station, SSP=Space Shuttle Program, NET=no earlier than, SSCN=Space Station Change Notice, CY=Calendar Year, TBR=To Be Revised (or Reviewed), MIOCB=Mission Integration and Operations Control Board, FPWG=Flight Planning Working Group, OA/MA=Space Station Office Symbol/Shuttle Program Office Symbol, SSPCB=Space Station Program Control Board, JPRCB=Space Shuttle/Space Station Joint Program Requirements Control Board, MMIOCB=Multi-Lateral Mission Integration and Operations Control Board, SSCB=Space Station Control Board, ULF2=U.S. Logistics Flight 2, OMM=Orbiter Major Modification, OV-105=Endeavour]
This was the environment for October and November of 2002. During this time, a bipod foam event occurred on STS112. For the first time in the history of the Shuttle Program, the Program Requirements Control Board chose to classify that bipod foam loss as an "action" rather than a more serious In-Flight Anomaly. At the STS-113 Flight Readiness Review, managers accepted with little question the rationale that it was safe to fly with the known foam problem.
Report Volume I
The Operations Tempo Following STS-107 After STS-107, the tempo was only going to increase. The vehicle processing schedules, training schedules, and mission control flight staffing assignments were all overburdened. The vehicle-processing schedule for flights from February 2003, through February 2004, was optimistic. The schedule
August 2003
could not be met with only two shifts of workers per day. In late 2002, NASA Headquarters approved plans to hire a third shift. There were four Shuttle launches to the Space Station scheduled in the five months from October 2003, through the launch of Node 2 in February 2004. To put this in perspective, the launch rate in 1985, for which NASA was criticized by the Rogers Commission, was nine flights in 12 months – and that was accomplished with four Orbiters and a manifest that was not complicated by Space Station assembly. Endeavour was the Orbiter on the critical path. Figure 6.2-4 shows the schedule margin for STS-115, STS-117, and STS-120 (Node 2). To preserve the margin going into 2003, the vehicle processing team would be required to work the late 2002-early 2003 winter holidays. The third shift of workers at Kennedy would be available in March 2003, and would buy eight more days of margin for STS-117 and STS-120. The workforce would likely have to work on the 2003 winter holidays to meet the Node 2 date. Figure 6.2-5 shows the margin for each vehicle (Discovery, OV-103, was in extended maintenance). The large boxes indicate the "margin to critical path" (to Node 2 launch date). The three smaller boxes underneath indicate (from
left to right) vehicle processing margin, holiday margin, and Dryden margin. The vehicle processing margin indicates how many days there are in addition to the days required for that mission's vehicle processing. Endeavour (OV-105) had zero days of margin for the processing flows for STS-115, STS-117, and STS-120. The holiday margin is the number of days that could be gained by working holidays. The Dryden margin is the six days that are always reserved to accommodate an Orbiter landing at Edwards Air Force Base in California and having to be ferried to Kennedy. If the Orbiter landed at Kennedy, those six days would automatically be regained. Note that the Dryden margin had already been surrendered in the STS-114 and STS-115 schedules. If bad weather at Kennedy forced those two flights to land at Edwards, the schedule would be directly affected. The clear message in these charts is that any technical problem that resulted in a slip to one launch would now directly affect the Node 2 launch. The lack of housing for the Orbiters was becoming a factor as well. Prior to launch, an Orbiter can be placed in an Orbiter Processing Facility, the Vehicle Assembly Building, or on one of the two Shuttle launch pads. Maintenance and
SSP Schedule Reserve
Time Now
Mar 03
"3rd shift". Adds + 8 day reserve per flow to mitigate "threats"
STS-115 Flow
STS-120 Flow
STS-117 Flow
Work 2003 Xmas holidays to hold schedule, if req'd
Work 2003 Xmas holidays to preserve 18 day margin
Potential 15 day schedule impact for each flow = 45 day total threat (+/- 15 days)
11 2002 12 1 2 3 4 5 2003 6 7 8 9
10 11 12 1
2/19/04 STS-120 Node 2 Core Complete
2 2004 3 4 5
SSP Core Complete Schedule Threats
STS-120/Node 2 launch subject to 45 days of schedule risk • Space Shuttle technical problems • Station on-orbit technical problems/mission requirements impact • Range launch cutouts • Weather delays • Soyuz and Progress conflicts
Management Options
• USA commit holiday/weekend reserves and apply additional resources to hold schedule 1. Flex 3rd shift avail––Mar 03 2. LCC 3rd shift avail––Apr 03 • HQ mitigate Range Cutout • HQ and ISS mitigate Soyuz conflict threat
Figure 6.2-4. By late 2002, the vehicle processing team at the Kennedy Space Center would be required to work through the winter holidays, and a third shift was being hired in order to meet the February 19, 2004, schedule for U.S. Core Complete.
Report Volume I
August 2003
refurbishment is performed in the three Orbiter Processing Facilities at Kennedy. One was occupied by Discovery during its scheduled extended maintenance. This left two to serve the other three Orbiters over the next several months. The 2003 schedule indicated plans to move Columbia (after its return from STS-107) from an Orbiter Processing Facility to the Vehicle Assembly Building and back several times in order to make room for Atlantis (OV-104) and Endeavour (OV-105) and prepare them for missions. Moving an Orbiter is tedious, time-consuming, carefully orchestrated work. Each move introduces an opportunity for problems. Those 2003 moves were often slated to occur without a day of margin between them – another indication of the additional risks that managers were willing to incur to meet the schedule. The effect of the compressed schedule was also evident in the Mission Operations Directorate. The plans for flight controller staffing of Mission Control showed that of the seven flight controllers who lacked current certifications during STS-107 (see Chapter 4), five were scheduled to work the next mission, and three were scheduled to work the next three missions (STS-114, -115, and -116). These controllers would have been constantly either supporting missions or supporting mission training, and were unlikely to have
the time to complete the recertification requirements. With the pressure of the schedule, the things perceived to be less important, like recertification (which was not done before STS-107), would likely continue to be deferred. As a result of the schedule pressure, managers either were willing to delay recertification or were too busy to notice that deadlines for recertification had passed. Columbia: Caught in the Middle STS-112 flew in October 2002. At 33 seconds into the flight, a piece of the bipod foam from the External Tank struck one of the Solid Rocket Boosters. As described in Section 6.1, the STS-112 foam strike was discussed at the Program Requirements Control Board following the flight. Although the initial recommendation was to treat the foam loss as an In-Flight Anomaly, the Shuttle Program instead assigned it as an action, with a due date after the next launch. (This was the first instance of bipod foam loss that was not designated an In-Flight Anomaly.) The action was noted at the STS-113 Flight Readiness Review. Those Flight Readiness Review charts (see Section 6.1) provided a flawed flight rationale by concluding that the foam loss was "not a safety-of-flight" issue.
Margin to Critical Path
Processing Holiday Dryden Margin Reserve Margin
SSP Schedule Reserve
Days of Reserve
Range Cutout
Launch Cutout
Launch Cutout
Launch Cutout
Launch Cutout
Launch Cutout
8 9
26 3
5/23/03 10/2/03
Critical path
18 9
17 2
6 19 10 6
02/03 05/03 08/03
STS-120 Node 2
SSP Core Complete Schedule Threats
STS-120/Node 2 launch subject to 45 days of schedule risk • Space Shuttle technical problems • Station on-orbit technical problems/mission requirements impact • Range launch cutouts • Weather delays • Soyuz and Progress conflicts
Management Options
• USA commit holiday/weekend reserves and apply additional resources (i.e., 3rd shift) to hold schedule (Note: 3rd shift not yet included) • HQ mitigate Range Cutout • HQ and ISS mitigate Soyuz conflict threat
Figure 6.2-5. This slide shows the margin for each Orbiter. The large boxes show the number of days margin to the Node 2 launch date, while the three smaller boxes indicate vehicle processing margin, holiday margin, and the margin if a Dryden landing was not required.
Report Volume I August 2003
Interestingly, during Columbia's mission, the Chair of the Mission Management Team, Linda Ham, would characterize that reasoning as "lousy" – though neither she nor Shuttle Program Manager Ron Dittemore, who were both present at the meeting, questioned it at the time. The pressing need to launch STS-113 to retrieve the International Space Station Expedition 5 crew before they surpassed the 180-day limit and to continue the countdown to Node 2 were surely in the back of managers' minds during these reviews. By December 2002, every bit of padding in the schedule had disappeared. Another chart from the Shuttle and Station Program Managers' briefing to the NASA Administrator summarizes the schedule dilemma (see Figure 6.2-6). Even with work scheduled on holidays, a third shift of workers being hired and trained, future crew rotations drifting beyond 180 days, and some tests previously deemed "requirements" being skipped or deferred, Program managers estimated that Node 2 launch would be one to two months late. They were slowly accepting additional risk in trying to meet a schedule that probably could not be met. Interviews with workers provided insight into how this situation occurred. They noted that people who work at NASA have the legendary can-do attitude, which contributes to the agency's successes. But it can also cause problems. When workers are asked to find days of margin, they work furiously to do so and are praised for each extra day they find. But
those same people (and this same culture) have difficulty admitting that something "can't" or "shouldn't" be done, that the margin has been cut too much, or that resources are being stretched too thin. No one at NASA wants to be the one to stand up and say, "We can't make that date." STS-107 was launched on January 16, 2003. Bipod foam separated from the External Tank and struck Columbia's left wing 81.9 seconds after liftoff. As the mission proceeded over the next 16 days, critical decisions about that event would be made. The STS-107 Mission Management Team Chair, Linda Ham, had been present at the Program Requirements Control Board discussing the STS-112 foam loss and the STS-113 Flight Readiness Review. So had many of the other Shuttle Program managers who had roles in STS-107. Ham was also the Launch Integration Manager for the next mission, STS114. In that capacity, she would chair many of the meetings leading up to the launch of that flight, and many of those individuals would have to confront Columbia's foam strike and its possible impact on the launch of STS-114. Would the Columbia foam strike be classified as an In-Flight Anomaly? Would the fact that foam had detached from the bipod ramp on two out of the last three flights have made this problem a constraint to flight that would need to be solved before the next launch? Could the Program develop a solid rationale to fly STS-114, or would additional analysis be required to clear the flight for launch?
• Critical Path to U.S. Core Complete driven by Shuttle Launch
Program Station assessment: up to 14 days late Program Shuttle assessment: up to 45 days late
• Program proactively managing schedule threats • Most probable launch date is March 19-April 19
Program Target Remains 2/19/04
Figure 6.2-6. By December 2002, every bit of padding in the schedule had disappeared. Another chart from the Shuttle and Station Program Managers' briefing to the NASA Administrator summarizes the schedule dilemma.
Report Volume I
August 2003
-----Original Message----From: HAM, LINDA J. (JSC-MA2) (NASA) Sent: Wednesday, January 22, 2003 10:16 AM To: DITTEMORE, RONALD D. (JSC-MA) (NASA) Subject: RE: ET Briefing - STS-112 Foam Loss
Yes, I remember....It was not good. I told Jerry to address it at the ORR next Tuesday (even though he won't have any more data and it really doesn't impact Orbiter roll to the VAB). I just want him to be thinking hard about this now, not wait until IFA review to get a formal action.
[ORR=Orbiter Rollout Review, VAB=Vehicle Assembly Building, IFA=In-Flight Anomaly]
In fact, most of Linda Ham's inquiries about the foam strike were not to determine what action to take during Columbia's mission, but to understand the implications for STS-114. During a Mission Management Team meeting on January 21, she asked about the rationale put forward at the STS-113 Flight Readiness Review, which she had attended. Later that morning she reviewed the charts presented at that Flight Readiness Review. Her assessment, which she e-mailed to Shuttle Program Manager Ron Dittemore on January 21, was "Rationale was lousy then and still is …" (See Section 6.3 for the e-mail.) One of Ham's STS-114 duties was to chair a review to determine if the mission's Orbiter, Atlantis, should be rolled from the Orbiter Processing Facility to the Vehicle Assembly Building, per its pre-launch schedule. In the above e-mail to Ron Dittemore, Ham indicates a desire to have the same individual responsible for the "lousy" STS-113 flight rationale start working the foam shedding issue – and presumably present a new flight rationale – very soon. As STS-107 prepared for re-entry, Shuttle Program managers prepared for STS-114 flight rationale by arranging to have post-flight photographs taken of Columbia's left wing rushed to Johnson Space Center for analysis. As will become clear in the next section, most of the Shuttle Program's concern about Columbia's foam strike were not about the threat it might pose to the vehicle in orbit, but about the threat it might pose to the schedule. Conclusion The agency's commitment to hold firm to a February 19, 2004, launch date for Node 2 influenced many of decisions in the months leading up to the launch of STS-107, and may well have subtly influenced the way managers handled the STS-112 foam strike and Columbia's as well. When a program agrees to spend less money or accelerate a schedule beyond what the engineers and program managers think is reasonable, a small amount of overall risk is added. These little pieces of risk add up until managers are no longer aware of the total program risk, and are, in fact, gambling. Little by little, NASA was accepting more and more risk in order to stay on schedule.
Findings F6.2-1 F6.2-2 NASA Headquarters' focus was on the Node 2 launch date, February 19, 2004. The intertwined nature of the Space Shuttle and Space Station programs significantly increased the complexity of the schedule and made meeting the schedule far more challenging. The capabilities of the system were being stretched to the limit to support the schedule. Projections into 2003 showed stress on vehicle processing at the Kennedy Space Center, on flight controller training at Johnson Space Center, and on Space Station crew rotation schedules. Effects of this stress included neglecting flight controller recertification requirements, extending crew rotation schedules, and adding incremental risk by scheduling additional Orbiter movements at Kennedy. The four flights scheduled in the five months from October 2003, to February 2004, would have required a processing effort comparable to the effort immediately before the Challenger accident. There was no schedule margin to accommodate unforeseen problems. When flights come in rapid succession, there is no assurance that anomalies on one flight will be identified and appropriately addressed before the next flight. The environment of the countdown to Node 2 and the importance of maintaining the schedule may have begun to influence managers' decisions, including those made about the STS-112 foam strike. During STS-107, Shuttle Program managers were concerned with the foam strike's possible effect on the launch schedule.
Recommendation: R6.2-1 Adopt and maintain a Shuttle flight schedule that is consistent with available resources. Although schedule deadlines are an important management tool, those deadlines must be regularly evaluated to ensure that any additional risk incurred to meet the schedule is recognized, understood, and acceptable.
Report Volume I
August 2003
Initial Foam Strike Identification As soon as Columbia reached orbit on the morning of January 16, 2003, NASA's Intercenter Photo Working Group began reviewing liftoff imagery by video and film cameras on the launch pad and at other sites at and nearby the Kennedy Space Center. The debris strike was not seen during the first review of video imagery by tracking cameras, but it was noticed at 9:30 a.m. EST the next day, Flight Day Two, by Intercenter Photo Working Group engineers at Marshall Space Flight Center. Within an hour, Intercenter Photo Working Group personnel at Kennedy also identified the strike on higher-resolution film images that had just been developed. The images revealed that a large piece of debris from the left bipod area of the External Tank had struck the Orbiter's left wing. Because the resulting shower of post-impact fragments could not be seen passing over the top of the wing, analysts concluded that the debris had apparently impacted the left wing below the leading edge. Intercenter Photo Working Group members were concerned about the size of the object and the apparent momentum of the strike. In searching for better views, Intercenter Photo Working Group members realized that none of the other cameras provided a higher-quality view of the impact and the potential damage to the Orbiter. Of the dozen ground-based camera sites used to obtain images of the ascent for engineering analyses, each of which has film and video cameras, five are designed to track the Shuttle from liftoff until it is out of view. Due to expected angle of view and atmospheric limitations, two sites did not capture the debris event. Of the remaining three sites positioned to "see" at least a portion of the event, none provided a clear view of the actual debris impact to the wing. The first site lost track of Columbia on ascent, the second site was out of focus – because of an improperly maintained lens – and the third site captured only a view of the upper side of Columbia's left wing. The Board notes that camera problems also hindered the Challenger investigation. Over the years, it appears that due to budget and camera-team staff cuts, NASA's ability to track ascending Shuttles has atrophied – a development that reflects NASA's disregard of the developmental nature of the Shuttle's technology. (See recommendation R3.4-1.) Because they had no sufficiently resolved pictures with which to determine potential damage, and having never seen such a large piece of debris strike the Orbiter so late in ascent, Intercenter Photo Working Group members decided to ask for ground-based imagery of Columbia.
To accomplish this, the Intercenter Photo Working Group's Chair, Bob Page, contacted Wayne Hale, the Shuttle Program Manager for Launch Integration at Kennedy Space Center, to request imagery of Columbia's left wing on-orbit. Hale, who agreed to explore the possibility, holds a Top Secret clearance and was familiar with the process for requesting military imaging from his experience as a Mission Control Flight Director. This would be the first of three discrete requests for imagery by a NASA engineer or manager. In addition to these three requests, there were, by the Board's count, at least eight "missed opportunities" where actions may have resulted in the discovery of debris damage. Shortly after confirming the debris hit, Intercenter Photo Working Group members distributed a "L+1" (Launch plus one day) report and digitized clips of the strike via e-mail throughout the NASA and contractor communities. This report provided an initial view of the foam strike and served as the basis for subsequent decisions and actions. Mission Management's Response to the Foam Strike As soon as the Intercenter Working Group report was distributed, engineers and technical managers from NASA, United Space Alliance, and Boeing began responding. Engineers and managers from Kennedy Space Center called engineers and Program managers at Johnson Space Center. United Space Alliance and Boeing employees exchanged e-mails with details of the initial film analysis and the work in progress to determine the result of the impact. Details of the strike, actions taken in response to the impact, and records of telephone conversations were documented in the Mission Control operational log. The following section recounts in
Report Volume I August 2003
chronological order many of these exchanges and provides insight into why, in spite of the debris strike's severity, NASA managers ultimately declined to request images of Columbia's left wing on-orbit. Flight Day Two, Friday, January 17, 2003 In the Mission Evaluation Room, a support function of the Shuttle Program office that supplies engineering expertise for missions in progress, a set of consoles are staffed by engineers and technical managers from NASA and contractor organizations. For record keeping, each Mission Evaluation Room member types mission-related comments into a running log. A log entry by a Mission Evaluation Room manager at 10:58 a.m. Central Standard Time noted that the vehicle may have sustained damage from a debris strike. "John Disler [a photo lab engineer at Johnson Space Center] called to report a debris hit on the vehicle. The debris appears to originate from the ET Forward Bipod area…travels down the left side and hits the left wing leading edge near the fuselage…The launch video review team at KSC think that the vehicle may have been damaged by the impact. Bill Reeves and Mike Stoner (USA SAM) were notified." [ET=External Tank, KSC=Kennedy Space
Center, USA SAM=United Space Alliance Sub-system Area Manager]
At 3:15 p.m., Bob Page, Chair of the Intercenter Photo Working Group, contacted Wayne Hale, the Shuttle Program Manager for Launch Integration at Kennedy Space Center, and Lambert Austin, the head of the Space Shuttle Systems Integration at Johnson Space Center, to inform them that Boeing was performing an analysis to determine trajectories, velocities, angles, and energies for the debris impact. Page also stated that photo-analysis would continue over the Martin Luther King Jr. holiday weekend as additional film from tracking cameras was developed. Shortly thereafter, Wayne Hale telephoned Linda Ham, Chair of the Mission Management Team, and Ron Dittemore, Space Shuttle Program Manager, to pass along information about the debris strike and let them know that a formal report would be issued by the end of the day. John Disler, a member of the Intercenter Photo Working Group, notified the Mission Evaluation Room manager that a newly formed group of analysts, to be known as the Debris Assessment Team, needed the entire weekend to conduct a more thorough analysis. Meanwhile, early opinions about Reinforced Carbon-Carbon (RCC) resiliency were circulated via e-mail between United Space Alliance technical managers and NASA engineers, which may have contributed to a mindset that foam hitting the RCC was not a concern.
-----Original Message----From: Stoner-1, Michael D Sent: Friday, January 17, 2003 4:03 PM To: Woodworth, Warren H; Reeves, William D Cc: Wilder, James; White, Doug; Bitner, Barbara K; Blank, Donald E; Cooper, Curt W; Gordon, Michael P. Subject: RE: STS 107 Debris
Just spoke with Calvin and Mike Gordon (RCC SSM) about the impact. Basically the RCC is extremely resilient to impact type damage. The piece of debris (most likely foam/ice) looked like it most likely impacted the WLE RCC and broke apart. It didn't look like a big enough piece to pose any serious threat to the system and Mike Gordon the RCC SSM concurs. At T +81seconds the piece wouldn't have had enough energy to create a large damage to the RCC WLE system. Plus they have analysis that says they have a single mission safe re-entry in case of impact that penetrates the system. As far as the tile go in the wing leading edge area they are thicker than required (taper in the outer mold line) and can handle a large area of shallow damage which is what this event most likely would have caused. They have impact data that says the structure would get slightly hotter but still be OK. Mike Stoner USA TPS SAM
[RCC=Reinforced Carbon-Carbon, SSM=Sub-system Manager, WLE=Wing Leading Edge, TPS=Thermal Protection System, SAM= Sub-system Area Manager]
Report Volume I August 2003
After United Space Alliance became contractually responsible for most aspects of Shuttle operations, NASA developed procedures to ensure that its own engineering expertise was coordinated with that of contractors for any "out-of-family" issue. In the case of the foam strike on STS-107, which was classified as out-of-family, clearly defined written guidance led United Space Alliance technical managers to liaise with their NASA counterparts. Once NASA managers were officially notified of the foam strike classification, and NASA engineers joined their contractor peers in an early analysis, the resultant group should, according to standing procedures, become a Mission Evaluation Room Tiger Team. Tiger Teams have clearly defined roles and responsibilities.43 Instead, the group of analysts came to be called a Debris Assessment Team. While they were the right group of engineers working the problem at the right time, by not being classified as a Tiger Team, they did not fall under the Shuttle Program procedures described in Tiger Team checklists, and as a result were not "owned" or led by Shuttle Program managers. This left the Debris Assessment Team in a kind of organizational limbo, with no guidance except the date by which Program managers expected to hear their results:
January 24th.
Already, by Friday afternoon, Shuttle Program managers and working engineers had different levels of concern about what the foam strike might have meant. After reviewing available film, Intercenter Photo Working Group engineers believed the Orbiter may have been damaged by the strike. They wanted on-orbit images of Columbia's left wing to confirm their suspicions and initiated action to obtain them. Boeing and United Space Alliance engineers decided to work through the holiday weekend to analyze the strike. At the same time, high-level managers Ralph Roe, head of the Shuttle Program Office of Vehicle Engineering, and Bill Reeves, from United Space Alliance, voiced a lower level of concern. It was at this point, before any analysis had started, that Shuttle Program managers officially shared their belief that the strike posed no safety issues, and that there was no need for a review to be conducted over the weekend. The following is a 4:28 p.m. Mission Evaluation Room manager log entry: "Bill Reeves called, after a meeting with Ralph Roe, it is confirmed that USA/Boeing will not work the debris issue over the weekend, but will wait till Monday when the films are released. The LCC constraints on ice, the energy/speed of impact at +81 seconds, and the toughness of the RCC are two main factors for the low concern. Also, analysis supports single mission safe re-entry for an impact that penetrates the system…" [USA=United Space
Alliance, LCC=Launch Commit Criteria]
The following is a 4:37 p.m. Mission Evaluation Room manager log entry. "Bob Page told MER that KSC/TPS engineers were sent by the USA SAM/Woody Woodworth to review the video and films. Indicated that Page had said that Woody had said this was an action from the MER to work this issue and a possible early landing on Tuesday. MER Manager told Bob that no official action was given by USA or Boeing and they had no concern about landing early. Woody indicated that the TPS engineers at KSC have been ʻturned away' from reviewing the films. It was stated that the film reviews wouldn't be finished till Monday." [MER=Mission Evaluation Room, KSC=Kennedy Space Center, TPS=Thermal
Protection System, USA SAM=United Space Alliance Sub-system Area Manager]
The Mission Evaluation Room manager also wrote: "I also confirmed that there was no rush on this issue and that it was okay to wait till the film reviews are finished on Monday to do a TPS review." In addition to individual log entries by Mission Evaluation Room members, managers prepared "handover" notes for delivery from one working shift to the next. Handovers from Shift 1 to 2 on January 17 included the following entry under a "problem" category. "Disler Report – Debris impact on port wing edge-appears to have originated at the ET fwd bipod – foam?- if so, it shouldn't be a problem – video clip will be available on the web soon – will look at high-speed film today." [ET=External Tank, fwd=forward]
Report Volume I August 2003
Shortly after these entries were made, the deputy manager of Johnson Space Center Shuttle Engineering notified Rodney Rocha, NASA's designated chief engineer for the Thermal Protection System, of the strike and the approximate debris size. It was Rocha's responsibility to coordinate NASA engineering resources and work with contract engineers at United Space Alliance, who together would form a Debris Assessment Team that would be Co-Chaired by United Space Alliance engineering manager Pam Madera. The United Space Alliance deputy manager of Shuttle Engineering signaled that the debris strike was initially classified as "out-of-family" and therefore of greater concern than previous debris strikes. At about the same time, the Intercenter Photo Working Group's L+1 report, containing both video clips and still images of the debris strike, was e-mailed to engineers and technical managers both inside and outside of NASA. Flight Days Three and Four, Saturday and Sunday, January 18 and 19, 2003 Though senior United Space Alliance Manager Bill Reeves had told Mission Evaluation Room personnel that the debris problem would not be worked over the holiday weekend, engineers from Boeing did in fact work through the weekend. Boeing analysts conducted a preliminary damage assessment on Saturday. Using video and photo images, they generated two estimates of possible debris size – 20 inches by 20 inches by 2 inches, and 20 inches by 16 inches by 6 inches – and determined that the debris was traveling at a approximately 750 feet per second, or 511 miles per hour, when it struck the Orbiter at an estimated impact angle of less than 20 degrees. These estimates later proved remarkably accurate. To calculate the damage that might result from such a strike, the analysts turned to a Boeing mathematical modeling tool called Crater that uses a specially developed algorithm to predict the depth of a Thermal Protection System tile to which debris will penetrate. This algorithm, suitable for estimating small (on the order of three cubic inches) debris impacts, had been calibrated by the results of foam, ice, and metal debris impact testing. A similar Crater-like algorithm was also developed and validated with test results to assess the damage caused by ice projectiles impacting the RCC leading edge panels. These tests showed that within certain limits, the Crater algorithm predicted more severe damage than was observed. This led engineers to classify Crater as a "conservative" tool – one that predicts more damage than will actually occur. Until STS-107, Crater was normally used only to predict whether small debris, usually ice on the External Tank, would pose a threat to the Orbiter during launch. The use of Crater to assess the damage caused by foam during the launch of STS-107 was the first use of the model while a mission was on orbit. Also of note is that engineers used Crater during STS-107 to analyze a piece of debris that was at maximum 640 times larger in volume than the pieces of debris used to calibrate and validate the Crater model (the Board's best estimate is that it actually was 400 times larger). Therefore, the use of Crater in this new and very different situation compromised NASA's ability to accurately predict debris damage in ways that Debris Assessment Team engineers did not fully comprehend (see Figure 6.3-1).
3.75" .875"
Figure 6.3-1. The small cylinder at top illustrates the size of debris Crater was intended to analyze. The larger cylinder was used for the STS-107 analysis; the block at right is the estimated size of the foam.
Report Volume I August 2003
p L d ρP V V* ST ρT 0.0195 = = = = = = = = =
0.0195(L/d)0.45(d)(ρP)0.27(V-V*)2/3 (ST)1/4(ρT)1/6
penetration depth length of foam projectile diameter of foam projectile density of foam component of foam velocity at right angle to foam velocity required to break through the tile coating compressive strength of tile density of tile empirical constant
In 1966, during the Apollo program, engineers developed an equation to assess impact damage, or "cratering," by micrometeoroids.44 The equation was modified between 1979 and 1985 to enable the analysis of impacts to "acreage" tiles that cover the lower surface of the Orbiter.45 The modified equation, now known as Crater, predicts possible damage from sources such as foam, ice, and launch site debris, and is most often used in the day-of-launch analysis of ice debris falling off the External Tank.46 When used within its validated limits, Crater provides conservative predictions (that is, Crater predictions are larger than actual damage). When used outside its validated limits, Crater's precision is unknown. Crater has been correlated to actual impact data using results from several tests. Preliminary ice drop tests were performed in 1978,47 and additional tests using sprayed-on foam insulation projectiles were conducted in 1979 and 1999.48 However, the test projectiles were relatively small (maximum volume of 3 cubic inches), and targeted only single tiles, not groups of tiles as actually installed on the Orbiter. No tests were performed with larger debris objects because it was not believed such debris could ever impact the Orbiter. This resulted in a very limited set of conditions under which Crater's results were empirically validated. During 1984, tests were conducted using ice projectiles against the Reinforced Carbon-Carbon used on the Orbiters' wing leading edges.49 These tests used an 0.875-inch diameter, 3.75-inch long ice projectile to validate an algorithm that was similar to Crater. Unlike Crater, which was designed to predict damage during a flight, the RCC predictions were intended to determine the thickness of RCC required to withstand ice impacts as an aid to design engineers. Like Crater, however, the limited set of test data significantly restricts the potential application of the model. Other damage assessment methods available today, such as hydrodynamic structural codes, like Dyna, are able to analyze a larger set of projectile sizes and materials than Crater. Boeing and NASA did not currently sanction these finite element codes because of the time required to correlate their results in order to use the models effectively. Although Crater was designed, and certified, for a very limited set of impact events, the results from Crater simulations can be generated quickly. During STS-107, this led to Crater being used to model an event that was well outside the parameters against which it had been empirically validated. As the accompanying table shows, many of the STS-107 debris characteristics were orders of magnitude outside the validated envelope. For instance, while Crater had been designed and validated for projectiles up to 3 cubic inches in volume, the initial STS-107 analysis estimated the piece of debris at 1,200 cubic inches – 400 times larger. Crater parameters used during development of experimental test data versus STS-107 analysis: Test Parameter Volume Length Cylinder Dimensions Projectile Block Dimensions Tile Material Projectile Shape
* Outside experimental test limits
Test Value Up to 3 cu.in Up to 1 inch <= 3/8" dia x 3" <= 3"x 1"x 1" LI-900 "acreage" tile Cylinder
STS-107 Analysis 10" x 6" x 20" = 1200 cu.in. * ~ 20 inches * 6" dia x 20" 6" x 10" x 20" * LI-2200 * and LI-900 Block
Report Volume I
August 2003
Crater equation parameter limits: Crater Equation Parameter L/d L
d V
Applicable Range 1 – 20 n/a 1 – 3 pounds per cu.ft. 0.4 – 2.0 inches up to 810 fps
STS-107 Analysis 3.3 ~ 20 inches 2.4 pounds per cu.ft. 6 inches * ~ 700 fps
* Outside validated limits
Over the weekend, an engineer certified by Boeing to use Crater entered the two estimated debris dimensions, the estimated debris velocity, and the estimated angle of impact. The engineer had received formal training on Crater from senior Houston-based Boeing engineering staff, but he had only used the program twice before, and had reservations about using it to model the piece of foam debris that struck Columbia. The engineer did not consult with more experienced engineers from Boeing's Huntington Beach, California, facility, who up until the time of STS-107 had performed or overseen Crater analysis. (Boeing completed the transfer of responsibilities for Crater analysis from its Huntington Beach engineers to its Houston office in January 2003. STS-107 was the first mission that the Huntington Beach engineers were not directly involved with.) For the Thermal Protection System tile, Crater predicted damage deeper than the actual tile thickness. This seemingly alarming result suggested that the debris that struck Columbia would have exposed the Orbiter's underlying aluminum airframe to extreme temperatures, resulting in a possible burn-through during re-entry. Debris Assessment Team engineers discounted the possibility of burn through for two reasons. First, the results of calibration tests with small projectiles showed that Crater predicted a deeper penetration than would actually occur. Second, the Crater equation does not take into account the increased density of a tile's lower "densified" layer, which is much stronger than tile's fragile outer layer. Therefore, engineers judged that the actual damage from the large piece of foam lost on STS-107 would not be as severe as Crater predicted, and assumed that the debris did not penetrate the Orbiter's skin. This uncertainty, however, meant that determining the precise location of the impact was paramount for an accurate damage estimate. Some areas on the Orbiter's lower surface, such as the seals around the landing gear doors, are more vulnerable than others. Only by knowing precisely where the debris struck could the analysts more accurately determine if the Orbiter had been damaged. To determine potential RCC damage, analysts used a Crater-like algorithm that was calibrated in 1984 by impact data from ice projectiles. At the time the algorithm was empirically tested, ice was considered the only realistic threat to RCC integrity. (See Appendix E.4, RCC Impact Analysis.) The Debris Assessment Team analysis indicated that impact angles greater than 15 degrees would result in RCC penetration. A separate "transport" analysis, which attempts to determine the path the debris took, identified 15 strike regions and angles of impact. Twelve transport scenarios predicted an impact in regions of Shuttle tile. Only one scenario predicted an impact on the RCC leading edge, at a 21-degree angle. Because the foam that struck Columbia was less dense than ice, Debris Assessment Team analysts used a qualitative extrapolation of the test data and engineering judgment to conclude that a foam impact angle up to 21 degrees would not penetrate the RCC. Although some engineers were uncomfortable with this extrapolation, no other analyses were performed to assess RCC damage. The Debris Assessment Team focused on analyzing the impact at locations other than the RCC leading edge. This may have been due, at least in part, to the transport analysis presentation and the long-standing belief that foam was not a threat to RCC panels. The assumptions and uncertainty embedded in this analysis were never fully presented to the Mission Evaluation Room or the Mission Management Team.
On Sunday, Rodney Rocha e-mailed a Johnson Space Center Engineering Directorate manager to ask if a Mission Action Request was in progress for Columbia's crew to visually inspect the left wing for damage. Rocha never received an answer.
Report Volume I August 2003
RCC Panel 12
This photo from the aft flight deck window of an Orbiter shows that RCC panels 1 – 11 are not visible from inside the Orbiter. Since Columbia did not have a manipulator arm for STS-107, it would have been necessary for an astronaut to take a spacewalk to visibly inspect the inboard leading edge of the wing.
Flight Day Five, Monday, January 20, 2003 On Monday morning, the Martin Luther King Jr. holiday, the Debris Assessment Team held an informal meeting before its first formal meeting, which was scheduled for Tuesday afternoon. The team expanded to include NASA and Boeing transport analysts expert in the movement of debris in airflows, tile and RCC experts from Boeing and NASA, aerothermal and thermal engineers from NASA, United Space Alliance, and Boeing, and a safety representative from the NASA contractor Science Applications International Corporation. Engineers emerged from that informal meeting with a goal of obtaining images from groundbased assets. Uncertainty as to precisely where the debris had struck Columbia generated concerns about the possibility of a breach in the left main landing gear door seal. They conducted further analysis using angle and thickness variables and thermal data obtained by personnel at Boeing's Huntington Beach facility for STS-87 and STS-50, the two missions that had incurred Thermal Protection System damage. (See Section 6.1.) Debris Assessment Team Co-Chair Pam Madera distributed an e-mail summarizing the day's events and outlined the agenda for Tuesday's first formal Debris Assessment Team meeting. Included on the agenda was the desire to obtain on-orbit images of Columbia's left wing. According to an 11:39 a.m. entry in the Mission Evaluation Room Manager's log: "…the debris ʻblob' is estimated at 20" +/-10" in some direction, using the Orbiter hatch as a basis. It appears to be similar size as that seen in STS-112. There will be more comparison work done, and more info and details in tomorrow's report." This entry illustrates, in NASA language, an initial attempt by managers to classify this bipod ramp foam strike as close to being within the experience base and therefore, being almost an "in-family" event, not necessarily a safety concern. While the size and source of STS-107 debris was somewhat similar to what STS-112 had experienced, the impact sites (the wing versus the Solid Rocket Booster) differed – a distinction not examined by mission managers.
Report Volume I August 2003
Flight Day Six, Tuesday, January 21, 2003 At 7:00 a.m., the Debris Assessment Team briefed Don McCormack, the chief Mission Evaluation Room manager, that the foam's source and size was similar to what struck STS-112, and that an analysis of measured versus predicted tile damage from STS-87 was being scrutinized by Boeing. An hour later, McCormack related this information to the Mission Management Team at its first post-holiday meeting. Although Space Shuttle Program requirements state that the Mission Management Team will convene daily during a mission, the STS-107 Mission Management Team met only on January 17, 21, 24, 27, and 31. The transcript below is the first record of an official discussion of the debris impact at a Mission Management Team meeting. Before even referring to the debris strike, the Mission Management Team focused on end-ofmission "downweight" (the Orbiter was 150 pounds over the limit), a leaking water separator, a jammed Hasselblad camera, payload and experiment status, and a communications downlink problem. McCormack then stated that engineers planned to determine what could be done if Columbia had sustained damage. STS-107 Mission Management Team Chair Linda Ham suggested the team learn what rationale had been used to fly after External Tank foam losses on STS-87 and STS-112. Transcript Excerpts from the January 21, Mission Management Team Meeting Ham: "Alright, I know you guys are looking at the debris." McCormack: "Yeah, as everybody knows, we took a hit on the, somewhere on the left wing leading edge and the photo TV guys have completed I think, pretty much their work although I'm sure they are reviewing their stuff and they've given us an approximate size for the debris and approximate area for where it came from and approximately where it hit, so we are talking about doing some sort of parametric type of analysis and also we're talking about what you can do in the event we have some damage there." Ham: "That comment, I was thinking that the flight rationale at the FRR from tank and orbiter from STS-112 was.… I'm not sure that the area is exactly the same where the foam came from but the carrier properties and density of the foam wouldn't do any damage. So we ought to pull that along with the 87 data where we had some damage, pull this data from 112 or whatever flight it was and make sure that…you know I hope that we had good flight rationale then." McCormack: "Yeah, and we'll look at that, you mentioned 87, you know we saw some fairly significant damage in the area between RCC panels 8 and 9 and the main landing gear door on the bottom on STS-87 we did some analysis prior to STS-89 so uh…" Ham: "And I'm really I don't think there is much we can do so it's not really a factor during the flight because there is not much we can do about it. But what I'm really interested in is making sure our flight rationale to go was good, and maybe this is foam from a different area and I'm not sure and it may not be co-related, but you can try to see what we have." McCormack: "Okay." After the meeting, the rationale for continuing to fly after the STS-112 foam loss was sent to Ham for review. She then exchanged e-mails with her boss, Space Shuttle Program Manager Ron Dittemore:
-----Original Message----From: DITTEMORE, RONALD D. (JSC-MA) (NASA) Sent: Wednesday, January 22, 2003 9:14 AM To: HAM, LINDA J. (JSC-MA2) (NASA) Subject: RE: ET Briefing - STS-112 Foam Loss
You remember the briefing! Jerry did it and had to go out and say that the hazard report had not changed and that the risk had not changed...But it is worth looking at again.
-----Original Message----From: HAM, LINDA J. (JSC-MA2) (NASA) Sent: Tuesday, January 21, 2003 11:14 AM e p o r t (NASA) Volume I To: DITTEMORE, RONALD D. R (JSC-MA) Subject: FW: ET Briefing - STS-112 Foam Loss
[continued on next page]
August 2003
-----Original Message----From: DITTEMORE, RONALD D. (JSC-MA) (NASA) Sent: Wednesday, January 22, 2003 9:14 AM ACCIDENT INVESTIGATION BOARD To: HAM, LINDA J. (JSC-MA2) (NASA) Subject: RE: ET Briefing - STS-112 Foam Loss
You remember the briefing! Jerry did it and had to go out and say that the hazard report had not [continued from previous changed and thatpage] the risk had not changed...But it is worth looking at again.
-----Original Message----From: HAM, LINDA J. (JSC-MA2) (NASA) Sent: Tuesday, January 21, 2003 11:14 AM To: DITTEMORE, RONALD D. (JSC-MA) (NASA) Subject: FW: ET Briefing - STS-112 Foam Loss
You probably can't open the attachment. But, the ET rationale for flight for the STS-112 loss of foam was lousy. Rationale states we haven't changed anything, we haven't experienced any 'safety of flight' damage in 112 flights, risk of loss of bi-pod ramp TPS is same as previous flghts...So ET is safe to fly with no added risk Rationale was lousy then and still is....
-----Original Message----From: MCCORMACK, DONALD L. (DON) (JSC-MV6) (NASA) Sent: Tuesday, January 21, 2003 9:45 AM To: HAM, LINDA J. (JSC-MA2) (NASA) Subject: FW: ET Briefing - STS-112 Foam Loss Importance: High
FYI - it kinda says that it will probably be all right
[ORR=Operational Readiness Review, VAB=Vehicle Assembly Building, IFA=In-Flight Anomaly, TPS=Thermal Protection System, ET=External Tank]
Ham's focus on examining the rationale for continuing to fly after the foam problems with STS-87 and STS-112 indicates that her attention had already shifted from the threat the foam posed to STS-107 to the downstream implications of the foam strike. Ham was due to serve, along with Wayne Hale, as the launch integration manager for the next mission, STS-114. If the Shuttle Program's rationale to fly with foam loss was found to be flawed, STS-114, due to be launched in about a month, would have to be delayed per NASA rules that require serious problems to be resolved before the next flight. An STS-114 delay could in turn delay completion of the International Space Station's Node 2, which was a high-priority goal for NASA managers. (See Section 6.2 for a detailed description of schedule pressures.) During this same Mission Management Team meeting, the Space Shuttle Integration Office's Lambert Austin reported that engineers were reviewing long-range tracking film and that the foam debris that appeared to hit the left wing leading edge may have come from the bipod area of the External Tank. Austin said that the Engineering Directorate would continue to run analyses and compare this foam loss to that of STS-112. Austin also said that after STS-107 landed, engineers were anxious to see the crew-filmed footage of External Tank separation that might show the bipod ramp and therefore could be checked for missing foam.
Reviews of flight-deck footage confirm that on Flight Day One, Mission Specialist David Brown filmed parts of the External Tank separation with a Sony PD-100 Camcorder, and Payload Commander Mike Anderson photographed it with a Nikon F-5 camera with a 400-millimeter lens. Brown later downlinked 35 seconds of this video to the ground as part of his Flight Day One mission summary, but the bipod ramp area had rotated out of view, so no evidence of missing foam was seen when this footage was reviewed during the mission. However, after the Intercenter Photo Working Group caught the debris strike on January 17, ground personnel failed to ask Brown if he had additional footage of External Tank separation. Based on how crews are trained to film External Tank separation, the Board concludes Brown did in fact have more film than the 35 seconds he downlinked. Such footage may have confirmed that foam was missing from the bipod ramp area or could have identified other areas of missing foam. Austin's mention of the crew's filming of External Tank separation should have prompted someone at the meeting to ask Brown if he had more External Tank separation film, and if so, to downlink it immediately.
Report Volume I August 2003
Flight Director Steve Stich discussed the debris strike with Phil Engelauf, a member of the Mission Operations Directorate, after Engelauf returned from the Mission Management Team meeting. As written in a timeline Stich composed after the accident, the conversation included the following. "Phil said the Space Shuttle Program community is not concerned and that Orbiter Project is analyzing ascent debris…relayed that there had been no direction for MOD to ask DOD for any photography of possible damaged tiles" [MOD=Mission Operations Directorate, or
Mission Control, DOD=Department of Defense]
"No direction for DOD photography" seems to refer to either a previous discussion of photography with Mission managers or an expectation of future activity. Since the interagency agreement on imaging support stated that the Flight Dynamics Officer is responsible for initiating such a request, Engelauf's comments demonstrates that an informal chain of command, in which the Mission Operations Directorate figures prominently, was at work. About an hour later, Calvin Schomburg, a Johnson Space Center engineer with close connections to Shuttle management, sent the following e-mail to other Johnson engineering managers.
-----Original Message----From: SCHOMBURG, CALVIN (JSC-EA) (NASA) Sent: Tuesday, January 21, 2003 9:26 AM To: SHACK, PAUL E. (JSC-EA42) (NASA); SERIALE-GRUSH, JOYCE M. (JSC-EA) (NASA); HAMILTON, DAVID A. (DAVE) (JSC-EA) (NASA) Subject: FW: STS-107 Post-Launch Film Review - Day 1
FYI-TPS took a hit-should not be a problem-status by end of week.
[FYI=For Your Information, TPS=Thermal Protection System]
Shuttle Program managers regarded Schomburg as an expert on the Thermal Protection System. His message downplays the possibility that foam damaged the Thermal Protection System. However, the Board notes that Schomburg was not an expert on Reinforced Carbon-Carbon (RCC), which initial debris analysis indicated the foam may have struck. Because neither Schomburg nor Shuttle management rigorously differentiated between tiles and RCC panels, the bounds of Schomburg's expertise were never properly qualified or questioned. Seven minutes later, Paul Shack, Manager of the Shuttle Engineering Office, Johnson Engineering Directorate, e-mailed to Rocha and other Johnson engineering managers information on how previous bipod ramp foam losses were handled.
-----Original Message----From: SHACK, PAUL E. (JSC-EA42) (NASA) Sent: Tuesday, January 21, 2003 9:33 AM To: ROCHA, ALAN R. (RODNEY) (JSC-ES2) (NASA); SERIALE-GRUSH, JOYCE M. (JSC-EA) (NASA) Cc: KRAMER, JULIE A. (JSC-EA4) (NASA); MILLER, GLENN J. (JSC-EA) (NASA); RICKMAN, STEVEN L. (JSC-ES3) (NASA); MADDEN, CHRISTOPHER B. (CHRIS) (JSC-ES3) (NASA) Subject: RE: STS-107 Debris Analysis Team Plans
This reminded me that at the STS-113 FRR the ET Project reported on foam loss from the Bipod Ramp during STS-112. The foam (estimated 4X5X12 inches) impacted the ET Attach Ring and dented an SRB electronics box cover. Their charts stated "ET TPS foam loss over the life of the Shuttle program has never been a 'Safety of Flight' issue". They were severely wire brushed over this and Brian O'Conner (Associate Administrator for Safety) asked for a hazard assessment for loss of foam. [continued on next page]
The suspected cause for foam loss is trapped air pockets which expand due to altitude and aerotherReport Volume I August 2003 149 mal heating.
This reminded me that at the STS-113 FRR the ET Project reported on foam loss from the Bipod COLUMBIA Ramp during STS-112. The foam (estimated 4X5X12 inches) impacted the ET Attach Ring and dented an SRB electronics box cover.A C C I D E N T I N V E S T I G A T I O N B O A R D Their charts stated "ET TPS foam loss over the life of the Shuttle program has never been a 'Safety of [continued from previous page] Flight' issue". They were severely wire brushed over this and Brian O'Conner (Associate Administrator for Safety) asked for a hazard assessment for loss of foam. The suspected cause for foam loss is trapped air pockets which expand due to altitude and aerothermal heating.
[FRR=Flight Readiness Review, ET=External Tank, SRB=Solid Rocket Booster, TPS=Thermal Protection System]
Shack's message informed Rocha that during the STS-113 Flight Readiness Review, foam loss was not considered to be a safety-of-flight issue. The "wirebrushing" that the External Tank Project received for stating that foam loss has "never been a ʻSafety of Flight' issue" refers to the wording used to justify continuing to fly. Officials at the Flight Readiness Review insisted on classifying the foam loss as an "accepted risk" rather than "not a safety-of-flight problem" to indicate that although the Shuttle would continue to fly, the threat posed by foam is not zero but rather a known and acceptable risk. It is here that the decision to fly before resolving the foam problem at the STS-113 Flight Readiness Review influences decisions made during STS-107. Having at hand a previously accepted rationale – reached just one mission ago – that foam strikes are not a safety-of-flight issue provides a strong incentive for Mission managers and working engineers to use that same judgment for STS-107. If managers and engineers were to argue that foam strikes are a safety-of-flight issue, they would contradict an established consensus that was a product of the Shuttle Program's most rigorous review – a review in which many of them were active participants. An entry in a Mission Evaluation Room console log included a 10:30 a.m. report that compared the STS-107 foam loss to previous foam losses and subsequent tile damage, which reinforced management acceptance about foam strikes by indicating that the foam strike appeared to be more of an "in-family" event. "…STS-107 debris measured at 22" long +/- 10". On STS-112 the debris spray pattern was a lot smaller than that of STS-107. On STS-50 debris that was determined to be the Bipod ramp which measured 26" x 10" caused damage to the left wing…to 1 tile and 20% of the adjacent tile. Same event occurred on STS-7 (no data available)."
The foam strike to STS-107 was mentioned by a speaker at an unrelated meeting of NASA Headquarters and National Imagery and Mapping Agency personnel, who then discussed a possible NASA request for Department of Defense imagery support. However, no action was taken.
Responding to concerns from his employees who were participating in the Debris Assessment Team, United Space Alliance manager Bob White called Lambert Austin on Flight Day Six to ask what it would take to get imagery of Columbia on orbit. They discussed the analytical debris damage work plan, as well as the belief of some integration team members that such imaging might be beneficial. Austin subsequently telephoned the Department of Defense Manned Space Flight Support Office representative to ask about actions necessary to get imagery of Columbia on orbit. Austin emphasized that this was merely information gathering, not a request for action. This call indicates that Austin was unfamiliar with NASA/National Imagery and Mapping Agency imagery request procedures. An e-mail that Lieutenant Colonel Timothy Lee sent to Don McCormack the following day shows that the Defense Department had begun to implement Austin's request.
Report Volume I
August 2003
-----Original Message----From: LEE, TIMOTHY F., LTCOL. (JSC-MT) (USAF) Sent: Wednesday, January 22, 2003 9:01 AM To: MCCORMACK, DONALD L. (DON) (JSC-MV6) (NASA) Subject: NASA request for DOD
Don, FYI: Lambert Austin called me yesterday requesting DOD photo support for STS-107. Specifically, he is asking us if we have a ground or satellite asset that can take a high resolution photo of the shuttle while on-orbit--to see if there is any FOD damage on the wing. We are working his request. Tim
[DOD=Department of Defense, FOD=Foreign Object Debris]
At the same time, managers Ralph Roe, Lambert Austin, and Linda Ham referred to conversations with Calvin Schomburg, whom they referred to as a Thermal Protection System "expert." They indicated that Schomburg had advised that any tile damage should be considered a turnaround maintenance concern and not a safety-of-flight issue, and that imagery of Columbia's left wing was not necessary. There was no discussion of potential RCC damage. First Debris Assessment Team Meeting On Flight Day Six, the Debris Assessment Team held its first formal meeting to finalize Orbiter damage estimates and their potential consequences. Some participants joined the proceedings via conference call.
After two hours of discussing the Crater results and the need to learn precisely where the debris had hit Columbia, the Debris Assessment Team assigned its NASA Co-Chair, Rodney Rocha, to pursue a request for imagery of the vehicle on-orbit. Each team member supported the idea to seek imagery from an outside source. Rather than working the request up the usual mission chain of command through the Mission Evaluation Room to the Mission Management Team to the Flight Dynamics Officer, the Debris Assessment Team agreed, largely due to a lack of participation by Mission Management Team and Mission Evaluation Room managers, that Rocha would pursue the request through his division, the Engineering Directorate at Johnson Space Center. Rocha sent the following e-mail to Paul Shack shortly after the meeting adjourned.
-----Original Message----From: ROCHA, ALAN R. (RODNEY) (JSC-ES2) (NASA) Sent: Tuesday, January 21, 2003 4:41 PM To: SHACK, PAUL E. (JSC-EA42) (NASA); HAMILTON, DAVID A. (DAVE) (JSC-EA) (NASA); MILLER, GLENN J. (JSCEA) (NASA) Cc: SERIALE-GRUSH, JOYCE M. (JSC-EA) (NASA); ROGERS, JOSEPH E. (JOE) (JSC-ES2) (NASA); GALBREATH, GREGORY F. (GREG) (JSC-ES2) (NASA) Subject: STS-107 Wing Debris Impact, Request for Outside Photo-Imaging Help
Paul and Dave, The meeting participants (Boeing, USA, NASA ES2 and ES3, KSC) all agreed we will always have big uncertainties in any transport/trajectory analyses and applicability/extrapolation of the old Arc-Jet test data until we get definitive, better, clearer photos of the wing and body underside. Without better images it will be very difficult to even bound the problem and initialize thermal, trajectory, and structural analyses. Their answers may have a wide spread ranging from acceptable to not-acceptable to horrible, and no way to reduce uncertainty. Thus, giving MOD options for entry will be very difficult. [continued on next page]
Report Volume I August 2003
Paul and Dave, The meeting participants (Boeing, USA, COLUMBIA NASA ES2 and ES3, KSC) all agreed we will always have A C C I D E analyses NT INVESTIG A T I O applicability/extrapolation N BOARD big uncertainties in any transport/trajectory and of the old Arc-Jet test data until we get definitive, better, clearer photos of the wing and body underside. Without better images it will be very difficult to even bound the problem and initialize thermal, trajectory, and structural analyses. Their answers may have a wide spread ranging from acceptable to not-acceptable to [continued from and previous page] horrible, no way to reduce uncertainty. Thus, giving MOD options for entry will be very difficult. Can we petition (beg) for outside agency assistance? We are asking for Frank Benz with Ralph Roe or Ron Dittemore to ask for such. Some of the old timers here remember we got such help in the early 1980's when we had missing tile concerns. Despite some nay-sayers, there are some options for the team to talk about: On-orbit thermal conditioning for the major structure (but is in contradiction with tire pressure temp. cold limits), limiting high cross-range de-orbit entries, constraining right or left had turns during the Heading Alignment Circle (only if there is struc. damage to the RCC panels to the extent it affects flight control. Rodney Rocha Structural Engineering Division (ES-SED) • ES Div. Chief Engineer (Space Shuttle DCE) • Chair, Space Shuttle Loads & Dynamics Panel Mail Code ES2
[USA=United Space Alliance, NASA ES2, ES3=separate divisions of the Johnson Space Center Engineering Directorate, KSC=Kennedy Space Center, MOD=Missions Operations Directorate, or Mission Control]
Routing the request through the Engineering department led in part to it being viewed by Shuttle Program managers as a non-critical engineering desire rather than a critical operational need. Flight Day Seven, Wednesday, January 22, 2003 Conversations and log entries on Flight Day Seven document how three requests for images (Bob Page to Wayne Hale, Bob White to Lambert Austin, and Rodney Rocha to Paul Shack) were ultimately dismissed by the Mission Management Team, and how the order to halt those requests was then interpreted by the Debris Assessment Team as a direct and final denial of their request for imagery.
On the morning of Flight Day Seven, Wayne Hale responded to the earlier Flight Day Two request from Bob Page and a call from Lambert Austin on Flight Day Five, during which Austin mentioned that "some analysts" from the Debris Assessment Team were interested in getting imagery. Hale called a Department of Defense representative at Kennedy Space Center (who was not the designated Department of Defense official for coordinating imagery requests) and asked that the military start the planning process for imaging Columbia on orbit. Within an hour, the Defense Department representative at NASA contacted U.S. Strategic Command (USSTRATCOM) at Colorado's Cheyenne Mountain Air Force Station and asked what it would take to get imagery of Columbia on orbit. (This call was similar to Austin's call to the Department of Defense Manned Space Flight Support Office in that the caller characterized it as "information gathering" rather than a request for action.) A representative from the USSTRATCOM Plans Office initiated actions to identify ground-based and other imaging assets that could execute the request. Hale's earlier call to the Defense Department representative at Kennedy Space Center was placed without authorization from Mission Management Team Chair Linda Ham. Also, the call was made to a Department of Defense Representative who was not the designated liaison for handling such requests. In order to initiate the imagery request through official channels, Hale also called Phil Engelauf at the Mission Operations Directorate, told him he had started Defense Department action, and asked if Engelauf could have the Flight Dynamics Officer at Johnson Space Center make an official request to the Cheyenne Mountain Operations Center. Engelauf started to comply with Hale's request.
Report Volume I August 2003
After the Department of Defense representatives were called, Lambert Austin telephoned Linda Ham to inform her about the imagery requests that he and Hale had initiated. Austin also told Wayne Hale that he had asked Lieutenant Colonel Lee at the Department of Defense Manned Space Flight Support Office about what actions were necessary to get on-orbit imagery.
Mike Card, a NASA Headquarters manager from the Safety and Mission Assurance Office, called Mark Erminger at the Johnson Space Center Safety and Mission Assurance for Shuttle Safety Program and Bryan O'Connor, Associate Administrator for Safety and Mission Assurance, to discuss a potential Department of Defense imaging request. Erminger said that he was told this was an "in-family" event. O'Connor stated he would defer to Shuttle management in handling such a request. Despite two safety officials being contacted, one of whom was NASA's highest-ranking safety official, safety personnel took no actions to obtain imagery. The following is an 8:09 a.m. entry in the Mission Evaluation Room Console log. "We received a visit from Mission Manager/Vanessa Ellerbe and FD Office/Phil Engelauf regarding two items: (1) the MMT's action item to the MER to determine the impacts to the vehicle's 150 lbs of additional weight…and (2) Mr. Engelauf wants to know who is requesting the Air Force to look at the vehicle." [FD=Flight Director, MMT=Mission Management Team,
MER=Mission Evaluation Room]
At 8:30 a.m., the NASA Department of Defense liaison officer called USSTRATCOM and cancelled the request for imagery. The reason given for the cancellation was that NASA had identified its own in-house resources and no longer needed the military's help. The NASA request to the Department of Defense to prepare to image Columbia on-orbit was both made and rescinded within 90 minutes. The Board has determined that the following sequence of events likely occurred within that 90minute period. Linda Ham asked Lambert Austin if he knew who was requesting the imagery. After admitting his participation in helping to make the imagery request outside the official chain of command and without first gaining Ham's permission, Austin referred to his conversation with United Space Alliance Shuttle Integration manager Bob White on Flight Day Six, in which White had asked Austin, in response to White's Debris Assessment Team employee concerns, what it would take to get Orbiter imagery. Even though Austin had already informed Ham of the request for imagery, Ham later called Mission Management Team members Ralph Roe, Manager of the Space Shuttle Vehicle Engineering Office, Loren Shriver, United Space Alliance Deputy Program Manager for Shuttle, and David Moyer, the on-duty Mission Evaluation Room manager, to determine the origin of the request and to confirm that there was a "requirement" for a request. Ham also asked Flight Director Phil Engelauf if he had a "requirement" for imagery of Columbia's left wing. These individuals all stated that they had not requested imagery, were not aware of any "official" requests for imagery, and could not identify a "requirement" for imagery. Linda Ham later told several individuals that nobody had a requirement for imagery. What started as a request by the Intercenter Photo Working Group to seek outside help in obtaining images on Flight Day Two in anticipation of analysts' needs had become by Flight Day Six an actual engineering request by members of the Debris Assessment Team, made informally through Bob White to Lambert Austin, and formally in Rodney Rocha's e-mail to Paul Shack. These requests had then caused Lambert Austin and Wayne Hale to contact Department of Defense representatives. When Ham officially terminated the actions that the Department of Defense had begun, she effectively terminated both the Intercenter Photo Working Group request and the Debris Assessment Team request. While Ham has publicly stated she did not know of the Debris Assessment Team members' desire for imagery, she never asked them directly if the request was theirs, even though they were the team analyzing the foam strike. Also on Flight Day Seven, Ham raised concerns that the extra time spent maneuvering Columbia to make the left wing visible for imaging would unduly impact the mission schedule; for exReport Volume I August 2003
ample, science experiments would have to stop while the imagery was taken. According to personal notes obtained by the Board: "Linda Ham said it was no longer being pursued since even if we saw something, we couldn't do anything about it. The Program didn't want to spend the resources." Shuttle managers, including Ham, also said they were looking for very small areas on the Orbiter and that past imagery resolution was not very good. The Board notes that no individuals in the STS-107 operational chain of command had the security clearance necessary to know about National imaging capabilities. Additionally, no evidence has been uncovered that anyone from NASA, United Space Alliance, or Boeing sought to determine the expected quality of images and the difficulty and costs of obtaining Department of Defense assistance. Therefore, members of the Mission Management Team were making critical decisions about imagery capabilities based on little or no knowledge. The following is an entry in the Flight Director Handover Log. "NASA Resident Office, Peterson AFB called and SOI at USSPACECOM was officially turned off. This went all the way up to 4 star General. Post flight we will write a memo to USSPACECOM telling them whom they should take SOI requests from."50 [AFB=Air Force
Base, SOI=Spacecraft Object Identification, USSPACECOM=U.S. Space Command]
After canceling the Department of Defense imagery request, Linda Ham continued to explore whether foam strikes posed a safety of flight issue. She sent an e-mail to Lambert Austin and Ralph Roe.
-----Original Message--From: HAM, LINDA J. (JSC-MA2) (NASA) Sent: Wednesday, January 22, 2003 9:33 AM To: AUSTIN, LAMBERT D. (JSC-MS) (NASA); ROE, RALPH R. (JSC-MV) (NASA) Subject: ET Foam Loss
Can we say that for any ET foam lost, no 'safety of flight' damage can occur to the Orbiter because of the density?
[ET=External Tank]
Responses included the following.
-----Original Message----From: ROE, RALPH R. (JSC-MV) (NASA) Sent: Wednesday, January 22, 2003 9:38 AM To: SCHOMBURG, CALVIN (JSC-EA) (NASA) Subject: FW: ET Foam Loss
Calvin, I wouldn't think we could make such a generic statement but can we bound it some how by size or acreage?
[Acreage=larger areas of foam coverage]
Ron Dittermore e-mailed Linda Ham the following.
Report Volume I
August 2003
-----Original Message----From: DITTEMORE, RONALD D. (JSC-MA) (NASA) Sent: Wednesday, January 22, 2003 10:15 AM To: HAM, LINDA J. (JSC-MA2) (NASA) Subject: RE: ET Briefing - STS-112 Foam Loss
Another thought, we need to make sure that the density of the ET foam cannot damage the tile to where it is an impact to the orbiter...Lambert and Ralph need to get some folks working with ET.
The following is an e-mail from Calvin Schomburg to Ralph Roe.
-----Original Message----From: SCHOMBURG, CALVIN (JSC-EA) (NASA) Sent: Wednesday, January 22, 2003 10:53 AM To: ROE, RALPH R. (JSC-MV) (NASA) Subject: RE: ET Foam Loss
No-the amount of damage ET foam can cause to the TPS material-tiles is based on the amount of impact energy-the size of the piece and its velocity( from just after pad clear until about 120 secondsafter that it will not hit or it will not enough energy to cause any damage)-it is a pure kinetic problemthere is a size that can cause enough damage to a tile that enough of the material is lost that we could burn a hole through the skin and have a bad day-(loss of vehicle and crew -about 200-400 tile locations( out of the 23,000 on the lower surface)-the foam usually fails in small popcorn pieces-that is why it is vented-to make small hits-the two or three times we have been hit with a piece as large as the one this flight-we got a gouge about 8-10 inches long about 2 inches wide and 3/4 to an 1 inch deep across two or three tiles. That is what I expect this time-nothing worst. If that is all we get we have have no problem-will have to replace a couple of tiles but nothing else.
[ET=External Tank, TPS=Thermal Protection System]
The following is a response from Lambert Austin to Linda Ham.
-----Original Message----From: AUSTIN, LAMBERT D. (JSC-MS) (NASA) Sent: Wednesday, January 22, 2003 3:22 PM To: HAM, LINDA J. (JSC-MA2) (NASA) Cc: WALLACE, RODNEY O. (ROD) (JSC-MS2) (NASA); NOAH, DONALD S. (DON) (JSC-MS) (NASA) Subject: RE: ET Foam Loss
NO. I will cover some of the pertinent rationale....there could be more if I spent more time thinking about it. Recall this issue has been discussed from time to time since the inception of the basic "no debris" requirement in Vol. X and at each review the SSP has concluded that it is not possible to PRECLUDE a potential catastrophic event as a result of debris impact damage to the flight elements. As regards the Orbiter, both windows and tiles are areas of concern. You can talk to Cal Schomberg and he will verify the many times we have covered this in SSP reviews. While there is much tolerance to window and tile damage, ET foam loss can result in impact damage that under subsequent entry environments can lead to loss of structural integrity of the Orbiter area impacted or a penetration in a critical function area that results in loss of that function. My recollection of the most critical Orbiter bottom acreage areas are the wing spar, main landing gear door seal and RCC panels...of course Cal can give you a much better rundown. We can and have generated parametric impact zone characterizations for many areas of the Orbiter for a few of our more typical ET foam loss areas. Of course, the impact/damage significance is always a function of debris size and density, impact velocity, and impact angle--these latter 2 being a function of the flight time at which the ET foam becomes debris. For STS-107 specifically, we have generated [continued on next page] this info and provided it to Orbiter. Of course, even this is based on the ASSUMPTION that the location and size of the debris is the same as occurred on STS-112------this cannot be verified until we R e pphoto o r t Vevidence o l u m e I post A uOrbiter gust 2 003 receive the on-board ET separation landing. We are requesting that this 1 5 5 be expedited. I have the STS-107 Orbiter impact map based on the assumptions noted herein being
Orbiter area impacted or a penetration in a critical function area that results in loss of that function. My recollection of the most critical Orbiter bottom acreage areas are the wing spar, main landing gear COLUMBIA door seal and RCC panels...of course Cal can give you a much better rundown.
We can and have generated parametric impact zone characterizations for many areas of the Orbiter for a few of our more typical ET foam loss areas. Of course, the impact/damage significance is always a function of debris size and density, impact velocity, and impact angle--these latter 2 being a function [continued from previous page] of the flight time at which the ET foam becomes debris. For STS-107 specifically, we have generated this info and provided it to Orbiter. Of course, even this is based on the ASSUMPTION that the location and size of the debris is the same as occurred on STS-112------this cannot be verified until we receive the on-board ET separation photo evidence post Orbiter landing. We are requesting that this be expedited. I have the STS-107 Orbiter impact map based on the assumptions noted herein being sent down to you. Rod is in a review with Orbiter on this info right now.
[SSP=Space Shuttle Program, ET=External Tank]
The Board notes that these e-mail exchanges indicate that senior Mission Management Team managers, including the Shuttle Program Manager, Mission Management Team Chair, head of Space Shuttle Systems Integration, and a Shuttle tile expert, correctly identified the technical bounds of the foam strike problem and its potential seriousness. Mission managers understood that the relevant question was not whether foam posed a safety-of-flight issue – it did – but rather whether the observed foam strike contained sufficient kinetic energy to cause damage that could lead to a burn-through. Here, all the key managers were asking the right question and admitting the danger. They even identified RCC as a critical impact zone. Yet little followthrough occurred with either the request for imagery or the Debris Assessment Team analysis. (See Section 3.4 and 3.6 for details on the kinetics of foam strikes.) A Mission Evaluation Room log entry at 10:37 a.m. records the decision not to seek imaging of Columbia's left wing. "USA Program Manager/Loren Shriver, NASA Manager, Program Integration/Linda Ham, & NASA SSVEO/Ralph Roe have stated that there is no need for the Air Force to take a look at the vehicle." [USA=United Space Alliance, SSVEO=Space Shuttle Vehicle Engineering Office] At 11:22 a.m., Debris Assessment Team Co-Chair Pam Madera sent an e-mail to team members setting the agenda for the team's second formal meeting that afternoon that included: "… Discussion on Need/Rationale for Mandatory Viewing of damage site (All)…" Earlier e-mail agenda wording did not include "Need/Rationale for Mandatory" wording as listed here, which indicates that Madera knew of management's decision to not seek images of Columbia's left wing and anticipated having to articulate a "mandatory" rationale to reverse that decision. In fact, a United Space Alliance manager had informed Madera that imagery would be sought only if the request was a "mandatory need." Twenty-three minutes later, an e-mail from Paul Shack to Rodney Rocha, who the day before had carried forward the Debris Assessment Team's request for imaging, stated the following. "… FYI, According to the MER, Ralph Roe has told program that Orbiter is not requesting any outside imaging help …" [MER=Mission Evaluation Room] Earlier that morning, Ralph Roe's deputy manager, Trish Petite, had separate conversations with Paul Shack and tile expert Calvin Schomburg. In those conversations, Petite noted that an analysis of potential damage was in progress, and they should wait to see what the analysis showed before asking for imagery. Schomburg, though aware of the Debris Assessment Team's request for imaging, told Shack and Petite that he believed on-orbit imaging of potentially damaged areas was not necessary. As the morning wore on, Debris Assessment Team engineers, Shuttle Program management, and other NASA personnel exchanged e-mail. Most messages centered on technical matters to be discussed at the Debris Assessment Team's afternoon meeting, including debris density, computer-aided design models, and the highest angle of incidence to use for a particular material property. One e-mail from Rocha to his managers and other Johnson engineers at 11:19 a.m., included the following. "… there are good scenarios (acceptable and minimal damage) to horrible ones, depending on the extent of the damage incurred by the wing and location. The most critical loca156
Report Volume I August 2003
tions seem to be the 1191 wing spar region, the main landing gear door seal, and the RCC panels. We do not know yet the exact extent or nature of the damage without being provided better images, and without such all the high powered analyses and assessments in work will retain significant uncertainties …" Second Debris Assessment Team Meeting Some but not all of the engineers attending the Debris Assessment Team's second meeting had learned that the Shuttle Program was not pursuing imaging of potentially damaged areas. What team members did not realize was the Shuttle Program's decision not to seek on-orbit imagery was not necessarily a direct and final response to their request. Rather, the "no" was partly in response to the Kennedy Space Center action initiated by United Space Alliance engineers and managers and finally by Wayne Hale. Not knowing that this was the case, Debris Assessment Team members speculated as to why their request was rejected and whether their analysis was worth pursuing without new imagery. Discussion then moved on to whether the Debris Assessment Team had a "mandatory need" for Department of Defense imaging. Most team members, when asked by the Board what "mandatory need" meant, replied with a shrug of their shoulders. They believed the need for imagery was obvious: without better pictures, engineers would be unable to make reliable predictions of the depth and area of damage caused by a foam strike that was outside of the experience base. However, team members concluded that although their need was important, they could not cite a "mandatory" requirement for the request. Analysts on the Debris Assessment Team were in the unenviable position of wanting images to more accurately assess damage while simultaneously needing to prove to Program managers, as a result of their assessment, that there was a need for images in the first place. After the meeting adjourned, Rocha read the 11:45 a.m. e-mail from Paul Shack, which said that the Orbiter Project was not requesting any outside imaging help. Rocha called Shack to ask if Shack's boss, Johnson Space Center engineering director Frank Benz, knew about the request. Rocha then sent several e-mails consisting of questions about the ongoing analyses and details on the Shuttle Program's cancellation of the imaging request. An e-mail that he did not send but instead printed out and shared with a colleague follows. "In my humble technical opinion, this is the wrong (and bordering on irresponsible) answer from the SSP and Orbiter not to request additional imaging help from any outside source. I must emphasize (again) that severe enough damage (3 or 4 multiple tiles knocked out down to the densification layer) combined with the heating and resulting damage to the underlying structure at the most critical location (viz., MLG door/wheels/tires/hydraulics or the X1191 spar cap) could present potentially grave hazards. The engineering team will admit it might not achieve definitive high confidence answers without additional images, but, without action to request help to clarify the damage visually, we will guarantee it will not. Can we talk to Frank Benz before Friday's MMT? Remember the NASA safety posters everywhere around stating, ʻIf it's not safe, say so'? Yes, it's that serious." [SSP=Space
Shuttle Program, MLG=Main Landing Gear, MMT=Mission Management Team]
When asked why he did not send this e-mail, Rocha replied that he did not want to jump the chain of command. Having already raised the need to have the Orbiter imaged with Shack, he would defer to management's judgment on obtaining imagery. Even after the imagery request had been cancelled by Program management, engineers in the Debris Assessment Team and Mission Control continued to analyze the foam strike. A structural engineer in the Mechanical, Maintenance, Arm and Crew Systems sent an e-mail to a flight dynamics engineer that stated: "There is lots of speculation as to extent of the damage, and we could get a burn through into the wheel well upon entry." Less than an hour later, at 6:09 p.m., a Mission Evaluation Room Console log entry stated the following. "MMACS is trying to view a Quicktime movie on the debris impact but doesn't have QuickReport Volume I August 2003
time software on his console. He needs either an avi, mpeg file or a vhs tape. He is asking us for help." [MMACS=Mechanical, Maintenance, Arm and Crew Systems] The controller at the Mechanical, Maintenance, Arm and Crew Systems console would be among the first in Mission Control to see indications of burn-through during Columbia's re-entry on the morning of February 1. This log entry also indicates that Mission Control personnel were aware of the strike. Flight Day Eight, Thursday, January 23, 2003 The morning after Shuttle Program Management decided not to pursue on-orbit imagery, Rodney Rocha received a return call from Mission Operations Directorate representative Barbara Conte to discuss what kinds of imaging capabilities were available for STS-107.
Conte explained to Rocha that the Mission Operations Directorate at Johnson did have U.S. Air Force standard services for imaging the Shuttle during Solid Rocket Booster separation and External Tank separation. Conte explained that the Orbiter would probably have to fly over Hawaii to be imaged. The Board notes that this statement illustrates an unfamiliarity with National imaging assets. Hawaii is only one of many sites where relevant assets are based. Conte asked Rocha if he wanted her to pursue such a request through Missions Operations Directorate channels. Rocha said no, because he believed Program managers would still have to support such a request. Since they had already decided that imaging of potentially damaged areas was not necessary, Rocha thought it unlikely that the Debris Assessment Team could convince them otherwise without definitive data. Later that day, Conte and another Mission Operations Directorate representative were attending an unrelated meeting with Leroy Cain, the STS-107 ascent/entry Flight Director. At that meeting, they conveyed Rocha's concern to Cain and offered to help with obtaining imaging. After checking with Phil Engelauf, Cain distributed the following e-mail.
-----Original Message----From: CAIN, LEROY E. (JSC-DA8) (NASA) Sent: Thursday, January 23, 2003 12:07 PM To: JONES, RICHARD S. (JSC-DM) (NASA); OLIVER, GREGORY T. (GREG) (JSC-DM4) (NASA); CONTE, BARBARA A. (JSC-DM) (NASA) Cc: ENGELAUF, PHILIP L. (JSC-DA8) (NASA); AUSTIN, BRYAN P. (JSC-DA8) (NASA); BECK, KELLY B. (JSC-DA8) (NASA); HANLEY, JEFFREY M. (JEFF) (JSC-DA8) (NASA); STICH, J. S. (STEVE) (JSC-DA8) (NASA) Subject: Help with debris hit
The SSP was asked directly if they had any interest/desire in requesting resources outside of NASA to view the Orbiter (ref. the wing leading edge debris concern). They said, No. After talking to Phil, I consider it to be a dead issue.
[SSP=Space Shuttle Program]
Also on Flight Day Eight, Debris Assessment Team engineers presented their final debris trajectory estimates to their NASA, United Space Alliance, and Boeing managers. These estimates formed the basis for predicting the Orbiter's damaged areas as well as the extent of damage, which in turn determined the ultimate threat to the Orbiter during re-entry. Mission Control personnel thought they should tell Commander Rick Husband and Pilot William McCool about the debris strike, not because they thought that it was worthy of the crew's attention but because the crew might be asked about it in an upcoming media interview. Flight Director Steve Stitch sent the following e-mail to Husband and McCool and copied other Flight Directors.
Report Volume I August 2003
-----Original Message----From: STICH, J. S. (STEVE) (JSC-DA8) (NASA) Sent: Thursday, January 23, 2003 11:13 PM To: CDR; PLT Cc: BECK, KELLY B. (JSC-DA8) (NASA); ENGELAUF, PHILIP L. (JSC-DA8) (NASA); CAIN, LEROY E. (JSC-DA8) (NASA); HANLEY, JEFFREY M. (JEFF) (JSC-DA8) (NASA); AUSTIN, BRYAN P. (JSC-DA8) (NASA) Subject: INFO: Possible PAO Event Question
Rick and Willie, You guys are doing a fantastic job staying on the timeline and accomplishing great science. Keep up the good work and let us know if there is anything that we can do better from an MCC/POCC standpoint. There is one item that I would like to make you aware of for the upcoming PAO event on Blue FD 10 and for future PAO events later in the mission. This item is not even worth mentioning other than wanting to make sure that you are not surprised by it in a question from a reporter. During ascent at approximately 80 seconds, photo analysis shows that some debris from the area of the -Y ET Bipod Attach Point came loose and subsequently impacted the orbiter left wing, in the area of transition from Chine to Main Wing, creating a shower of smaller particles. The impact appears to be totally on the lower surface and no particles are seen to traverse over the upper surface of the wing. Experts have reviewed the high speed photography and there is no concern for RCC or tile damage. We have seen this same phenomenon on several other flights and there is absolutely no concern for entry. That is all for now. It's a pleasure working with you every day.
[MCC/POCC=Mission Control Center/Payload Operations Control Center, PAO=Public Affairs Officer, FD 10=Flight Day Ten, -Y=left, ET=External Tank]
This e-mail was followed by another to the crew with an attachment of the video showing the debris impact. Husband acknowledged receipt of these messages. Later, a NASA liaison to USSTRATCOM sent an e-mail thanking personnel for the prompt response to the imagery request. The e-mail asked that they help NASA observe "official channels" for this type of support in the future. Excerpts from this message follow. "Let me assure you that, as of yesterday afternoon, the Shuttle was in excellent shape, mission objectives were being performed, and that there were no major debris system problems identified. The request that you received was based on a piece of debris, most likely ice or insulation from the ET, that came off shortly after launch and hit the underside of the vehicle. Even though this is not a common occurrence it is something that has happened before and is not considered to be a major problem. The one problem that this has identified is the need for some additional coordination within NASA to assure that when a request is made it is done through the official channels. The NASA/ USSTRAT (USSPACE) MOA identifies the need for this type of support and that it will be provided by USSTRAT. Procedures have been long established that identifies the Flight Dynamics Officer (for the Shuttle) and the Trajectory Operations Officer (for the International Space Station) as the POCs to work these issues with the personnel in Cheyenne Mountain. One of the primary purposes for this chain is to make sure that requests like this one does not slip through the system and spin the community up about potential problems that have not been fully vetted through the proper channels. Two things that you can help us with is to make sure that future requests of this sort are confirmed through the proper channels. For the Shuttle it is via CMOC to the Flight Dynamics Officer. For the International Space Station it is via CMOC to the Trajectory Operations Officer. The second request is that no resources are spent unless the request has been confirmed. These requests are not meant to diminish the responsibilities of the DDMS office or to change any previous agreements but to eliminate the confusion that can be caused by a lack of proper coordination." [ET=External Tank,
Report Volume I August 2003
MOA=Memorandum of Agreement, POC=Point of Contact, CMOC=Cheyenne Mountain Operations Center, DDMS=Department of Defense Manned Space Flight Support Office]
Third Debris Assessment Team Meeting The Debris Assessment Team met for the third time Thursday afternoon to review updated impact analyses. Engineers noted that there were no alternate re-entry trajectories that the Orbiter could fly to substantially reduce heating in the general area of the foam strike. Engineers also presented final debris trajectory data that included three debris size estimates to cover the continuing uncertainty about the size of the debris. Team members were told that imaging would not be forthcoming. In the face of this denial, the team discussed whether to include a presentation slide supporting their desire for images of the potentially damaged area. Many still felt it was a valid request and wanted their concerns aired at the upcoming Mission Evaluation Room brief and then at the Mission Management Team level. Eventually, the idea of including a presentation slide about the imaging request was dropped. Just prior to attending the third assessment meeting, tile expert Calvin Schomburg and Rodney Rocha met to discuss foam impacts from other missions. Schomburg implied that the STS-107 foam impact was in the Orbiter's experience base and represented only a maintenance issue. Rocha disagreed and argued about the potential for burn-through on re-entry. Calvin Schomburg stated a belief that if there was severe damage to the tiles, "nothing could be done." (See Section 6.4.) Both then joined the meeting already in progress. According to Boeing analysts who were members of the Debris Assessment Team, Schomburg called to ask about their rationale for pursuing imagery. The Boeing analysts told him that something the size of a large cooler had hit the Orbiter at 500 miles per hour. Pressed for additional reasons and not fully understanding why their original justification was insufficient, the analysts said that at least they would know what happened if something were to go terribly wrong. The Boeing analysts next asked why they were working so hard analyzing potential damage areas if Shuttle Program management believed the damage was minor and that no safety-of-flight issues existed. Schomburg replied that the analysts were new and would learn from this exercise. Flight Day Nine, Friday, January 24, 2003 At 7:00 a.m., Boeing and United Space Alliance contract personnel presented the Debris Assessment Team's findings to Don McCormack, the Mission Evaluation Room manager. In yet another signal that working engineers and mission personnel shared a high level of concern for Columbia's condition, so many engineers crowded the briefing room that it was standing room only, with people lining the hallway. The presentation included viewgraphs that discussed the team's analytical methodology and five scenarios for debris damage, each based on different estimates of debris size and impact point. A sixth scenario had not yet been completed, but early indications suggested that it would not differ significantly from the other five. Each case was presented with a general overview of transport mechanics, results from the Crater modeling, aerothermal considerations, and predicted thermal and structural effects for Columbia's re-entry. The briefing focused primarily on potential damage to the tiles, not the RCC panels. (An analysis of how the poor construction of these viewgraphs effectively minimized key assumptions and uncertainties is presented in Chapter 7.) While the team members were confident that they had conducted the analysis properly – within the limitations of the information they had – they stressed that many uncertainties remained. First, there was great uncertainty about where the debris had struck. Second, Crater, the analytical tool they used to predict the penetration depth of debris impact, was being used on a piece of debris that was 400 times larger than the standard in Boeing's database. (At the time, the team believed that the debris was 640 times larger.) Engineers ultimately concluded that their analysis, limited as it was, did not show that a safety-of-flight issue existed. Engineers who attended this briefing indicated a belief that management focused on the answer – that analysis proved there was no safety-of-flight issue – rather than concerns about the large uncertainties that may have undermined the analysis that provided that answer.
Report Volume I
August 2003
At the Mission Management Team's 8:00 a.m. meeting, Mission Evaluation Room manager Don McCormack verbally summarized the Debris Assessment Team's 7:00 a.m. brief. It was the third topic discussed. Unlike the earlier briefing, McCormack's presentation did not include the Debris Assessment Team's presentation charts. The Board notes that no supporting analysis or examination of minority engineering views was asked for or offered, that neither Mission Evaluation Room nor Mission Management Team members requested a technical paper of the Debris Assessment Team analysis, and that no technical questions were asked. January 24, 2003, Mission Management Team Meeting Transcript The following is a transcript of McCormack's verbal briefing to the Mission Management Team, which Linda Ham Chaired. Early in the meeting, Phil Engelauf, Chief of the Flight Director's office, reported that he had made clear in an e-mail to Columbia's crew that there were "no concerns" that the debris strike had caused serious damage. The Board notes that this conclusion about whether the debris strike posed a safety-of-flight issue was presented to Mission Management Team members before they discussed the debris strike damage assessment. Engelauf: "I will say that crew did send down a note last night asking if anybody is talking about extension days or going to go with that and we sent up to the crew about a 15 second video clip of the strike just so they are armed if they get any questions at the press conferences or that sort of thing, but we made it very clear to them no, no concerns." Linda Ham: "When is the press conference? Is it today?" Engelauf: "It's later today." Ham: "They may get asked because the press is aware of it." Engelauf: "The press is aware of it I know folks have asked me because the press corps at the cape have been asking…wanted to make sure they were properly…" Ham: "Okay, back on the temperature…" The meeting went on for another 25 minutes. Other mission-related subjects were discussed before team members returned to the debris strike. Ham: "Go ahead, Don." Don McCormack: "Okay. And also we've received the data from the systems integration guys of the potential ranges of sizes and impact angles and where it might have hit. And the guys have gone off and done an analysis, they use a tool they refer to as Crater which is their official evaluation tool to determine the potential size of the damage. So they went off and done all that work and they've done thermal analysis to the areas where there may be damaged tiles. The analysis is not complete. There is one case yet that they wish to run, but kind of just jumping to the conclusion of all that, they do show that, obviously, a potential for significant tile damage here, but thermal analysis does not indicate that there is potential for a burn-through. I mean there could be localized heating damage. There is… obviously there is a lot of uncertainty in all this in terms of the size of the debris and where it hit and the angle of incidence." Ham: "No burn through, means no catastrophic damage and the localized heating damage would mean a tile replacement?" McCormack: "Right, it would mean possible impacts to turnaround repairs and that sort of thing, but we do not see any kind of safety of flight issue here yet in anything that we've looked at." Ham: "And no safety of flight, no issue for this mission, nothing that we're going to do different, there may be a turnaround." McCormack: "Right, it could potentially hit the RCC and we don't indicate any other possible coating damage or something, we don't see any issue if it hit the RCC. Although we could have some significant tile damage if we don't see a safety-of-flight issue."
Report Volume I August 2003
Ham: "What do you mean by that?" McCormack: "Well it could be down through the … we could lose an entire tile and then the ramp into and out of that, I mean it could be a significant area of tile damage down to the SIP perhaps, so it could be a significant piece missing, but…" [SIP refers to the denser lower layers of
tile to which the debris may have penetrated.]
Ham.: "It would be a turnaround issue only?" McCormack: "Right." (Unintelligible speaker) At this point, tile expert Calvin Schomburg states his belief that no safety-of-flight issue exists. However, some participants listening via teleconference to the meeting are unable to hear his comments. Ham: "Okay. Same thing you told me about the other day in my office. We've seen pieces of this size before haven't we?" Unknown speaker. "Hey Linda, we're missing part of that conversation." Ham: "Right." Unknown speaker: "Linda, we can't hear the speaker." Ham: "He was just reiterating with Calvin that he doesn't believe that there is any burn-through so no safety of flight kind of issue, it's more of a turnaround issue similar to what we've had on other flights. That's it? Alright, any questions on that?" The Board notes that when the official minutes of the January 24 Mission Management Team were produced and distributed, there was no mention of the debris strike. These minutes were approved and signed by Frank Moreno, STS-107 Lead Payload Integration Manager, and Linda Ham. For anyone not present at the January 24 Mission Management Team who was relying on the minutes to update them on key issues, they would have read nothing about the debris-strike discussions between Don McCormack and Linda Ham. A subsequent 8:59 a.m. Mission Evaluation Room console log entry follows. "MMT Summary…McCormack also summarized the debris assessment. Bottom line is that there appears to be no safety of flight issue, but good chance of turnaround impact to repair tile damage." [MMT=Mission Management Team] Flight Day 10 through 16, Saturday through Friday, January 25 through 31, 2003 Although "no safety-of-flight issue" had officially been noted in the Mission Evaluation Room log, the Debris Assessment Team was still working on parts of its analysis of potential damage to the wing and main landing gear door. On Sunday, January 26, Rodney Rocha spoke with a Boeing thermal analyst and a Boeing stress analyst by telephone to express his concern about the Debris Assessment Team's overall analysis, as well as the remaining work on the main landing gear door analysis. After the Boeing engineers stated their confidence with their analyses, Rocha became more comfortable with the damage assessment and sent the following e-mail to his management.
Report Volume I
August 2003
-----Original Message----From: ROCHA, ALAN R. (RODNEY) (JSC-ES2) (NASA) Sent: Sunday, January 26, 2003 7:45 PM To: SHACK, PAUL E. (JSC-EA42) (NASA); MCCORMACK, DONALD L. (DON) (JSC-MV6) (NASA); OUELLETTE, FRED A. (JSC-MV6) (NASA) Cc: ROGERS, JOSEPH E. (JOE) (JSC-ES2) (NASA); GALBREATH, GREGORY F. (GREG) (JSC-ES2) (NASA); JACOBS, JEREMY B. (JSC-ES4) (NASA); SERIALE-GRUSH, JOYCE M. (JSC-EA) (NASA); KRAMER, JULIE A. (JSC-EA4) (NASA); CURRY, DONALD M. (JSC-ES3) (NASA); KOWAL, T. J. (JOHN) (JSC-ES3) (NASA); RICKMAN, STEVEN L. (JSC-ES3) (NASA); SCHOMBURG, CALVIN (JSC-EA) (NASA); CAMPBELL, CARLISLE C., JR (JSC-ES2) (NASA) Subject: STS-107 Wing Debris Impact on Ascent: Final analysis case completed
As you recall from Friday's briefing to the MER, there remained open work to assess analytically predicted impact damage to the wing underside in the region of the main landing gear door. This area was considered a low probability hit area by the image analysis teams, but they admitted a debris strike here could not be ruled out. As with the other analyses performed and reported on Friday, this assessment by the Boeing multitechnical discipline engineering teams also employed the system integration's dispersed trajectories followed by serial results from the Crater damage prediction tool, thermal analysis, and stress analysis. It was reviewed and accepted by the ES-DCE (R. Rocha) by Sunday morning, Jan. 26. The case is defined by a large area gouge about 7 inch wide and about 30 inch long with sloped sides like a crater, and reaching down to the densified layer of the TPS. SUMMARY: Though this case predicted some higher temperatures at the outer layer of the honeycomb aluminum face sheet and subsequent debonding of the sheet, there is no predicted burnthrough of the door, no breeching of the thermal and gas seals, nor is there door structural deformation or thermal warpage to open the seal to hot plasma intrusion. Though degradation of the TPS and door structure is likely (if the impact occurred here), there is no safety of flight (entry, descent, landing) issue. Note to Don M. and Fred O.: On Friday I believe the MER was thoroughly briefed and it was clear that open work remained (viz., the case summarized above), the message of open work was not clearly given, in my opinion, to Linda Ham at the MMT. I believe we left her the impression that engineering assessments and cases were all finished and we could state with finality no safety of flight issues or questions remaining. This very serious case could not be ruled out and it was a very good thing we carried it through to a finish.
Rodney Rocha (ES2)
• Division Shuttle Chief Engineer (DCE), ES-Structural Engineering Division • Chair, Space Shuttle Loads & Dynamics Panel
[MER=Mission Evaluation Room, ES-DCE=Structural Engineering-Division Shuttle Chief Engineer, TPS=Thermal Protection System]
In response to this e-mail, Don McCormack told Rocha that he would make sure to correct Linda Ham's possible misconception that the Debris Assessment Team's analysis was finished as of the briefing to the Mission Management Team. McCormack informed Ham at the next Mission Management Team meeting on January 27, that the damage assessment had in fact been ongoing and that their final conclusion was that no safety-of-flight issue existed. The debris strike, in the official estimation of the Debris Assessment Team, amounted to only a postlanding turn-around maintenance issue. On Monday morning, January 27, Doug Drewry, a structural engineering manager from Johnson Space Center, summoned several Johnson engineers and Rocha to his office and asked them if they all agreed with the completed analyses and with the conclusion that no safety-of-flight issues existed. Although all participants agreed with that conclusion, they also knew that the Debris Assessment Team members and most structural engineers at Johnson still wanted images of Columbia's left wing but had given up trying to make that desire fit the "mandatory" requirement that Shuttle management had set.
Report Volume I
August 2003
Langley Research Center Although the Debris Analysis Team had completed its analysis and rendered a "no safety-offlight" verdict, concern persisted among engineers elsewhere at NASA as they learned about the debris strike and potential damage. On Monday, January 27, Carlisle Campbell, the design engineer responsible for landing gear/tires/brakes at Johnson Space Center forwarded Rodney Rocha's January 26, e-mail to Bob Daugherty, an engineer at Langley Research Center who specialized in landing gear design. Engineers at Langley and Ames Research Center and Johnson Space Center did not entertain the possibility of Columbia breaking up during re-entry, but rather focused on the idea that landing might not be safe, and that the crew might need to "ditch" the vehicle (crash land in water) or be prepared to land with damaged landing gear. Campbell initially contacted Daugherty to ask his opinion of the arguments used to declare the debris strike "not a safety-of-flight issue." Campbell commented that someone had brought up worst-case scenarios in which a breach in the main landing gear door causes two tires to go flat. To help Daugherty understand the problem, Campbell forwarded him e-mails, briefing slides, and film clips from the debris damage analysis. Both engineers felt that the potential ramifications of landing with two flat tires had not been sufficiently explored. They discussed using Shuttle simulator facilities at Ames Research Center to simulate a landing with two flat tires, but initially ruled it out because there was no formal request from the Mission Management Team to work the problem. Because astronauts were training in the Ames simulation facility, the two engineers looked into conducting the simulations after hours. Daugherty contacted his management on Tuesday, January 28, to update them on the plan for after-hours simulations. He reviewed previous data runs, current simulation results, and prepared scenarios that could result from main landing gear problems. The simulated landings with two flat tires that Daugherty eventually conducted indicated that it was a survivable but very serious malfunction. Of the various scenarios he prepared, Daugherty shared the most unfavorable only with his management and selected Johnson Space Center engineers. In contrast, his favorable simulation results were forwarded to a wider Johnson audience for review, including Rodney Rocha and other Debris Assessment Team members. The Board is disappointed that Daugherty's favorable scenarios received a wider distribution than his discovery of a potentially serious malfunction, and also does not approve of the reticence that he and his managers displayed in not notifying the Mission Management Team of their concerns or his assumption that they could not displace astronauts who were training in the Ames simulator. At 4:36 p.m. on Monday, January 27, Daugherty sent the following to Campbell.
-----Original Message----From: Robert H. Daugherty Sent: Monday, January 27, 2003 3:35 PM To: CAMPBELL, CARLISLE C., JR (JSC-ES2) (NASA) Subject: Video you sent
WOW!!! I bet there are a few pucker strings pulled tight around there! Thinking about a belly landing versus bailout...... (I would say that if there is a question about main gear well burn thru that its crazy to even hit the deploy gear button...the reason being that you might have failed the wheels since they are aluminum..they will fail before the tire heating/pressure makes them fail..and you will send debris all over the wheel well making it a possibility that the gear would not even deploy due to ancillary damage...300 feet is the wrong altitude to find out you have one gear down and the other not down...you're dead in that case) Think about the pitch-down moment for a belly landing when hitting not the main gear but the trailing edge of the wing or body flap when landing gear up...even if you come in fast and at slightly less pitch attitude...the nose slapdown with that pitching moment arm seems to me to be pretty scary...so much so that I would bail out before I would let a loved one land like that. My two cents. See ya, Bob
Report Volume I August 2003
The following reply from Campbell to Daugherty was sent at 4:49 p.m.
-----Original Message----From: "CAMPBELL, CARLISLE C., JR (JSC-ES2) (NASA)" To: "'Bob Daugherty'" Subject: FW: Video you sent Date: Mon, 27 Jan 2003 15:59:53 -0600 X-Mailer: ßInternet Mail Service (5.5.2653.19)
Thanks. That's why they need to get all the facts in early on--such as look at impact damage from the spy telescope. Even then, we may not know the real effect of the damage. The LaRC ditching model tests 20 some years ago showed that the Orbiter was the best ditching shape that they had ever tested, of many. But, our structures people have said that if we ditch we would blow such big holes in the lower panels that the orbiter might break up. Anyway, they refuse to even consider water ditching any more--I still have the test results[ Bailout seems best.
[LaRC=Langley Research Center]
On the next day, Tuesday, Daugherty sent the following to Campbell.
-----Original Message----From: Robert H. Daugherty Sent: Tuesday, January 28, 2003 12:39 PM To: CAMPBELL, CARLISLE C., JR (JSC-ES2) (NASA) Subject: Tile Damage
Any more activity today on the tile damage or are people just relegated to crossing their fingers and hoping for the best? See ya, Bob Campbell's reply:
-----Original Message----From: "CAMPBELL, CARLISLE C., JR (JSC-ES2) (NASA)" To: "'Robert H. Daugherty'" Subject: RE: Tile Damage Date: Tue, 28 Jan 2003 13:29:58 -0600 X-Mailer: Internet Mail Service (5.5.2653.19)
I have not heard anything new. I'll let you know if I do. CCC
Carlisle Campbell sent the following e-mail to Johnson Space Center engineering managers on January 31. "In order to alleviate concerns regarding the worst case scenario which could potentially be caused by the debris impact under the Orbiter's left wing during launch, EG conducted some landing simulations on the Ames Vertical Motion Simulator which tested the ability of the crew and vehicle to survive a condition where two main gear tires are deflated before landing. The results, although limited, showed that this condition is controllable, including the nose slap down rates. These results may give MOD a different decision path should this scenario become a reality. Previous opinions were that bailout was the only answer."
[EG=Aeroscience and Flight Mechanics Division, MOD=Mission Operations Directorate]
Report Volume I August 2003
In the Mission Evaluation Room, a safety representative from Science Applications International Corporation, NASA's contract safety company, made a log entry at the Safety and Quality Assurance console on January 28, at 12:15 p.m. It was only the second mention of the debris strike in the safety console log during the mission (the first was also minor). "[MCC SAIC] called asking if any SR&QA people were involved in the decision to say that the ascent debris hit (left wing) is safe. [SAIC engineer] has indeed been involved in the analysis and stated that he concurs with the analysis. Details about the debris hit are found in the Flight Day 12 MER Manager and our Daily Report." [MCC=Mission Control Center,
SAIC=Science Applications International Corporation, SR&QA=Safety, Reliability, and Quality Assurance, MER=Mission Evaluation Room]
According to a Memorandum for the Record written by William Readdy, Associate Administrator for Space Flight, Readdy and Michael Card, from NASA's Safety and Mission Assurance Office, discussed an offer of Department of Defense imagery support for Columbia. This January 29, conversation ended with Readdy telling Card that NASA would accept the offer but because the Mission Management Team had concluded that this was not a safety-of-flight issue, the imagery should be gathered only on a low priority "not-to-interfere" basis. Ultimately, no imagery was taken. The Board notes that at the January 31, Mission Management Team meeting, there was only a minor mention of the debris strike. Other issues discussed included onboard crew consumables, the status of the leaking water separator, an intercom anomaly, SPACEHAB water flow rates, an update of the status of onboard experiments, end-of-mission weight concerns, landing day weather forecasts, and landing opportunities. The only mention of the debris strike was a brief comment by Bob Page, representing Kennedy Space Center's Launch Integration Office, who stated that the crew's hand-held cameras and External Tank films would be expedited to Marshall Space Flight Center via the Shuttle Training Aircraft for post-flight foam/debris imagery analysis, per Linda Ham's request. Summary: Mission Management Decision Making Discovery and Initial Analysis of Debris Strike In the course of examining film and video images of Columbia's ascent, the Intercenter Photo Working Group identified, on the day after launch, a large debris strike to the leading edge of Columbia's left wing. Alarmed at seeing so severe a hit so late in ascent, and at not having a clear view of damage the strike might have caused, Intercenter Photo Working Group members alerted senior Program managers by phone and sent a digitized clip of the strike to hundreds of NASA personnel via e-mail. These actions initiated a contingency plan that brought together an interdisciplinary group of experts from NASA, Boeing, and the United Space Alliance to analyze the strike. So concerned were Intercenter Photo Working Group personnel that on the day they discovered the debris strike, they tapped their Chair, Bob Page, to see through a request to image the left wing with Department of Defense assets in anticipation of analysts needing these images to better determine potential damage. By the Board's count, this would be the first of three requests to secure imagery of Columbia on-orbit during the 16-day mission.
1. Flight Day 2. Bob Page, Chair, Intercenter Photo Working Group to Wayne Hale, Shuttle Program Manager for Launch Integration at Kennedy Space Center (in person). 2. Flight Day 6. Bob White, United Space Alliance manager, to Lambert Austin, head of the Space Shuttle Systems Integration at Johnson Space Center (by phone). 3. Flight Day 6. Rodney Rocha, Co-Chair of Debris Assessment Team to Paul Shack, Manager, Shuttle Engineering Office (by e-mail).
Report Volume I
August 2003
1. Flight Day 4. Rodney Rocha inquires if crew has been asked to inspect for damage. No response. 2. Flight Day 6. Mission Control fails to ask crew member David Brown to downlink video he took of External Tank separation, which may have revealed missing bipod foam. 3. Flight Day 6. NASA and National Imagery and Mapping Agency personnel discuss possible request for imagery. No action taken. 4. Flight Day 7. Wayne Hale phones Department of Defense representative, who begins identifying imaging assets, only to be stopped per Linda Ham's orders. 5. Flight Day 7. Mike Card, a NASA Headquarters manager from the Safety and Mission Assurance Office, discusses imagery request with Mark Erminger, Johnson Space Center Safety and Mission Assurance. No action taken. 6. Flight Day 7. Mike Card discusses imagery request with Bryan O'Connor, Associate Administrator for Safety and Mission Assurance. No action taken. 7. Flight Day 8. Barbara Conte, after discussing imagery request with Rodney Rocha, calls LeRoy Cain, the STS-107 ascent/entry Flight Director. Cain checks with Phil Engelauf, and then delivers a "no" answer. 8. Flight Day 14. Michael Card, from NASA's Safety and Mission Assurance Office, discusses the imaging request with William Readdy, Associate Administrator for Space Flight. Readdy directs that imagery should only be gathered on a "not-to-interfere" basis. None was forthcoming.
Upon learning of the debris strike on Flight Day Two, the responsible system area manager from United Space Alliance and her NASA counterpart formed a team to analyze the debris strike in accordance with mission rules requiring the careful examination of any "out-of-family" event. Using film from the Intercenter Photo Working Group, Boeing systems integration analysts prepared a preliminary analysis that afternoon. (Initial estimates of debris size and speed, origin of debris, and point of impact would later prove remarkably accurate.) As Flight Day Three and Four unfolded over the Martin Luther King Jr. holiday weekend, engineers began their analysis. One Boeing analyst used Crater, a mathematical prediction tool, to assess possible damage to the Thermal Protection System. Analysis predicted tile damage deeper than the actual tile depth, and penetration of the RCC coating at impact angles above 15 degrees. This suggested the potential for a burn-through during re-entry. Debris Assessment Team members judged that the actual damage would not be as severe as predicted because of the inherent conservatism in the Crater model and because, in the case of tile, Crater does not take into account the tile's stronger and more impact-resistant "densified" layer, and in the case of RCC, the lower density of foam would preclude penetration at impact angles under 21 degrees. On Flight Day Five, impact assessment results for tile and RCC were presented at an informal meeting of the Debris Assessment Team, which was operating without direct Shuttle Program or Mission Management leadership. Mission Control's engineering support, the Mission Evaluation Room, provided no direction for team activities other than to request the team's results by January 24. As the problem was being worked, Shuttle managers did not formally direct the actions of or consult with Debris Assessment Team leaders about the team's assumptions, uncertainties, progress, or interim results, an unusual circumstance given that NASA managers are normally engaged in analyzing what they view as problems. At this meeting, participants agreed that an image of the area of the wing in question was essential to refine their analysis and reduce the uncertainties in their damage assessment. Each member supported the idea to seek imagery from an outside source. Due in part to a lack of guidance from the Mission Management Team or Mission Evaluation Room managers, the Debris Assessment Team chose an unconventional route for its request. Rather than working the request up the normal chain of command – through the Mission Evaluation Room to the Mission Management Team for action to Mission Control – team members nominated Rodney Rocha, the team's Co-Chair, to pursue the request through the Engineering Directorate at Johnson Space Center. As a result, even after the accident the Debris Assessment Team's request was viewed by Shuttle Program managers as a non-critical engineering desire rather than a critical operational need.
Report Volume I
August 2003
When the team learned that the Mission Management Team was not pursuing on-orbit imaging, members were concerned. What Debris Assessment Team members did not realize was the negative response from the Program was not necessarily a direct and final response to their official request. Rather, the "no" was in part a response to requests for imagery initiated by the Intercenter Photo Working Group at Kennedy on Flight Day 2 in anticipation of analysts' needs that had become by Flight Day 6 an actual engineering request by the Debris Assessment Team, made informally through Bob White to Lambert Austin, and formally through Rodney Rocha's e-mail to Paul Shack. Even after learning that the Shuttle Program was not going to provide the team with imagery, some members sought information on how to obtain it anyway. Debris Assessment Team members believed that imaging of potentially damaged areas was necessary even after the January 24, Mission Management Team meeting, where they had reported their results. Why they did not directly approach Shuttle Program managers and share their concern and uncertainty, and why Shuttle Program managers claimed to be isolated from engineers, are points that the Board labored to understand. Several reasons for this communications failure relate to NASA's internal culture and the climate established by Shuttle Program management, which are discussed in more detail in Chapters 7 and 8. A Flawed Analysis An inexperienced team, using a mathematical tool that was not designed to assess an impact of this estimated size, performed the analysis of the potential effect of the debris impact. Crater was designed for "in-family" impact events and was intended for day-of-launch analysis of debris impacts. It was not intended for large projectiles like those observed on STS-107. Crater initially predicted possible damage, but the Debris Assessment Team assumed, without theoretical or experimental validation, that because Crater is a conservative tool – that is, it predicts more damage than will actually occur – the debris would stop at the tile's densified layer, even though their experience did not involve debris strikes as large as STS-107's. Crater-like equations were also used as part of the analysis to assess potential impact damage to the wing leading edge RCC. Again, the tool was used for something other than that for which it was designed; again, it predicted possible penetration; and again, the Debris Assessment Team used engineering arguments and their experience to discount the results. As a result of a transition of responsibility for Crater analysis from the Boeing Huntington Beach facility to the Houston-based Boeing office, the team that conducted the Crater analyses had been formed fairly recently, and therefore could be considered less experienced when compared with the more senior Huntington Beach analysts. In fact, STS-107 was the first mission for which they were solely responsible for providing analysis with the Crater tool. Though post-accident interviews suggested that the training for the Houston Boeing analysts was of high quality and adequate in substance and duration, communications and theoretical understandings of the Crater model among the Houston-based team members had not yet developed to the standard of a more senior team. Due in part to contractual arrangements related to the transition, the Houston-based team did not take full advantage of the Huntington Beach engineers' experience. At the January 24, Mission Management Team meeting at which the "no safety-of-flight" conclusion was presented, there was little engineering discussion about the assumptions made, and how the results would differ if other assumptions were used. Engineering solutions presented to management should have included a quantifiable range of uncertainty and risk analysis. Those types of tools were readily available, routinely used, and would have helped management understand the risk involved in the decision. Management, in turn, should have demanded such information. The very absence of a clear and open discussion of uncertainties and assumptions in the analysis presented should have caused management to probe further. Shuttle Program Management's Low Level of Concern While the debris strike was well outside the activities covered by normal mission flight rules, Mission Management Team members and Shuttle Program managers did not treat the debris strike as an issue that required operational action by Mission Control. Program managers, from Ron Dittemore to individual Mission Management Team members, had, over the course of the Space Shuttle Program, gradually become inured to External Tank foam losses and on a funda168
Report Volume I August 2003
mental level did not believe foam striking the vehicle posed a critical threat to the Orbiter. In particular, Shuttle managers exhibited a belief that RCC panels are impervious to foam impacts. Even after seeing the video of Columbia's debris impact, learning estimates of the size and location of the strike, and noting that a foam strike with sufficient kinetic energy could cause Thermal Protection System damage, management's level of concern did not change. The opinions of Shuttle Program managers and debris and photo analysts on the potential severity of the debris strike diverged early in the mission and continued to diverge as the mission progressed, making it increasingly difficult for the Debris Assessment Team to have their concerns heard by those in a decision-making capacity. In the face of Mission managers' low level of concern and desire to get on with the mission, Debris Assessment Team members had to prove unequivocally that a safety-of-flight issue existed before Shuttle Program management would move to obtain images of the left wing. The engineers found themselves in the unusual position of having to prove that the situation was unsafe – a reversal of the usual requirement to prove that a situation is safe. Other factors contributed to Mission management's ability to resist the Debris Assessment Team's concerns. A tile expert told managers during frequent consultations that strike damage was only a maintenance-level concern and that on-orbit imaging of potential wing damage was not necessary. Mission management welcomed this opinion and sought no others. This constant reinforcement of managers' pre-existing beliefs added another block to the wall between decision makers and concerned engineers. Another factor that enabled Mission management's detachment from the concerns of their own engineers is rooted in the culture of NASA itself. The Board observed an unofficial hierarchy among NASA programs and directorates that hindered the flow of communications. The effects of this unofficial hierarchy are seen in the attitude that members of the Debris Assessment Team held. Part of the reason they chose the institutional route for their imagery request was that without direction from the Mission Evaluation Room and Mission Management Team, they felt more comfortable with their own chain of command, which was outside the Shuttle Program. Further, when asked by investigators why they were not more vocal about their concerns, Debris Assessment Team members opined that by raising contrary points of view about Shuttle mission safety, they would be singled out for possible ridicule by their peers and managers. A Lack of Clear Communication Communication did not flow effectively up to or down from Program managers. As it became clear during the mission that managers were not as concerned as others about the danger of the foam strike, the ability of engineers to challenge those beliefs greatly diminished. Managers' tendency to accept opinions that agree with their own dams the flow of effective communications. After the accident, Program managers stated privately and publicly that if engineers had a safety concern, they were obligated to communicate their concerns to management. Managers did not seem to understand that as leaders they had a corresponding and perhaps greater obligation to create viable routes for the engineering community to express their views and receive information. This barrier to communications not only blocked the flow of information to managers, but it also prevented the downstream flow of information from managers to engineers, leaving Debris Assessment Team members no basis for understanding the reasoning behind Mission Management Team decisions. The January 27 to January 31, phone and e-mail exchanges, primarily between NASA engineers at Langley and Johnson, illustrate another symptom of the "cultural fence" that impairs open communications between mission managers and working engineers. These exchanges and the reaction to them indicated that during the evaluation of a mission contingency, the Mission Management Team failed to disseminate information to all system and technology experts who could be consulted. Issues raised by two Langley and Johnson engineers led to the development of "what-if" landing scenarios of the potential outcome if the main landing gear door sustained damaged. This led to behind-the-scenes networking by these engineers to use NASA facilities to make simulation runs of a compromised landing configuration. These engineers – who understood their systems and related technology – saw the potential for a problem on landing and ran it down in case the unthinkable occurred. But their concerns never reached the managers on the Mission Management Team that had operational control over Columbia.
Report Volume I August 2003
A Lack of Effective Leadership The Shuttle Program, the Mission Management Team, and through it the Mission Evaluation Room, were not actively directing the efforts of the Debris Assessment Team. These management teams were not engaged in scenario selection or discussions of assumptions and did not actively seek status, inputs, or even preliminary results from the individuals charged with analyzing the debris strike. They did not investigate the value of imagery, did not intervene to consult the more experienced Crater analysts at Boeing's Huntington Beach facility, did not probe the assumptions of the Debris Assessment Team's analysis, and did not consider actions to mitigate the effects of the damage on re-entry. Managers' claims that they didn't hear the engineers' concerns were due in part to their not asking or listening. The Failure of Safety's Role As will be discussed in Chapter 7, safety personnel were present but passive and did not serve as a channel for the voicing of concerns or dissenting views. Safety representatives attended meetings of the Debris Assessment Team, Mission Evaluation Room, and Mission Management Team, but were merely party to the analysis process and conclusions instead of an independent source of questions and challenges. Safety contractors in the Mission Evaluation Room were only marginally aware of the debris strike analysis. One contractor did question the Debris Assessment Team safety representative about the analysis and was told that it was adequate. No additional inquiries were made. The highest-ranking safety representative at NASA headquarters deferred to Program managers when asked for an opinion on imaging of Columbia. The safety manager he spoke to also failed to follow up. Summary Management decisions made during Columbia's final flight reflect missed opportunities, blocked or ineffective communications channels, flawed analysis, and ineffective leadership. Perhaps most striking is the fact that management – including Shuttle Program, Mission Management Team, Mission Evaluation Room, and Flight Director and Mission Control – displayed no interest in understanding a problem and its implications. Because managers failed to avail themselves of the wide range of expertise and opinion necessary to achieve the best answer to the debris strike question – "Was this a safety-of-flight concern?" – some Space Shuttle Program managers failed to fulfill the implicit contract to do whatever is possible to ensure the safety of the crew. In fact, their management techniques unknowingly imposed barriers that kept at bay both engineering concerns and dissenting views, and ultimately helped create "blind spots" that prevented them from seeing the danger the foam strike posed. Because this chapter has focused on key personnel who participated in STS-107 bipod foam debris strike decisions, it is tempting to conclude that replacing them will solve all NASA's problems. However, solving NASA's problems is not quite so easily achieved. Peoples' actions are influenced by the organizations in which they work, shaping their choices in directions that even they may not realize. The Board explores the organizational context of decision making more fully in Chapters 7 and 8. Findings Intercenter Photo Working Group F6.3-1 The foam strike was first seen by the Intercenter Photo Working Group on the morning of Flight Day Two during the standard review of launch video and high-speed photography. The strike was larger than any seen in the past, and the group was concerned about possible damage to the Orbiter. No conclusive images of the strike existed. One camera that may have provided an additional view was out of focus because of an improperly maintained lens. The Chair of the Intercenter Photo Working Group asked management to begin the process of getting outside imagery to help in damage assessment. This request, the first of three, began its journey through the management hierarchy on Flight Day Two. The Intercenter Photo Working Group distributed its first report, including a digitized video clip and initial assessment of the strike, on Flight Day Two. This information
Report Volume I August 2003
was widely disseminated to NASA and contractor engineers, Shuttle Program managers, and Mission Operations Directorate personnel. Initial estimates of debris size, speed, and origin were remarkably accurate. Initial information available to managers stated that the debris originated in the left bipod area of the External Tank, was quite large, had a high velocity, and struck the underside of the left wing near its leading edge. The report stated that the debris could have hit the RCC or tile.
The Debris Assessment Team F6.3-5 A Debris Assessment Team began forming on Flight Day two to analyze the impact. Once the debris strike was categorized as "out of family" by United Space Alliance, contractual obligations led to the Team being Co-Chaired by the cognizant contractor sub-system manager and her NASA counterpart. The team was not designated a Tiger Team by the Mission Evaluation Room or Mission Management Team. Though the Team was clearly reporting its plans (and final results) through the Mission Evaluation Room to the Mission Management Team, no Mission manager appeared to "own" the Team's actions. The Mission Management Team, through the Mission Evaluation Room, provided no direction for team activities, and Shuttle managers did not formally consult the Team's leaders about their progress or interim results. During an organizational meeting, the Team discussed the uncertainty of the data and the value of on-orbit imagery to "bound" their analysis. In its first official meeting the next day, the Team gave its NASA Co-Chair the action to request imagery of Columbia on-orbit. The Team routed its request for imagery through Johnson Space Center's Engineering Directorate rather than through the Mission Evaluation Room to the Mission Management Team to the Flight Dynamics Officer, the channel used during a mission. This routing diluted the urgency of their request. Managers viewed it as a noncritical engineering desire rather than a critical operational need. Team members never realized that management's decision against seeking imagery was not intended as a direct or final response to their request. The Team's assessment of possible tile damage was performed using an impact simulation that was well outside Crater's test database. The Boeing analyst was inexperienced in the use of Crater and the interpretation of its results. Engineers with extensive Thermal Protection System expertise at Huntington Beach were not actively involved in determining if the Crater results were properly interpreted. Crater initially predicted tile damage deeper than the actual tile depth, but engineers used their judgment to conclude that damage would not penetrate the densified layer of tile. Similarly, RCC damage conclusions were based primarily on judgment and experience rather than analysis. For a variety of reasons, including management failures, communication breakdowns, inadequate imagery, inappropriate use of assessment tools, and flawed engineering judgments, the damage assessments contained substantial uncertainties. The assumptions (and their uncertainties) used in the analysis were never presented or discussed in full to either the Mission Evaluation Room or the Mission Management Team. While engineers and managers knew the foam could have struck RCC panels; the briefings on the analysis to the Mission Evaluation Room and Mission Management Team did not address RCC damage, and neither Mission Evaluation Room nor Mission Management Team managers asked about it.
Space Shuttle Program Management F6.3-15 F6.3-16 F6.3-17 There were lapses in leadership and communication that made it difficult for engineers to raise concerns or understand decisions. Management failed to actively engage in the analysis of potential damage caused by the foam strike. Mission Management Team meetings occurred infrequently (five times during a 16 day mission), not every day, as specified in Shuttle Program management rules. Shuttle Program Managers entered the mission with the belief, recently reinforced by the STS-113 Flight Readiness Review, that a foam strike is not a safety-of-flight issue.
Report Volume I August 2003
After Program managers learned about the foam strike, their belief that it would not be a problem was confirmed (early, and without analysis) by a trusted expert who was readily accessible and spoke from "experience." No one in management questioned this conclusion. Managers asked "Who's requesting the photos?" instead of assessing the merits of the request. Management seemed more concerned about the staff following proper channels (even while they were themselves taking informal advice) than they were about the analysis. No one in the operational chain of command for STS-107 held a security clearance that would enable them to understand the capabilities and limitations of National imagery resources. Managers associated with STS-107 began investigating the implications of the foam strike on the launch schedule, and took steps to expedite post-flight analysis. Program managers required engineers to prove that the debris strike created a safetyof-flight issue: that is, engineers had to produce evidence that the system was unsafe rather than prove that it was safe. In both the Mission Evaluation Room and Mission Management Team meetings over the Debris Assessment Team's results, the focus was on the bottom line – was there a safety-of-flight issue, or not? There was little discussion of analysis, assumptions, issues, or ramifications.
Communication F6.3-24 F6.3-25 F6.3-26 F6.3-27 Communication did not flow effectively up to or down from Program managers. Three independent requests for imagery were initiated. Much of Program managers' information came through informal channels, which prevented relevant opinion and analysis from reaching decision makers. Program Managers did not actively communicate with the Debris Assessment Team. Partly as a result of this, the Team went through institutional, not mission-related, channels with its request for imagery, and confusion surrounded the origin of imagery requests and their subsequent denial. Communication was stifled by the Shuttle Program attempts to find out who had a "mandatory requirement" for imagery.
Safety Representative's Role F6.3-29 Safety representatives from the appropriate organizations attended meetings of the Debris Assessment Team, Mission Evaluation Room, and Mission Management Team, but were passive, and therefore were not a channel through which to voice concerns or dissenting views.
Recommendation: R6.3-1 Implement an expanded training program in which the Mission Management Team faces potential crew and vehicle safety contingences beyond launch and ascent. These contingences should involve potential loss of Shuttle or crew, contain numerous uncertainties and unknowns, and require the Mission Management Team to assemble and interact with support organizations across NASA/Contractor lines and in various locations. Modify the Memorandum of Agreement with the National Imagery and Mapping Agency (NIMA) to make the imaging of each Shuttle flight while on orbit a standard requirement.
Report Volume I
August 2003
To put the decisions made during the flight of STS-107 into perspective, the Board asked NASA to determine if there were options for the safe return of the STS-107 crew. In this study, NASA was to assume that the extent of damage to the leading edge of the left wing was determined by national imaging assets or by a spacewalk. NASA was then asked to evaluate the possibility of: 1. Rescuing the STS-107 crew by launching Atlantis. Atlantis would be hurried to the pad, launched, rendezvous with Columbia, and take on Columbia's crew for a return. It was assumed that NASA would be willing to expose Atlantis and its crew to the same possibility of External Tank bipod foam loss that damaged Columbia. 2. Repairing damage to Columbia's wing on orbit. In the repair scenario, astronauts would use onboard materials to rig a temporary fix. Some of Columbia's cargo might be jettisoned and a different re-entry profile would be flown to lessen heating on the left wing leading edge. The crew would be prepared to bail out if the wing structure was predicted to fail on landing. In its study of these two options, NASA assumed the following timeline. Following the debris strike discovery on Flight Day Two, Mission Managers requested imagery by Flight Day Three. That imagery was inconclusive, leading to a decision on Flight Day Four to perform a spacewalk on Flight Day Five. That spacewalk revealed potentially catastrophic damage. The crew was directed to begin conserving consumables, such as oxygen and water, and Shuttle managers began around-the-clock processing of Atlantis to prepare it for launch. Shuttle managers pursued both the rescue and the repair options from Flight Day Six to Flight Day 26, and on that day (February 10) decided which one to abandon. The NASA team deemed this timeline realistic for several reasons. First, the team determined that a spacewalk to inspect the left wing could be easily accomplished. The team then assessed how the crew could limit its use of consumables to determine how long Columbia could stay in orbit. The limiting consumable was the lithium hydroxide canisters, which scrub from the cabin atmosphere the carbon dioxide the crew exhales. After consulting with flight surgeons, the team concluded that by modifying crew activity and sleep time carbon dioxide could be kept to acceptable levels until Flight Day 30 (the morning of February 15). All other consumables would last longer. Oxygen, the next most critical, would require the crew to return on Flight Day 31. Repairing Damage On Orbit The repair option (see Figure 6.4-1), while logistically viable using existing materials onboard Columbia, relied on so many uncertainties that NASA rated this option "high risk." To complete a repair, the crew would perform a spacewalk to fill an assumed 6-inch hole in an RCC panel with heavy metal tools, small pieces of titanium, or other metal scavenged from the crew cabin. These heavy metals, which would help protect the wing structure, would be held in place during
Report Volume I
Figure 6.4-1. The speculative repair option would have sent astronauts hanging over the payload bay door to reach the left wing RCC panels using a ladder scavenged from the crew module.
re-entry by a water-filled bag that had turned into ice in the cold of space. The ice and metal would help restore wing leading edge geometry, preventing a turbulent airflow over the wing and therefore keeping heating and burn-through levels low enough for the crew to survive re-entry and bail out before landing. Because the NASA team could not verify that the repairs would survive even a modified re-entry, the rescue option had a considerably higher chance of bringing Columbia's crew back alive. Rescuing the STS-107 Crew with Atlantis Accelerating the processing of Atlantis for early launch and rendezvous with Columbia was by far the most complex task in the rescue scenario. On Columbia's Flight Day Four, Atlantis was in the Orbiter Processing Facility at Kennedy Space Center with its main engines installed and only 41 days from its scheduled March 1 launch. The Solid Rocket Boosters were already mated with the External Tank in the Vehicle Assembly Building. By working three around-theclock shifts seven days a week, Atlantis could be readied for launch, with no necessary testing skipped, by February 10. If launch processing and countdown proceeded smoothly, this would provide a five-day window, from February 10 to February 15, in which Atlantis could rendezvous with Columbia before Columbia's consumables ran out. According to records, the weather on these days allowed a launch. Atlantis would be launched with a crew of four: a commandAugust 2003
Figure 6.4-2. The rescue option had Atlantis (lower vehicle) rendezvousing with Columbia and the STS-107 crew transferring via ropes. Note that the payload bay of Atlantis is empty except for the external airlock/docking adapter.
er, pilot, and two astronauts trained for spacewalks. In January, seven commanders, seven pilots, and nine spacewalktrained astronauts were available. During the rendezvous on Atlantis's first day in orbit, the two Orbiters would maneuver to face each other with their payload bay doors open (see Figure 6.4-2). Suited Columbia crew members would then be transferred to Atlantis via spacewalks. Atlantis would return with four crew members on the flight deck and seven in the mid-deck. Mission Control would then configure Columbia for a de-orbit burn that would ditch the Orbiter in the Pacific Ocean, or would have the Columbia crew take it to a higher orbit for a possible subsequent repair mission if more thorough repairs could be developed. This rescue was considered challenging but feasible. To succeed, it required problem-free processing of Atlantis and a flawless launch countdown. If Program managers had understood the threat that the bipod foam strike posed and were able to unequivocally determine before Flight Day Seven that there was potentially catastrophic damage to the left wing, these repair and rescue plans would most likely have been developed, and a rescue would have been conceivable. For a detailed discussion of the rescue and repair options, see Appendix D.13. Findings: F6.4-1 The repair option, while logistically viable using existing materials onboard Columbia, relied on so many uncertainties that NASA rated this option "high risk." If Program managers were able to unequivocally determine before Flight Day Seven that there
Report Volume I
was potentially catastrophic damage to the left wing, accelerated processing of Atlantis might have provided a window in which Atlantis could rendezvous with Columbia before Columbia's limited consumables ran out. Recommendation: R6.4-1 For missions to the International Space Station, develop a practicable capability to inspect and effect emergency repairs to the widest possible range of damage to the Thermal Protection System, including both tile and Reinforced CarbonCarbon, taking advantage of the additional capabilities available when near to or docked at the International Space Station. For non-Station missions, develop a comprehensive autonomous (independent of Station) inspection and repair capability to cover the widest possible range of damage scenarios. Accomplish an on-orbit Thermal Protection System inspection, using appropriate assets and capabilities, early in all missions. The ultimate objective should be a fully autonomous capability for all missions to address the possibility that an International Space Station mission fails to achieve the correct orbit, fails to dock successfully, or is damaged during or after undocking.
August 2003
The crew cabin access arm in position against Columbia on Launch Complex 39-A.
Report Volume I
August 2003
27 28 29
The citations that contain a reference to "CAIB document" with CAB or CTF followed by seven to eleven digits, such as CAB001-0010, refer to a document in the Columbia Accident Investigation Board database maintained by the Department of Justice and archived at the National Archives.
1 2 3 4 5 6
"STS-45 Space Shuttle Mission Report," NSTS-08275, May 1992, pg. 17. CAIB document CTF003-00030006. "STS-45 Space Shuttle Mission Report," NSTS-08275, May 1992. CAIB document CTF003-00030006. Both STS-56 and STS-58 post mission PRCBs discussed the debris events and IFAs. Closeout rationale was based upon the events being considered "in family" and "within experience base." "Problem Reporting and Corrective Action System Requirements," NSTS08126, Revision H, November 22, 2000, Appendix C, Definitions, Out of Family. CAIB document CTF044-28652894. Post STS-87 PRCBD, S 062127, 18 Dec 1997. M. Elisabeth Paté-Cornell and Paul S. Fischbeck, "Risk Management for the Tiles of the Space Shuttle," pp. 64-86, Interfaces 24, JanuaryFebruary 1994. CAIB document CAB005-0141. Letter to M. Elisabeth Paté-Cornell, Stanford University, from Benjamin Buchbinder, Risk Management Program Manager, NASA, 10 May 1993. CAIB document CAB038-36973698. M. Elisabeth Paté-Cornell, "Follow-up on the Standard 1990 Study of the Risk of Loss of Vehicle and Crew of the NASA Space Shuttle Due to Tile Failure," Report to the Columbia Accident Investigation Board, 18 June 2003. CAIB document CAB006-00970104. M. Litwinsk and G. Wilson, et al., "End-to-End TPS Upgrades Plan for Space Shuttle Orbiter," February 1997; K. Hinkle and G. Wilson, "Advancements in TPS," M&P Engineering, 22 October 1998. Daniel B. Leiser, et al., "Toughened Uni-piece Fibrous Insulation (TUFI)" Patent #5,079,082, 7 January 1992. Karrie Hinkle, "High Density Tile for Enhanced Dimensional Stability," Briefing to Space Shuttle Program, October 19, 1998. CAIB document CAB033-32663280. Daniel B. Leiser, "Present/Future Tile Thermal Protection Systems," A presentation to the CAIB (Group 1), 16 May 2003. John Kowal, "Orbiter Thermal Protection System (TPS) Upgrades." Space Shuttle Upgrades Safety Panel Review, 10 February 2003. "Problem Reporting and Corrective Action System Requirements," NSTS-08126, Revision H, November 22, 2000. CAIB document CTF04428652894. Diane Vaughan, The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA (Chicago: University of Chicago Press, 1996). Richard Feynman, Minority Report on Challenger, The Pleasure of Finding Things Out, (New York: Perseus Publishing, 2002). See Appendix D.17 Tiger Team Checklists. Allen J. Richardson and A. H. McHugh, "Hypervelocity Impact Penetration Equation for Metal By Multiple Regression Analysis," STR153, North American Aviation, Inc., March 1966. Allen J. Richardson and J. C. Chou, "Correlation of TPS Tile Penetration Equation & Impact Test Data," 3 March 1985. "Review of Crater Program for Evaluating Impact Damage to Orbiter TPS Tiles," presented at Boeing-Huntington Beach, 29 Apr 2003. CAIB document CTF070-29492999. J. L. Rand, "Impact Testing of Orbiter HRSI Tiles," Texas Engineering Experiment Station Report (Texas A&M), 1979; Tests conducted by NASA (D. Arabian) ca. 1979. Drew L. Goodlin, "Orbiter Tile Impact Testing, Final Report", SwRI Project # 18-7503-005, March 5, 1999. Allen J. Richardson, "Evaluation of Flight Experience & Test Results for Ice Impaction on Orbiter RCC & ACC Surfaces," Rockwell International, November 26, 1984. Though this entry indicates that NASA contacted USSPACECOM, the correct entity is USSTRATCOM. USSPACECOM ceased to exist in October 2002.
"Space Shuttle Program Description and Requirements Baseline," NSTS07700, Volume X, Book 1. CAIB document CTF028-32643667. "External Tank End Item (CEI) Specification – Part 1," CPT01M09A, contract NAS8 –30300, April 9, 1980, WBS 220.127.116.11 and 18.104.22.168. "STS-1 Orbiter Final Mission Report," JSC-17378, August 1981, p. 85. Discussed in Craig Covault, "Investigators Studying Shuttle Tiles, Aviation Week & Space Technology, May 11, 1981, pg. 40. Report of the Presidential Commission on the Space Shuttle Challenger Accident, Volume V, 1986, pp. 1028-9, hearing section pp. 1845-1849. "Orbiter Vehicle End Item Specification for the Space Shuttle System, Part 1, Performance and Design Requirements," contract NAS9-20000, November 7, 2002. CAIB documents CAB006-06440645 and CAB03320242971. "Problem Reporting and Corrective Action System Requirements," NSTS08126, Revision H, November 22, 2000, Appendix C, Definitions, In Family. CAIB document CTF044-28652894. Ibid. Ibid. The umbilical wells are compartments on the underside of the Orbiter where External Tank liquid oxygen and hydrogen lines connect. After the Orbiters land, the umbilical well camera film is retrieved and developed. NSTS-08126, Paragraph 3.4, Additional Requirements for In-Flight Anomaly (IFA) Reporting. Integrated Hazard Analysis INTG 037, "Degraded Functioning of Orbiter TPS or Damage to the Windows Caused by SRB/ET Ablatives or Debonded ET or SRB TPS." Ibid. Ibid. During the flight of STS-112, the Intercenter Photo Working Group speculated that a second debris strike occurred at 72 seconds, possibly to the right wing. Although post-flight analysis showed that this did not occur, the Board notes that the Intercenter Photo Working Group failed to properly inform the Mission Management Team of this strike, and that the Mission Management Team subsequently failed to aggressively address the event during flight. "Safety and Mission Assurance Report for the STS-113 Mission, PreLaunch Mission Management Team Edition," Enterprise Safety and Mission Assurance Division, November 7, 2002. CAIB Document CTF024-00430061. Orbiter TPS damage numbers come from the Shuttle Flight Data and InFlight Anomaly List (JSC-19413). CAIB Meeting Minutes, presentation and discussion on IFAs for STS-27 and STS-28, March 28, 2003, Houston, Texas. "STS-27R National Space Transportation System Mission Report," NSTS23370, February 1989, p. 2. CAIB Meeting Minutes, presentation and discussion on IFAs for STS-27 and STS-28, March 28, 2003, Houston, Texas. Corrective Action Record, 27RF13, Closeout Report (no date). CAIB document CTF010-20822107. "STS-27R OV-104 Orbiter TPS Damage Review Team Summary Report," Volume I, February 1989, TM-100355, p. 64. CAIB document CAB03502290303. Ibid. "In-Flight Anomaly: STS-35/ET-35," External Tank Flight Readiness Report 3500.2.3/91. CAIB document CAB057-51185119. STS-36 PRCB, IFA Closure Rationale for STS-35. CAIB document CAB02903620433. Identified by MSFC in PRACA database as "not a safety of flight" concern. Briefed at post-STS-42 PRCB and STS-45 Flight Readiness Review. Report Volume I
36 37 30
31 32
8 9 10
11 12
38 39 40
13 14 15
42 43 44
17 18 19 20 21 22
45 46
48 49
23 24 25 26
August 2003
The Accident's Organizational Causes
In the Board's view, NASA's organizational culture and structure had as much to do with this accident as the External Tank foam. Organizational culture refers to the values, norms, beliefs, and practices that govern how an institution functions. At the most basic level, organizational culture defines the assumptions that employees make as they carry out their work. It is a powerful force that can persist through reorganizations and the reassignment of key personnel. Given that today's risks in human space flight are as high and the safety margins as razor thin as they have ever been, there is little room for overconfidence. Yet the attitudes and decision-making of Shuttle Program managers and engineers during the events leading up to this accident were clearly overconfident and often bureaucratic in nature. They deferred to layered and cumbersome regulations rather than the fundamentals of safety. The Shuttle Program's safety culture is straining to hold together the vestiges of a once robust systems safety program. As the Board investigated the Columbia accident, it expected to find a vigorous safety organization, process, and culture at NASA, bearing little resemblance to what the Rogers Commission identified as the ineffective "silent safety" system in which budget cuts resulted in a lack of resources, personnel, independence, and authority. NASA's initial briefings to the Board on its safety programs espoused a risk-averse philosophy that empowered any employee to stop an operation at the mere glimmer of a problem. Unfortunately, NASA's views of its safety culture in those briefings did not reflect reality. Shuttle Program safety personnel failed to adequately assess anomalies and frequently accepted critical risks without qualitative or quantitative support, even when the tools to provide more comprehensive assessments were available. Similarly, the Board expected to find NASA's Safety and Mission Assurance organization deeply engaged at every
August 2003
Many accident investigations make the same mistake in defining causes. They identify the widget that broke or malfunctioned, then locate the person most closely connected with the technical failure: the engineer who miscalculated an analysis, the operator who missed signals or pulled the wrong switches, the supervisor who failed to listen, or the manager who made bad decisions. When causal chains are limited to technical flaws and individual failures, the ensuing responses aimed at preventing a similar event in the future are equally limited: they aim to fix the technical problem and replace or retrain the individual responsible. Such corrections lead to a misguided and potentially disastrous belief that the underlying problem has been solved. The Board did not want to make these errors. A central piece of our expanded cause model involves NASA as an organizational whole.
The organizational causes of this accident are rooted in the Space Shuttle Program's history and culture, including the original compromises that were required to gain approval for the Shuttle Program, subsequent years of resource constraints, fluctuating priorities, schedule pressures, mischaracterizations of the Shuttle as operational rather than developmental, and lack of an agreed national vision. Cultural traits and organizational practices detrimental to safety and reliability were allowed to develop, including: reliance on past success as a substitute for sound engineering practices (such as testing to understand why systems were not performing in accordance with requirements/specifications); organizational barriers which prevented effective communication of critical safety information and stifled professional differences of opinion; lack of integrated management across program elements; and the evolution of an informal chain of command and decision-making processes that operated outside the organization's rules.
Report Volume I
level of Shuttle management: the Flight Readiness Review, the Mission Management Team, the Debris Assessment Team, the Mission Evaluation Room, and so forth. This was not the case. In briefing after briefing, interview after interview, NASA remained in denial: in the agency's eyes, "there were no safety-of-flight issues," and no safety compromises in the long history of debris strikes on the Thermal Protection System. The silence of Program-level safety processes undermined oversight; when they did not speak up, safety personnel could not fulfill their stated mission to provide "checks and balances." A pattern of acceptance prevailed throughout the organization that tolerated foam problems without sufficient engineering justification for doing so. This chapter presents an organizational context for understanding the Columbia accident. Section 7.1 outlines a short history of safety at NASA, beginning in the pre-Apollo era when the agency reputedly had the finest system safetyengineering programs in the world. Section 7.2 discusses organizational theory and its importance to the Board's investigation, and Section 7.3 examines the practices of three organizations that successfully manage high risk. Sections 7.4 and 7.5 look at NASA today and answer the question, "How could NASA have missed the foam signal?" by highlighting the blind spots that rendered the Shuttle Program's risk perspective myopic. The Board's conclusion and recommendations are presented in 7.6. (See Chapter 10 for a discussion of the differences between industrial safety and mission assurance/quality assurance.)
(ASAP). The ASAP was intended to be a senior advisory committee to NASA, reviewing space flight safety studies and operations plans, and evaluating "systems procedures and management policies that contribute to risk." The panel's main priority was human space flight missions.3 Although four of the panel's nine members can be NASA employees, in recent years few have served as members. While the panel's support staff generally consists of fulltime NASA employees, the group technically remains an independent oversight body. Congress simultaneously mandated that NASA create separate safety and reliability offices at the agency's headquarters and at each of its Human Space Flight Centers and Programs. Overall safety oversight became the responsibility of NASA's Chief Engineer. Although these offices were not totally independent – their funding was linked with the very programs they were supposed to oversee – their existence allowed NASA to treat safety as a unique function. Until the Challenger accident in 1986, NASA safety remained linked organizationally and financially to the agency's Human Space Flight Program. Challenger – 1986 In the aftermath of the Challenger accident, the Rogers Commission issued recommendations intended to remedy what it considered to be basic deficiencies in NASA's safety system. These recommendations centered on an underlying theme: the lack of independent safety oversight at NASA. Without independence, the Commission believed, the slate of safety failures that contributed to the Challenger accident – such as the undue influence of schedule pressures and the flawed Flight Readiness process – would not be corrected. "NASA should establish an Office of Safety, Reliability, and Quality Assurance to be headed by an Associate Administrator, reporting directly to the NASA Administrator," concluded the Commission. "It would have direct authority for safety, reliability, and quality assurance throughout the Agency. The office should be assigned the workforce to ensure adequate oversight of its functions and should be independent of other NASA functional and program responsibilities" [emphasis added]. In July 1986, NASA Administrator James Fletcher created a Headquarters Office of Safety, Reliability, and Quality Assurance, which was given responsibility for all agency-wide safety-related policy functions. In the process, the position of Chief Engineer was abolished.4 The new office's Associate Administrator promptly initiated studies on Shuttle in-flight anomalies, overtime levels, the lack of spare parts, and landing and crew safety systems, among other issues.5 Yet NASA's response to the Rogers Commission recommendation did not meet the Commission's intent: the Associate Administrator did not have direct authority, and safety, reliability, and mission assurance activities across the agency remained dependent on other programs and Centers for funding. General Accounting Office Review – 1990 A 1990 review by the U.S. General Accounting Office questioned the effectiveness of NASA's new safety organiAugust 2003
NASA's organizational culture is rooted in history and tradition. From NASA's inception in 1958 to the Challenger accident in 1986, the agency's Safety, Reliability, and Quality Assurance (SRQA) activities, "although distinct disciplines," were "typically treated as one function in the design, development, and operations of NASA's manned space flight programs."1 Contractors and NASA engineers collaborated closely to assure the safety of human space flight. Solid engineering practices emphasized defining goals and relating system performance to them; establishing and using decision criteria; developing alternatives; modeling systems for analysis; and managing operations.2 Although a NASA Office of Reliability and Quality Assurance existed for a short time during the early 1960s, it was funded by the human space flight program. By 1963, the office disappeared from the agency's organization charts. For the next few years, the only type of safety program that existed at NASA was a decentralized "loose federation" of risk assessment oversight run by each program's contractors and the project offices at each of the three Human Space Flight Centers. Fallout from Apollo – 1967 In January 1967, months before the scheduled launch of Apollo 1, three astronauts died when a fire erupted in a ground-test capsule. In response, Congress, seeking to establish an independent safety organization to oversee space flight, created the Aerospace Safety Advisory Panel
Report Volume I
zations in a report titled "Space Program Safety: Funding for NASA's Safety Organizations Should Be Centralized."6 The report concluded "NASA did not have an independent and effective safety organization" [emphasis added]. Although the safety organizational structure may have "appeared adequate," in the late 1980s the space agency had concentrated most of its efforts on creating an independent safety office at NASA Headquarters. In contrast, the safety offices at NASA's field centers "were not entirely independent because they obtained most of their funds from activities whose safety-related performance they were responsible for overseeing." The General Accounting Office worried that "the lack of centralized independent funding may also restrict the flexibility of center safety managers." It also suggested "most NASA safety managers believe that centralized SRM&QA [Safety, Reliability, Maintainability and Quality Assurance] funding would ensure independence." NASA did not institute centralized funding in response to the General Accounting Office report, nor has it since. The problems outlined in 1990 persist to this day. Space Flight Operations Contract – 1996 The Space Flight Operations Contract was intended to streamline and modernize NASA's cumbersome contracting practices, thereby freeing the agency to focus on research and development (see Chapter 5). Yet its implementation complicated issues of safety independence. A single contractor would, in principle, provide "oversight" on production, safety, and mission assurance, as well as cost management, while NASA maintained "insight" into safety and quality assurance through reviews and metrics. Indeed, the reduction to a single primary contract simplified some aspects of the NASA/contractor interface. However, as a result, experienced engineers changed jobs, NASA grew dependent on contractors for technical support, contract monitoring requirements increased, and positions were subsequently staffed by less experienced engineers who were placed in management roles. Collectively, this eroded NASA's in-house engineering and technical capabilities and increased the agency's reliance on the United Space Alliance and its subcontractors to identify, track, and resolve problems. The contract also involved substantial transfers of safety responsibility from the government to the private sector; rollbacks of tens of thousands of Government Mandated Inspection Points; and vast reductions in NASA's in-house safety-related technical expertise (see Chapter 10). In the aggregate, these mid-1990s transformations rendered NASA's already problematic safety system simultaneously weaker and more complex. The effects of transitioning Shuttle operations to the Space Flight Operations Contract were not immediately apparent in the years following implementation. In November 1996, as the contract was being implemented, the Aerospace Safety Advisory Panel published a comprehensive contract review, which concluded that the effort "to streamline the Space Shuttle program has not inadvertently created unacceptable flight or ground risks."7 The Aerospace Safety Advisory Panel's passing grades proved temporary.
Report Volume I
Shuttle Independent Assessment Team – 1999 Just three years later, after a number of close calls, NASA chartered the Shuttle Independent Assessment Team to examine Shuttle sub-systems and maintenance practices (see Chapter 5). The Shuttle Independent Assessment Team Report sounded a stern warning about the quality of NASA's Safety and Mission Assurance efforts and noted that the Space Shuttle Program had undergone a massive change in structure and was transitioning to "a slimmed down, contractor-run operation." The team produced several pointed conclusions: the Shuttle Program was inappropriately using previous success as a justification for accepting increased risk; the Shuttle Program's ability to manage risk was being eroded "by the desire to reduce costs;" the size and complexity of the Shuttle Program and NASA/contractor relationships demanded better communication practices; NASA's safety and mission assurance organization was not sufficiently independent; and "the workforce has received a conflicting message due to the emphasis on achieving cost and staff reductions, and the pressures placed on increasing scheduled flights as a result of the Space Station" [emphasis added].8 The Shuttle Independent Assessment Team found failures of communication to flow up from the "shop floor" and down from supervisors to workers, deficiencies in problem and waiver-tracking systems, potential conflicts of interest between Program and contractor goals, and a general failure to communicate requirements and changes across organizations. In general, the Program's organizational culture was deemed "too insular."9 NASA subsequently formed an Integrated Action Team to develop a plan to address the recommendations from previous Program-specific assessments, including the Shuttle Independent Assessment Team, and to formulate improvements.10 In part this effort was also a response to program missteps in the drive for efficiency seen in the "faster, better, cheaper" NASA of the 1990s. The NASA Integrated Action Team observed: "NASA should continue to remove communication barriers and foster an inclusive environment where open communication is the norm." The intent was to establish an initiative where "the importance of communication and a culture of trust and openness permeate all facets of the organization." The report indicated that "multiple processes to get the messages across the organizational structure" would need to be explored and fostered [emphasis added]. The report recommended that NASA solicit expert advice in identifying and removing barriers, providing tools, training, and education, and facilitating communication processes. The Shuttle Independent Assessment Team and NASA Integrated Action Team findings mirror those presented by the Rogers Commission. The same communication problems persisted in the Space Shuttle Program at the time of the Columbia accident. Space Shuttle Competitive Source Task Force – 2002 In 2002, a 14-member Space Shuttle Competitive Task Force supported by the RAND Corporation examined comAugust 2003
petitive sourcing options for the Shuttle Program. In its final report to NASA, the team highlighted several safety-related concerns, which the Board shares: • Flight and ground hardware and software are obsolete, and safety upgrades and aging infrastructure repairs have been deferred. • Budget constraints have impacted personnel and resources required for maintenance and upgrades. • International Space Station schedules exert significant pressures on the Shuttle Program. • Certain mechanisms may impede worker anonymity in reporting safety concerns. • NASA does not have a truly independent safety function with the authority to halt the progress of a critical mission element. 11 Based on these findings, the task force suggested that an Independent Safety Assurance function should be created that would hold one of "three keys" in the Certification of Flight Readiness process (NASA and the operating contractor would hold the other two), effectively giving this function the ability to stop any launch. Although in the Board's view the "third key" Certification of Flight Readiness process is not a perfect solution, independent safety and verification functions are vital to continued Shuttle operations. This independent function should possess the authority to shut down the flight preparation processes or intervene postlaunch when an anomaly occurs.
Board's deliberation. Fundamental to each theory is the importance of strong organizational culture and commitment to building successful safety strategies. The Board selected certain well-known traits from these models to use as a yardstick to assess the Space Shuttle Program, and found them particularly useful in shaping its views on whether NASA's current organization of its Human Space Flight Program is appropriate for the remaining years of Shuttle operation and beyond. Additionally, organizational theory, which encompasses organizational culture, structure, history, and hierarchy, is used to explain the Columbia accident, and, ultimately, combines with Chapters 5 and 6 to produce an expanded explanation of the accident's causes.16 The Board believes the following considerations are critical to understand what went wrong during STS-107. They will become the central motifs of the Board's analysis later in this chapter. • Commitment to a Safety Culture: NASA's safety culture has become reactive, complacent, and dominated by unjustified optimism. Over time, slowly and unintentionally, independent checks and balances intended to increase safety have been eroded in favor of detailed processes that produce massive amounts of data and unwarranted consensus, but little effective communication. Organizations that successfully deal with high-risk technologies create and sustain a disciplined safety system capable of identifying, analyzing, and controlling hazards throughout a technology's life cycle. • Ability to Operate in Both a Centralized and Decentralized Manner: The ability to operate in a centralized manner when appropriate, and to operate in a decentralized manner when appropriate, is the hallmark of a high-reliability organization. On the operational side, the Space Shuttle Program has a highly centralized structure. Launch commit criteria and flight rules govern every imaginable contingency. The Mission Control Center and the Mission Management Team have very capable decentralized processes to solve problems that are not covered by such rules. The process is so highly regarded that it is considered one of the best problemsolving organizations of its type.17 In these situations, mature processes anchor rules, procedures, and routines to make the Shuttle Program's matrixed workforce seamless, at least on the surface. Nevertheless, it is evident that the position one occupies in this structure makes a difference. When supporting organizations try to "push back" against centralized Program direction – like the Debris Assessment Team did during STS-107 – independent analysis generated by a decentralized decision-making process can be stifled. The Debris Assessment Team, working in an essentially decentralized format, was well-led and had the right expertise to work the problem, but their charter was "fuzzy," and the team had little direct connection to the Mission Management Team. This lack of connection to the Mission Management Team and the Mission Evaluation Room is the single most compelling reason why communications were so poor during the debris
August 2003
To develop a thorough understanding of accident causes and risk, and to better interpret the chain of events that led to the Columbia accident, the Board turned to the contemporary social science literature on accidents and risk and sought insight from experts in High Reliability, Normal Accident, and Organizational Theory.12 Additionally, the Board held a forum, organized by the National Safety Council, to define the essential characteristics of a sound safety program.13 High Reliability Theory argues that organizations operating high-risk technologies, if properly designed and managed, can compensate for inevitable human shortcomings, and therefore avoid mistakes that under other circumstances would lead to catastrophic failures.14 Normal Accident Theory, on the other hand, has a more pessimistic view of the ability of organizations and their members to manage high-risk technology. Normal Accident Theory holds that organizational and technological complexity contributes to failures. Organizations that aspire to failure-free performance are inevitably doomed to fail because of the inherent risks in the technology they operate.15 Normal Accident models also emphasize systems approaches and systems thinking, while the High Reliability model works from the bottom up: if each component is highly reliable, then the system will be highly reliable and safe. Though neither High Reliability Theory nor Normal Accident Theory is entirely appropriate for understanding this accident, insights from each figured prominently in the
Report Volume I
assessment. In this case, the Shuttle Program was unable to simultaneously manage both the centralized and decentralized systems. • Importance of Communication: At every juncture of STS-107, the Shuttle Program's structure and processes, and therefore the managers in charge, resisted new information. Early in the mission, it became clear that the Program was not going to authorize imaging of the Orbiter because, in the Program's opinion, images were not needed. Overwhelming evidence indicates that Program leaders decided the foam strike was merely a maintenance problem long before any analysis had begun. Every manager knew the party line: "we'll wait for the analysis – no safety-of-flight issue expected." Program leaders spent at least as much time making sure hierarchical rules and processes were followed as they did trying to establish why anyone would want a picture of the Orbiter. These attitudes are incompatible with an organization that deals with high-risk technology. • Avoiding Oversimplification: The Columbia accident is an unfortunate illustration of how NASA's strong cultural bias and its optimistic organizational thinking undermined effective decision-making. Over the course of 22 years, foam strikes were normalized to the point where they were simply a "maintenance" issue – a concern that did not threaten a mission's success. This oversimplification of the threat posed by foam debris rendered the issue a low-level concern in the minds of Shuttle managers. Ascent risk, so evident in Challenger, biased leaders to focus on strong signals from the Shuttle System Main Engine and the Solid Rocket Boosters. Foam strikes, by comparison, were a weak and consequently overlooked signal, although they turned out to be no less dangerous. • Conditioned by Success: Even after it was clear from the launch videos that foam had struck the Orbiter in a manner never before seen, Space Shuttle Program managers were not unduly alarmed. They could not imagine why anyone would want a photo of something that could be fixed after landing. More importantly, learned attitudes about foam strikes diminished management's wariness of their danger. The Shuttle Program turned "the experience of failure into the memory of success."18 Managers also failed to develop simple contingency plans for a re-entry emergency. They were convinced, without study, that nothing could be done about such an emergency. The intellectual curiosity and skepticism that a solid safety culture requires was almost entirely absent. Shuttle managers did not embrace safety-conscious attitudes. Instead, their attitudes were shaped and reinforced by an organization that, in this instance, was incapable of stepping back and gauging its biases. Bureaucracy and process trumped thoroughness and reason. • Significance of Redundancy: The Human Space Flight Program has compromised the many redundant processes, checks, and balances that should identify and correct small errors. Redundant systems essential to every
Report Volume I
high-risk enterprise have fallen victim to bureaucratic efficiency. Years of workforce reductions and outsourcing have culled from NASA's workforce the layers of experience and hands-on systems knowledge that once provided a capacity for safety oversight. Safety and Mission Assurance personnel have been eliminated, careers in safety have lost organizational prestige, and the Program now decides on its own how much safety and engineering oversight it needs. Aiming to align its inspection regime with the International Organization for Standardization 9000/9001 protocol, commonly used in industrial environments – environments very different than the Shuttle Program – the Human Space Flight Program shifted from a comprehensive "oversight" inspection process to a more limited "insight" process, cutting mandatory inspection points by more than half and leaving even fewer workers to make "second" or "third" Shuttle systems checks (see Chapter 10). Implications for the Shuttle Program Organization The Board's investigation into the Columbia accident revealed two major causes with which NASA has to contend: one technical, the other organizational. As mentioned earlier, the Board studied the two dominant theories on complex organizations and accidents involving high-risk technologies. These schools of thought were influential in shaping the Board's organizational recommendations, primarily because each takes a different approach to understanding accidents and risk. The Board determined that high-reliability theory is extremely useful in describing the culture that should exist in the human space flight organization. NASA and the Space Shuttle Program must be committed to a strong safety culture, a view that serious accidents can be prevented, a willingness to learn from mistakes, from technology, and from others, and a realistic training program that empowers employees to know when to decentralize or centralize problem-solving. The Shuttle Program cannot afford the mindset that accidents are inevitable because it may lead to unnecessarily accepting known and preventable risks. The Board believes normal accident theory has a key role in human spaceflight as well. Complex organizations need specific mechanisms to maintain their commitment to safety and assist their understanding of how complex interactions can make organizations accident-prone. Organizations cannot put blind faith into redundant warning systems because they inherently create more complexity, and this complexity in turn often produces unintended system interactions that can lead to failure. The Human Space Flight Program must realize that additional protective layers are not always the best choice. The Program must also remain sensitive to the fact that despite its best intentions, managers, engineers, safety professionals, and other employees, can, when confronted with extraordinary demands, act in counterproductive ways. The challenges to failure-free performance highlighted by these two theoretical approaches will always be present in an organization that aims to send humans into space. What
August 2003
can the Program do about these difficulties? The Board considered three alternatives. First, the Board could recommend that NASA follow traditional paths to improving safety by making changes to policy, procedures, and processes. These initiatives could improve organizational culture. The analysis provided by experts and the literature leads the Board to conclude that although reforming management practices has certain merits, it also has critical limitations. Second, the Board could recommend that the Shuttle is simply too risky and should be grounded. As will be discussed in Chapter 9, the Board is committed to continuing human space exploration, and believes the Shuttle Program can and should continue to operate. Finally, the Board could recommend a significant change to the organizational structure that controls the Space Shuttle Program's technology. As will be discussed at length in this chapter's conclusion, the Board believes this option has the best chance to successfully manage the complexities and risks of human space flight.
The Navy SUBSAFE and Naval Reactor programs exercise a high degree of engineering discipline, emphasize total responsibility of individuals and organizations, and provide redundant and rapid means of communicating problems to decision-makers. The Navy's nuclear safety program emerged with its first nuclear-powered warship (USS Nautilus), while non-nuclear SUBSAFE practices evolved from from past flooding mishaps and philosophies first introduced by Naval Reactors. The Navy lost two nuclear-powered submarines in the 1960s – the USS Thresher in 1963 and the Scorpion 1968 – which resulted in a renewed effort to prevent accidents.21 The SUBSAFE program was initiated just two months after the Thresher mishap to identify critical changes to submarine certification requirements. Until a ship was independently recertified, its operating depth and maneuvers were limited. SUBSAFE proved its value as a means of verifying the readiness and safety of submarines, and continues to do so today.22 The Naval Reactor Program is a joint Navy/Department of Energy organization responsible for all aspects of Navy nuclear propulsion, including research, design, construction, testing, training, operation, maintenance, and the disposition of the nuclear propulsion plants onboard many Naval ships and submarines, as well as their radioactive materials. Although the naval fleet is ultimately responsible for dayto-day operations and maintenance, those operations occur within parameters established by an entirely independent division of Naval Reactors. The U.S. nuclear Navy has more than 5,500 reactor years of experience without a reactor accident. Put another way, nuclear-powered warships have steamed a cumulative total of over 127 million miles, which is roughly equivalent to over 265 lunar roundtrips. In contrast, the Space Shuttle Program has spent about three years on-orbit, although its spacecraft have traveled some 420 million miles. Naval Reactor success depends on several key elements: • Concise and timely communication of problems using redundant paths • Insistence on airing minority opinions • Formal written reports based on independent peer-reviewed recommendations from prime contractors • Facing facts objectively and with attention to detail • Ability to manage change and deal with obsolescence of classes of warships over their lifetime These elements can be grouped into several thematic categories: • Communication and Action: Formal and informal practices ensure that relevant personnel at all levels are informed of technical decisions and actions that affect their area of responsibility. Contractor technical recommendations and government actions are documented in peer-reviewed formal written correspondence. Unlike NASA, PowerPoint briefings and papers for technical seminars are not substitutes for completed staff work. In addition, contractors strive to provide recommendations
Many of the principles of solid safety practice identified as crucial by independent reviews of NASA and in accident and risk literature are exhibited by organizations that, like NASA, operate risky technologies with little or no margin for error. While the Board appreciates that organizations dealing with high-risk technology cannot sustain accidentfree performance indefinitely, evidence suggests that there are effective ways to minimize risk and limit the number of accidents. In this section, the Board compares NASA to three specific examples of independent safety programs that have strived for accident-free performance and have, by and large, achieved it: the U.S. Navy Submarine Flooding Prevention and Recovery (SUBSAFE), Naval Nuclear Propulsion (Naval Reactors) programs, and the Aerospace Corporation's Launch Verification Process, which supports U.S. Air Force space launches.19 The safety cultures and organizational structure of all three make them highly adept in dealing with inordinately high risk by designing hardware and management systems that prevent seemingly inconsequential failures from leading to major accidents. Although size, complexity, and missions in these organizations and NASA differ, the following comparisons yield valuable lessons for the space agency to consider when re-designing its organization to increase safety. Navy Submarine and Reactor Safety Programs Human space flight and submarine programs share notable similarities. Spacecraft and submarines both operate in hazardous environments, use complex and dangerous systems, and perform missions of critical national significance. Both NASA and Navy operational experience include failures (for example, USS Thresher, USS Scorpion, Apollo 1 capsule fire, Challenger, and Columbia). Prior to the Columbia mishap, Administrator Sean O'Keefe initiated the NASA/Navy Benchmarking Exchange to compare and contrast the programs, specifically in safety and mission assurance.20
Report Volume I
August 2003
based on a technical need, uninfluenced by headquarters or its representatives. Accordingly, division of responsibilities between the contractor and the Government remain clear, and a system of checks and balances is therefore inherent. • Recurring Training and Learning From Mistakes: The Naval Reactor Program has yet to experience a reactor accident. This success is partially a testament to design, but also due to relentless and innovative training, grounded on lessons learned both inside and outside the program. For example, since 1996, Naval Reactors has educated more than 5,000 Naval Nuclear Propulsion Program personnel on the lessons learned from the Challenger accident.23 Senior NASA managers recently attended the 143rd presentation of the Naval Reactors seminar entitled "The Challenger Accident Re-examined." The Board credits NASA's interest in the Navy nuclear community, and encourages the agency to continue to learn from the mistakes of other organizations as well as from its own. • Encouraging Minority Opinions: The Naval Reactor Program encourages minority opinions and "bad news." Leaders continually emphasize that when no minority opinions are present, the responsibility for a thorough and critical examination falls to management. Alternate perspectives and critical questions are always encouraged. In practice, NASA does not appear to embrace these attitudes. Board interviews revealed that it is difficult for minority and dissenting opinions to percolate up through the agency's hierarchy, despite processes like the anonymous NASA Safety Reporting System that supposedly encourages the airing of opinions. • Retaining Knowledge: Naval Reactors uses many mechanisms to ensure knowledge is retained. The Director serves a minimum eight-year term, and the program documents the history of the rationale for every technical requirement. Key personnel in Headquarters routinely rotate into field positions to remain familiar with every aspect of operations, training, maintenance, development and the workforce. Current and past issues are discussed in open forum with the Director and immediate staff at "all-hands" informational meetings under an in-house professional development program. NASA lacks such a program. • Worst-Case Event Failures: Naval Reactors hazard analyses evaluate potential damage to the reactor plant, potential impact on people, and potential environmental impact. The Board identified NASA's failure to adequately prepare for a range of worst-case scenarios as a weakness in the agency's safety and mission assurance training programs. SUBSAFE The Board observed the following during its study of the Navy's SUBSAFE Program.
• SUBSAFE requirements are clearly documented and achievable, with minimal "tailoring" or granting of waivers. NASA requirements are clearly documented but are also more easily waived. • A separate compliance verification organization independently assesses program management.24 NASA's Flight Preparation Process, which leads to Certification of Flight Readiness, is supposed to be an independent check-and-balance process. However, the Shuttle Program's control of both engineering and safety compromises the independence of the Flight Preparation Process. • The submarine Navy has a strong safety culture that emphasizes understanding and learning from past failures. NASA emphasizes safety as well, but training programs are not robust and methods of learning from past failures are informal. • The Navy implements extensive safety training based on the Thresher and Scorpion accidents. NASA has not focused on any of its past accidents as a means of mentoring new engineers or those destined for management positions. • The SUBSAFE structure is enhanced by the clarity, uniformity, and consistency of submarine safety requirements and responsibilities. Program managers are not permitted to "tailor" requirements without approval from the organization with final authority for technical requirements and the organization that verifies SUBSAFE's compliance with critical design and process requirements.25 • The SUBSAFE Program and implementing organization are relatively immune to budget pressures. NASA's program structure requires the Program Manager position to consider such issues, which forces the manager to juggle cost, schedule, and safety considerations. Independent advice on these issues is therefore inevitably subject to political and administrative pressure. • Compliance with critical SUBSAFE design and process requirements is independently verified by a highly capable centralized organization that also "owns" the processes and monitors the program for compliance. • Quantitative safety assessments in the Navy submarine program are deterministic rather than probabilistic. NASA does not have a quantitative, program-wide risk and safety database to support future design capabilities and assist risk assessment teams. Comparing Navy Programs with NASA Significant differences exist between NASA and Navy submarine programs. • Requirements Ownership (Technical Authority): Both the SUBSAFE and Naval Reactors' organizational
Report Volume I
August 2003
approach separates the technical and funding authority from program management in safety matters. The Board believes this separation of authority of program managers – who, by nature, must be sensitive to costs and schedules – and "owners" of technical requirements and waiver capabilities – who, by nature, are more sensitive to safety and technical rigor – is crucial. In the Naval Reactors Program, safety matters are the responsibility of the technical authority. They are not merely relegated to an independent safety organization with oversight responsibilities. This creates valuable checks and balances for safety matters in the Naval Reactors Program technical "requirements owner" community. • Emphasis on Lessons Learned: Both Naval Reactors and the SUBSAFE have "institutionalized" their "lessons learned" approaches to ensure that knowledge gained from both good and bad experience is maintained in corporate memory. This has been accomplished by designating a central technical authority responsible for establishing and maintaining functional technical requirements as well as providing an organizational and institutional focus for capturing, documenting, and using operational lessons to improve future designs. NASA has an impressive history of scientific discovery, but can learn much from the application of lessons learned, especially those that relate to future vehicle design and training for contingencies. NASA has a broad Lessons Learned Information System that is strictly voluntary for program/project managers and management teams. Ideally, the Lessons Learned Information System should support overall program management and engineering functions and provide a historical experience base to aid conceptual developments and preliminary design. The Aerospace Corporation The Aerospace Corporation, created in 1960, operates as a Federally Funded Research and Development Center that supports the government in science and technology that is critical to national security. It is the equivalent of a $500 million enterprise that supports U.S. Air Force planning, development, and acquisition of space launch systems. The Aerospace Corporation employs approximately 3,200 people including 2,200 technical staff (29 percent Doctors of Philosophy, 41 percent Masters of Science) who conduct advanced planning, system design and integration, verify readiness, and provide technical oversight of contractors.26 The Aerospace Corporation's independent launch verification process offers another relevant benchmark for NASA's safety and mission assurance program. Several aspects of the Aerospace Corporation launch verification process and independent mission assurance structure could be tailored to the Shuttle Program. Aerospace's primary product is a formal verification letter to the Air Force Systems Program Office stating a vehicle has been independently verified as ready for launch. The verification includes an independent General Systems Engineering and Integration review of launch preparations by
Report Volume I
Aerospace staff, a review of launch system design and payload integration, and a review of the adequacy of flight and ground hardware, software, and interfaces. This "conceptto-orbit" process begins in the design requirements phase, continues through the formal verification to countdown and launch, and concludes with a post-flight evaluation of events with findings for subsequent missions. Aerospace Corporation personnel cover the depth and breadth of space disciplines, and the organization has its own integrated engineering analysis, laboratory, and test matrix capability. This enables the Aerospace Corporation to rapidly transfer lessons learned and respond to program anomalies. Most importantly, Aerospace is uniquely independent and is not subject to any schedule or cost pressures. The Aerospace Corporation and the Air Force have found the independent launch verification process extremely valuable. Aerospace Corporation involvement in Air Force launch verification has significantly reduced engineering errors, resulting in a 2.9 percent "probability-of-failure" rate for expendable launch vehicles, compared to 14.6 percent in the commercial sector.27 Conclusion The practices noted here suggest that responsibility and authority for decisions involving technical requirements and safety should rest with an independent technical authority. Organizations that successfully operate high-risk technologies have a major characteristic in common: they place a premium on safety and reliability by structuring their programs so that technical and safety engineering organizations own the process of determining, maintaining, and waiving technical requirements with a voice that is equal to yet independent of Program Managers, who are governed by cost, schedule and mission-accomplishment goals. The Naval Reactors Program, SUBSAFE program, and the Aerospace Corporation are examples of organizations that have invested in redundant technical authorities and processes to become highly reliable.
Perhaps the most perplexing question the Board faced during its seven-month investigation into the Columbia accident was "How could NASA have missed the signals the foam was sending?" Answering this question was a challenge. The investigation revealed that in most cases, the Human Space Flight Program is extremely aggressive in reducing threats to safety. But we also know – in hindsight – that detection of the dangers posed by foam was impeded by "blind spots" in NASA's safety culture. From the beginning, the Board witnessed a consistent lack of concern about the debris strike on Columbia. NASA managers told the Board "there was no safety-of-flight issue" and "we couldn't have done anything about it anyway." The investigation uncovered a troubling pattern in which Shuttle Program management made erroneous assumptions about the robustness of a system based on prior success rather than on dependable engineering data and rigorous testing.
August 2003
The Shuttle Program's complex structure erected barriers to effective communication and its safety culture no longer asks enough hard questions about risk. (Safety culture refers to an organization's characteristics and attitudes – promoted by its leaders and internalized by its members – that serve to make safety the top priority.) In this context, the Board believes the mistakes that were made on STS-107 are not isolated failures, but are indicative of systemic flaws that existed prior to the accident. Had the Shuttle Program observed the principles discussed in the previous two sections, the threat that foam posed to the Orbiter, particularly after the STS-112 and STS-107 foam strikes, might have been more fully appreciated by Shuttle Program management. In this section, the Board examines the NASA's safety policy, structure, and process, communication barriers, the risk assessment systems that govern decision-making and risk management, and the Shuttle Program's penchant for substituting analysis for testing. NASA's Safety: Policy, Structure, and Process Safety Policy NASA's current philosophy for safety and mission assurance calls for centralized policy and oversight at Head-
quarters and decentralized execution of safety programs at the enterprise, program, and project levels. Headquarters dictates what must be done, not how it should be done. The operational premise that logically follows is that safety is the responsibility of program and project managers. Managers are subsequently given flexibility to organize safety efforts as they see fit, while NASA Headquarters is charged with maintaining oversight through independent surveillance and assessment.28 NASA policy dictates that safety programs should be placed high enough in the organization, and be vested with enough authority and seniority, to "maintain independence." Signals of potential danger, anomalies, and critical information should, in principle, surface in the hazard identification process and be tracked with risk assessments supported by engineering analyses. In reality, such a process demands a more independent status than NASA has ever been willing to give its safety organizations, despite the recommendations of numerous outside experts over nearly two decades, including the Rogers Commission (1986), General Accounting Office (1990), and the Shuttle Independent Assessment Team (2000). Safety Organization Structure Center safety organizations that support the Shuttle Program are tailored to the missions they perform. Johnson and
Issue: Same Individual, 4 roles that cross Center, Program and Headquarters responsibilies Result: Failure of checks and balances
NASA Administrator
Code M Office of Space Flight AA
(Safety Advisor)
Code Q Safety and Mission Assurance AA
Code Q MMT Letter
Deputy AA ISS/SSP
JSC Center Director
Space Shuttle SR & QA Manager
Verbal Input
JSC SR & QA Director
ISS Program Manager
Space Shuttle Program Manager
JSC Organization Managers
SR & QA Director
Space Shuttle Division Chief
Independent Assessment Office
Shuttle Element Managers Endorse
Space Shuttle S & MA Manager
Space Shuttle Organization Managers
Funding via Integrated Task Agreements
United Space Alliance Vice President SQ & MA
Figure 7.4-1. Independent safety checks and balance failure.
Report Volume I August 2003
Marshall Safety and Mission Assurance organizations are organized similarly. In contrast, Kennedy has decentralized its Safety and Mission Assurance components and assigned them to the Shuttle Processing Directorate. This management change renders Kennedy's Safety and Mission Assurance structure even more dependent on the Shuttle Program, which reduces effective oversight. At Johnson, safety programs are centralized under a Director who oversees five divisions and an Independent Assessment Office. Each division has clearly-defined roles and responsibilities, with the exception of the Space Shuttle Division Chief, whose job description does not reflect the full scope of authority and responsibility ostensibly vested in the position. Yet the Space Shuttle Division Chief is empowered to represent the Center, the Shuttle Program, and NASA Headquarters Safety and Mission Assurance at critical junctures in the safety process. The position therefore represents a critical node in NASA's Safety and Mission Assurance architecture that seems to the Board to be plagued by conflict of interest. It is a single point of failure without any checks or balances. Johnson also has a Shuttle Program Safety and Mission Assurance Manager who oversees United Space Alliance's safety organization. The Shuttle Program further receives program safety support from the Center's Safety, Reliability, and Quality Assurance Space Shuttle Division. Johnson's Space Shuttle Division Chief has the additional role of Shuttle Program Safety, Reliability, and Quality Assurance Manager (see Figure 7.4-1). Over the years, this dual designation has resulted in a general acceptance of the fact that the Johnson Space Shuttle Division Chief performs duties on both the Center's and Program's behalf. The detached nature of the support provided by the Space Shuttle Division Chief, and the wide band of the position's responsibilities throughout multiple layers of NASA's hierarchy, confuses lines of authority, responsibility, and accountability in a manner that almost defies explanation. A March 2001 NASA Office of Inspector General Audit Report on Space Shuttle Program Management Safety Observations made the same point: The job descriptions and responsibilities of the Space Shuttle Program Manager and Chief, Johnson Safety Office Space Shuttle Division, are nearly identical with each official reporting to a different manager. This overlap in responsibilities conflicts with the SFOC [Space Flight Operations Contract] and NSTS 07700, which requires the Chief, Johnson Safety Office Space Shuttle Division, to provide matrixed personnel support to the Space Shuttle Program Safety Manager in fulfilling requirements applicable to the safety, reliability, and quality assurance aspects of the Space Shuttle Program. The fact that Headquarters, Center, and Program functions are rolled-up into one position is an example of how a carefully designed oversight process has been circumvented and made susceptible to conflicts of interest. This organizational construct is unnecessarily bureaucratic and defeats NASA's stated objective of providing an independent safety func186
Report Volume I
tion. A similar argument can be made about the placement of quality assurance in the Shuttle Processing Divisions at Kennedy, which increases the risk that quality assurance personnel will become too "familiar" with programs they are charged to oversee, which hinders oversight and judgment. The Board believes that although the Space Shuttle Program has effective safety practices at the "shop floor" level, its operational and systems safety program is flawed by its dependence on the Shuttle Program. Hindered by a cumbersome organizational structure, chronic understaffing, and poor management principles, the safety apparatus is not currently capable of fulfilling its mission. An independent safety structure would provide the Shuttle Program a more effective operational safety process. Crucial components of this structure include a comprehensive integration of safety across all the Shuttle programs and elements, and a more independent system of checks and balances. Safety Process In response to the Rogers Commission Report, NASA established what is now known as the Office of Safety and Mission Assurance at Headquarters to independently monitor safety and ensure communication and accountability agency-wide. The Office of Safety and Mission Assurance monitors unusual events like "out of family" anomalies and establishes agency-wide Safety and Mission Assurance policy. (An out-of-family event is an operation or performance outside the expected performance range for a given parameter or which has not previously been experienced.) The Office of Safety and Mission Assurance also screens the Shuttle Program's Flight Readiness Process and signs the Certificate of Flight Readiness. The Shuttle Program Manager, in turn, is responsible for overall Shuttle safety and is supported by a one-person safety staff. The Shuttle Program has been permitted to organize its safety program as it sees fit, which has resulted in a lack of standardized structure throughout NASA's various Centers, enterprises, programs, and projects. The level of funding a program is granted impacts how much safety the Program can "buy" from a Center's safety organization. In turn, Safety and Mission Assurance organizations struggle to anticipate program requirements and guarantee adequate support for the many programs for which they are responsible. It is the Board's view, shared by previous assessments, that the current safety system structure leaves the Office of Safety and Mission Assurance ill-equipped to hold a strong and central role in integrating safety functions. NASA Headquarters has not effectively integrated safety efforts across its culturally and technically distinct Centers. In addition, the practice of "buying" safety services establishes a relationship in which programs sustain the very livelihoods of the safety experts hired to oversee them. These idiosyncrasies of structure and funding preclude the safety organization from effectively providing independent safety analysis. The commit-to-flight review process, as described in Chapters 2 and 6, consists of program reviews and readiness polls that are structured to allow NASA's senior leaders to assess
August 2003
mission readiness. In like fashion, safety organizations affiliated with various projects, programs, and Centers at NASA, conduct a Pre-launch Assessment Review of safety preparations and mission concerns. The Shuttle Program does not officially sanction the Pre-launch Assessment Review, which updates the Associate Administrator for Safety and Mission Assurance on safety concerns during the Flight Readiness Review/Certification of Flight Readiness process. The Johnson Space Shuttle Safety, Reliability, and Quality Assurance Division Chief orchestrates this review on behalf of Headquarters. Note that this division chief also advises the Shuttle Program Manager of Safety. Because it lacks independent analytical rigor, the Pre-launch Assessment Review is only marginally effective. In this arrangement, the Johnson Shuttle Safety, Reliability, and Quality Assurance Division Chief is expected to render an independent assessment of his own activities. Therefore, the Board is concerned that the Pre-Launch Assessment Review is not an effective check and balance in the Flight Readiness Review. Given that the entire Safety and Mission Assurance organization depends on the Shuttle Program for resources and simultaneously lacks the independent ability to conduct detailed analyses, cost and schedule pressures can easily and unintentionally influence safety deliberations. Structure and process places Shuttle safety programs in the unenviable position of having to choose between rubber-stamping engineering analyses, technical efforts, and Shuttle program decisions, or trying to carry the day during a committee meeting in which the other side almost always has more information and analytic capability. NASA Barriers to Communication: Integration, Information Systems, and Databases By their very nature, high-risk technologies are exceptionally difficult to manage. Complex and intricate, they consist of numerous interrelated parts. Standing alone, components may function adequately, and failure modes may be anticipated. Yet when components are integrated into a total system and work in concert, unanticipated interactions can occur that can lead to catastrophic outcomes.29 The risks inherent in these technical systems are heightened when they are produced and operated by complex organizations that can also break down in unanticipated ways. The Shuttle Program is such an organization. All of these factors make effective communication – between individuals and between programs – absolutely critical. However, the structure and complexity of the Shuttle Program hinders communication. The Shuttle Program consists of government and contract personnel who cover an array of scientific and technical disciplines and are affiliated with various dispersed space, research, and test centers. NASA derives its organizational complexity from its origins as much as its widely varied missions. NASA Centers naturally evolved with different points of focus, a "divergence" that the Rogers Commission found evident in the propensity of Marshall personnel to resolve problems without including program managers outside their Center – especially managers at Johnson, to whom they officially reported (see Chapter 5).
Report Volume I
Despite periodic attempts to emphasize safety, NASA's frequent reorganizations in the drive to become more efficient reduced the budget for safety, sending employees conflicting messages and creating conditions more conducive to the development of a conventional bureaucracy than to the maintenance of a safety-conscious research-and-development organization. Over time, a pattern of ineffective communication has resulted, leaving risks improperly defined, problems unreported, and concerns unexpressed.30 The question is, why? The transition to the Space Flight Operations Contract – and the effects it initiated – provides part of the answer. In the Space Flight Operations Contract, NASA encountered a completely new set of structural constraints that hindered effective communication. New organizational and contractual requirements demanded an even more complex system of shared management reviews, reporting relationships, safety oversight and insight, and program information development, dissemination, and tracking. The Shuttle Independent Assessment Team's report documented these changes, noting that "the size and complexity of the Shuttle system and of the NASA/contractor relationships place extreme importance on understanding, communication, and information handling."31 Among other findings, the Shuttle Independent Assessment Team observed that: • The current Shuttle program culture is too insular • There is a potential for conflicts between contractual and programmatic goals • There are deficiencies in problem and waiver-tracking systems • The exchange of communication across the Shuttle program hierarchy is structurally limited, both upward and downward.32 The Board believes that deficiencies in communication, including those spelled out by the Shuttle Independent Assessment Team, were a foundation for the Columbia accident. These deficiencies are byproducts of a cumbersome, bureaucratic, and highly complex Shuttle Program structure and the absence of authority in two key program areas that are responsible for integrating information across all programs and elements in the Shuttle program. Integration Structures NASA did not adequately prepare for the consequences of adding organizational structure and process complexity in the transition to the Space Flight Operations Contract. The agency's lack of a centralized clearinghouse for integration and safety further hindered safe operations. In the Board's opinion, the Shuttle Integration and Shuttle Safety, Reliability, and Quality Assurance Offices do not fully integrate information on behalf of the Shuttle Program. This is due, in part, to an irregular division of responsibilities between the Integration Office and the Orbiter Vehicle Engineering Office and the absence of a truly independent safety organization. Within the Shuttle Program, the Orbiter Office handles many key integration tasks, even though the Integration Office apAugust 2003
pears to be the more logical office to conduct them; the Orbiter Office does not actively participate in the Integration Control Board; and Orbiter Office managers are actually ranked above their Integration Office counterparts. These uncoordinated roles result in conflicting and erroneous information, and support the perception that the Orbiter Office is isolated from the Integration Office and has its own priorities. The Shuttle Program's structure and process for Safety and Mission Assurance activities further confuse authority and responsibility by giving the Program's Safety and Mission Assurance Manager technical oversight of the safety aspects of the Space Flight Operations Contract, while simultaneously making the Johnson Space Shuttle Division Chief responsible for advising the Program on safety performance. As a result, no one office or person in Program management is responsible for developing an integrated risk assessment above the sub-system level that would provide a comprehensive picture of total program risks. The net effect is that many Shuttle Program safety, quality, and mission assurance roles are never clearly defined. Safety Information Systems Numerous reviews and independent assessments have noted that NASA's safety system does not effectively manage risk. In particular, these reviews have observed that the processes in which NASA tracks and attempts to mitigate the risks posed by components on its Critical Items List is flawed. The Post Challenger Evaluation of Space Shuttle Risk Assessment and Management Report (1988) concluded that: The committee views NASA critical items list (CIL) waiver decision-making process as being subjective, with little in the way of formal and consistent criteria for approval or rejection of waivers. Waiver decisions appear to be driven almost exclusively by the design based Failure Mode Effects Analysis (FMEA)/CIL retention rationale, rather than being based on an integrated assessment of all inputs to risk management. The retention rationales appear biased toward proving that the design is "safe," sometimes ignoring significant evidence to the contrary. The report continues, "… the Committee has not found an independent, detailed analysis or assessment of the CIL retention rationale which considers all inputs to the risk assessment process."33 Ten years later, the Shuttle Independent Assessment Team reported "Risk Management process erosion created by the desire to reduce costs …" 34 The Shuttle Independent Assessment Team argued strongly that NASA Safety and Mission Assurance should be restored to its previous role of an independent oversight body, and Safety and Mission Assurance not be simply a "safety auditor." The Board found similar problems with integrated hazard analyses of debris strikes on the Orbiter. In addition, the information systems supporting the Shuttle – intended to be tools for decision-making – are extremely cumbersome and difficult to use at any level.
Report Volume I
The following addresses the hazard tracking tools and major databases in the Shuttle Program that promote risk management. • Hazard Analysis: A fundamental element of system safety is managing and controlling hazards. NASA's only guidance on hazard analysis is outlined in the Methodology for Conduct of Space Shuttle Program Hazard Analysis, which merely lists tools available.35 Therefore, it is not surprising that hazard analysis processes are applied inconsistently across systems, subsystems, assemblies, and components. United Space Alliance, which is responsible for both Orbiter integration and Shuttle Safety Reliability and Quality Assurance, delegates hazard analysis to Boeing. However, as of 2001, the Shuttle Program no longer requires Boeing to conduct integrated hazard analyses. Instead, Boeing now performs hazard analysis only at the sub-system level. In other words, Boeing analyzes hazards to components and elements, but is not required to consider the Shuttle as a whole. Since the current Failure Mode Effects Analysis/Critical Item List process is designed for bottom-up analysis at the component level, it cannot effectively support the kind of "top-down" hazard analysis that is needed to inform managers on risk trends and identify potentially harmful interactions between systems. The Critical Item List (CIL) tracks 5,396 individual Shuttle hazards, of which 4,222 are termed "Critical-
NASA presented a Space Shuttle Safety Upgrade Initiative to Congress as part of its Fiscal Year 2001 budget in March 2000. This initiative sought to create a "Pro-active upgrade program to keep Shuttle flying safely and efficiently to 2012 and beyond to meet agency commitments and goals for human access to space." The planned Shuttle safety upgrades included: Electric Auxiliary Power Unit, Improved Main Landing Gear Tire, Orbiter Cockpit/Avionics Upgrades, Space Shuttle Main Engine Advanced Health Management System, Block III Space Shuttle Main Engine, Solid Rocket Booster Thrust Vector Control/Auxiliary Power Unit Upgrades Plan, Redesigned Solid Rocket Motor – Propellant Grain Geometry Modification, and External Tank Upgrades – Friction Stir Weld. The plan called for the upgrades to be completed by 2008. However, as discussed in Chapter 5, every proposed safety upgrade – with a few exceptions – was either not approved or was deferred. The irony of the Space Shuttle Safety Upgrade Program was that the strategy placed emphasis on keeping the "Shuttle flying safely and efficiently to 2012 and beyond," yet the Space Flight Leadership Council accepted the upgrades only as long as they were financially feasible. Funding a safety upgrade in order to fly safely, and then canceling it for budgetary reasons, makes the concept of mission safety rather hollow.
August 2003
ity 1/1R." Of those, 3,233 have waivers. CRIT 1/1R component failures are defined as those that will result in loss of the Orbiter and crew. Waivers are granted whenever a Critical Item List component cannot be redesigned or replaced. More than 36 percent of these waivers have not been reviewed in 10 years, a sign that NASA is not aggressively monitoring changes in system risk. It is worth noting that the Shuttle's Thermal Protection System is on the Critical Item List, and an existing hazard analysis and hazard report deals with debris strikes. As discussed in Chapter 6, Hazard Report #37 is ineffectual as a decision aid, yet the Shuttle Program never challenged its validity at the pivotal STS-113 Flight Readiness Review. Although the Shuttle Program has undoubtedly learned a great deal about the technological limitations inherent in Shuttle operations, it is equally clear that risk – as represented by the number of critical items list and waivers – has grown substantially without a vigorous effort to assess and reduce technical problems that increase risk. An information system bulging with over 5,000 critical items and 3,200 waivers is exceedingly difficult to manage. • Hazard Reports: Hazard reports, written either by the Space Shuttle Program or a contractor, document conditions that threaten the safe operation of the Shuttle. Managers use these reports to evaluate risk and justify flight.36 During mission preparations, contractors and Centers review all baseline hazard reports to ensure they are current and technically correct. Board investigators found that a large number of hazard reports contained subjective and qualitative judgments, such as "believed" and "based on experience from previous flights this hazard is an ʻAccepted Risk.'" A critical ingredient of a healthy safety program is the rigorous implementation of technical standards. These standards must include more than hazard analysis or low-level technical activities. Standards must integrate project engineering and management activities. Finally, a mechanism for feedback on the effectiveness of system safety engineering and management needs to be built into procedures to learn if safety engineering and management methods are weakening over time. Dysfunctional Databases In its investigation, the Board found that the information systems that support the Shuttle program are extremely cumbersome and difficult to use in decision-making at any level. For obvious reasons, these shortcomings imperil the Shuttle Program's ability to disseminate and share critical information among its many layers. This section explores the report databases that are crucial to effective risk management. • Problem Reporting and Corrective Action: The Problem Reporting and Corrective Action database
Report Volume I
records any non-conformances (instances in which a requirement is not met). Formerly, different Centers and contractors used the Problem Reporting and Corrective Action database differently, which prevented comparisons across the database. NASA recently initiated an effort to integrate these databases to permit anyone in the agency to access information from different Centers. This system, Web Program Compliance Assurance and Status System (WEBPCASS), is supposed to provide easier access to consolidated information and facilitates higher-level searches. However, NASA safety managers have complained that the system is too time-consuming and cumbersome. Only employees trained on the database seem capable of using WEBPCASS effectively. One particularly frustrating aspect of which the Board is acutely aware is the database's waiver section. It is a critical information source, but only the most expert users can employ it effectively. The database is also incomplete. For instance, in the case of foam strikes on the Thermal Protection System, only strikes that were declared "In-Fight Anomalies" are added to the Problem Reporting and Corrective Action database, which masks the full extent of the foam debris trends. • Lessons Learned Information System: The Lessons Learned Information System database is a much simpler system to use, and it can assist with hazard identification and risk assessment. However, personnel familiar with the Lessons Learned Information System indicate that design engineers and mission assurance personnel use it only on an ad hoc basis, thereby limiting its utility. The Board is not the first to note such deficiencies. Numerous reports, including most recently a General Accounting Office 2001 report, highlighted fundamental weaknesses in the collection and sharing of lessons learned by program and project managers.37 Conclusions Throughout the course of this investigation, the Board found that the Shuttle Program's complexity demands highly effective communication. Yet integrated hazard reports and risk analyses are rarely communicated effectively, nor are the many databases used by Shuttle Program engineers and managers capable of translating operational experiences into effective risk management practices. Although the Space Shuttle system has conducted a relatively small number of missions, there is more than enough data to generate performance trends. As it is currently structured, the Shuttle Program does not use data-driven safety methodologies to their fullest advantage.
In this section, the Board examines how and why an array of processes, groups, and individuals in the Shuttle Program failed to appreciate the severity and implications of the foam strike on STS-107. The Board believes that the Shuttle Program should have been able to detect the foam trend and
August 2003
more fully appreciate the danger it represented. Recall that "safety culture" refers to the collection of characteristics and attitudes in an organization – promoted by its leaders and internalized by its members – that makes safety an overriding priority. In the following analysis, the Board outlines shortcomings in the Space Shuttle Program, Debris Assessment Team, and Mission Management Team that resulted from a flawed safety culture. Shuttle Program Shortcomings The flight readiness process, which involves every organization affiliated with a Shuttle mission, missed the danger signals in the history of foam loss. Generally, the higher information is transmitted in a hierarchy, the more it gets "rolled-up," abbreviated, and simplified. Sometimes information gets lost altogether, as weak signals drop from memos, problem identification systems, and formal presentations. The same conclusions, repeated over time, can result in problems eventually being deemed non-problems. An extraordinary example of this phenomenon is how Shuttle Program managers assumed the foam strike on STS-112 was not a warning sign (see Chapter 6). During the STS-113 Flight Readiness Review, the bipod foam strike to STS-112 was rationalized by simply restating earlier assessments of foam loss. The question of why bipod foam would detach and strike a Solid Rocket Booster spawned no further analysis or heightened curiosity; nor did anyone challenge the weakness of External Tank Project Manager's argument that backed launching the next mission. After STS-113's successful flight, once again the STS-112 foam event was not discussed at the STS-107 Flight Readiness Review. The failure to mention an outstanding technical anomaly, even if not technically a violation of NASA's own procedures, desensitized the Shuttle Program to the dangers of foam striking the Thermal Protection System, and demonstrated just how easily the flight preparation process can be compromised. In short, the dangers of bipod foam got "rolled-up," which resulted in a missed opportunity to make Shuttle managers aware that the Shuttle required, and did not yet have a fix for the problem. Once the Columbia foam strike was discovered, the Mission Management Team Chairperson asked for the rationale the STS-113 Flight Readiness Review used to launch in spite of the STS-112 foam strike. In her e-mail, she admitted that the analysis used to continue flying was, in a word, "lousy" (Chapter 6). This admission – that the rationale to fly was rubber-stamped – is, to say the least, unsettling. The Flight Readiness process is supposed to be shielded from outside influence, and is viewed as both rigorous and systematic. Yet the Shuttle Program is inevitably influenced by external factors, including, in the case of the STS-107, schedule demands. Collectively, such factors shape how the Program establishes mission schedules and sets budget priorities, which affects safety oversight, workforce levels, facility maintenance, and contractor workloads. Ultimately, external expectations and pressures impact even data collection, trend analysis, information development, and the re190
Report Volume I
porting and disposition of anomalies. These realities contradict NASA's optimistic belief that pre-flight reviews provide true safeguards against unacceptable hazards. The schedule pressure to launch International Space Station Node 2 is a powerful example of this point (Section 6.2). The premium placed on maintaining an operational schedule, combined with ever-decreasing resources, gradually led Shuttle managers and engineers to miss signals of potential danger. Foam strikes on the Orbiter's Thermal Protection System, no matter what the size of the debris, were "normalized" and accepted as not being a "safety-of-flight risk." Clearly, the risk of Thermal Protection damage due to such a strike needed to be better understood in quantifiable terms. External Tank foam loss should have been eliminated or mitigated with redundant layers of protection. If there was in fact a strong safety culture at NASA, safety experts would have had the authority to test the actual resilience of the leading edge Reinforced Carbon-Carbon panels, as the Board has done. Debris Assessment Team Shortcomings Chapter Six details the Debris Assessment Team's efforts to obtain additional imagery of Columbia. When managers in the Shuttle Program denied the team's request for imagery, the Debris Assessment Team was put in the untenable position of having to prove that a safety-of-flight issue existed without the very images that would permit such a determination. This is precisely the opposite of how an effective safety culture would act. Organizations that deal with high-risk operations must always have a healthy fear of failure – operations must be proved safe, rather than the other way around. NASA inverted this burden of proof. Another crucial failure involves the Boeing engineers who conducted the Crater analysis. The Debris Assessment Team relied on the inputs of these engineers along with many others to assess the potential damage caused by the foam strike. Prior to STS-107, Crater analysis was the responsibility of a team at Boeing's Huntington Beach facility in California, but this responsibility had recently been transferred to Boeing's Houston office. In October 2002, the Shuttle Program completed a risk assessment that predicted the move of Boeing functions from Huntington Beach to Houston would increase risk to Shuttle missions through the end of 2003, because of the small number of experienced engineers who were willing to relocate. To mitigate this risk, NASA and United Space Alliance developed a transition plan to run through January 2003. The Board has discovered that the implementation of the transition plan was incomplete and that training of replacement personnel was not uniform. STS-107 was the first mission during which Johnson-based Boeing engineers conducted analysis without guidance and oversight from engineers at Huntington Beach. Even though STS-107's debris strike was 400 times larger than the objects Crater is designed to model, neither Johnson engineers nor Program managers appealed for assistance from the more experienced Huntington Beach engineers,
August 2003
The Debris Assessment Team presented its analysis in a formal briefing to the Mission Evaluation Room that relied on PowerPoint slides from Boeing. When engineering analyses and risk assessments are condensed to fit on a standard form or overhead slide, information is inevitably lost. In the process, the priority assigned to information can be easily misrepresented by its placement on a chart and the language that is used. Dr. Edward Tufte of Yale University, an expert in information presentation who also researched communications failures in the Challenger accident, studied how the slides used by the Debris Assessment Team in their briefing to the Mission Evaluation Room misrepresented key information.38 The slide created six levels of hierarchy, signified by the title and the symbols to the left of each line. These levels prioritized information that was already contained in 11 simple sentences. Tufte also notes that the title is confusing. "Review of Test Data Indicates Conservatism" refers not to the predicted tile damage, but to the choice of test models used to predict the damage. Only at the bottom of the slide do engineers state a key piece of information: that one estimate of the debris that struck Columbia was 640 times larger than the data used to calibrate the model on which engineers based their damage assessments. (Later analysis showed that the debris object was actually 400 times larger). This difference led Tufte to suggest that a more appropriate headline would be "Review of Test Data Indicates Irrelevance of Two Models." 39
Tufte also criticized the sloppy language on the slide. "The vaguely quantitative words ʻsignificant' and ʻsignificantly' are used 5 times on this slide," he notes, "with de facto meanings ranging from ʻdetectable in largely irrelevant calibration case study' to ʻan amount of damage so that everyone dies' to ʻa difference of 640-fold.' " 40 Another example of sloppiness is that "cubic inches" is written inconsistently: "3cu. In," "1920cu in," and "3 cu in." While such inconsistencies might seem minor, in highly technical fields like aerospace engineering a misplaced decimal point or mistaken unit of measurement can easily engender inconsistencies and inaccuracies. In another phrase "Test results do show that it is possible at sufficient mass and velocity," the word "it" actually refers to "damage to the protective tiles." As information gets passed up an organization hierarchy, from people who do analysis to mid-level managers to high-level leadership, key explanations and supporting information is filtered out. In this context, it is easy to understand how a senior manager might read this PowerPoint slide and not realize that it addresses a life-threatening situation. At many points during its investigation, the Board was surprised to receive similar presentation slides from NASA officials in place of technical reports. The Board views the endemic use of PowerPoint briefing slides instead of technical papers as an illustration of the problematic methods of technical communication at NASA.
The vaguely quantitative words "significant" and "significantly" are used 5 times on this slide, with de facto meanings ranging from "detectable in largely irrelevant calibration case study" to "an amount of damage so that everyone dies" to "a difference of 640-fold." None of these 5 usages appears to refer to the technical meaning of "statistical significance."
Review Of Test Data Indicates Conservatism for Tile Penetration
The existing SOFI on tile test data used to create Crater was reviewed along with STS-107 Southwest Research data – Crater overpredicted penetration of tile coating significantly
• Initial penetration to described by normal velocity Varies with volume/mass of projectile(e.g., 200ft/sec for 3cu. In) • Significant energy is required for the softer SOFI particle to penetrate the relatively hard tile coating Test results do show that it is possible at sufficient mass and velocity • Conversely, once tile is penetrated SOFI can cause significant damage Minor variations in total energy (above penetration level) can cause significant tile damage • Volume of ramp is 1920cu in vs 3 cu in for test
2/21/03 6
The low resolution of PowerPoint slides promotes the use of compressed phrases like "Tile Penetration." As is the case here, such phrases may well be ambiquous. (The low resolution and large font generate 3 typographic orphans, lonely words dangling on a seperate line.) This vague pronoun reference "it" alludes to damage to the protective tiles, which caused the destruction of the Columbia. The slide weakens important material with ambiquous language (sentence fragments, passive voice, multiple meanings of "significant"). The 3 reports were created by engineers for high-level NASA officials who were deciding whether the threat of wing damage required further investigation before the Columbia attempted return. The officials were satisfied that the reports indicated that the Columbia was not in danger, and no attempts to further examine the threat were made. The slides were part of an oral presentation and also were circulated as e-mail attachments. In this slide the same unit of measure for volume (cubic inches) is shown a different way every time 3cu. in 1920cu. in 3 cu. in rather than in clear and tidy exponential form 1920 in 3 . Perhaps the available font cannot show exponents. Shakiness in units of measurement provokes concern. Slides that use hierarchical bullet-outlines here do not handle statistical data and scientific notation gracefully. If PowerPoint is a corporate-mandated format for all engineering reports, then some competent scientific typography (rather than the PP market-pitch style) is essential. In this slide, the typography is so choppy and clunky that it impedes understanding.
– Flight condition is significantly outside of test database
The analysis by Dr. Edward Tufte of the slide from the Debris Assessment Team briefing. [SOFI=Spray-On Foam Insulation]
Report Volume I
August 2003
who might have cautioned against using Crater so far outside its validated limits. Nor did safety personnel provide any additional oversight. NASA failed to connect the dots: the engineers who misinterpreted Crater – a tool already unsuited to the task at hand – were the very ones the Shuttle Program identified as engendering the most risk in their transition from Huntington Beach. The Board views this example as characteristic of the greater turbulence the Shuttle Program experienced in the decade before Columbia as a result of workforce reductions and management reforms. Mission Management Team Shortcomings In the Board's view, the decision to fly STS-113 without a compelling explanation for why bipod foam had separated on ascent during the preceding mission, combined with the low number of Mission Management Team meetings during STS-107, indicates that the Shuttle Program had become overconfident. Over time, the organization determined it did not need daily meetings during a mission, despite regulations that state otherwise. Status update meetings should provide an opportunity to raise concerns and hold discussions across structural and technical boundaries. The leader of such meetings must encourage participation and guarantee that problems are assessed and resolved fully. All voices must be heard, which can be difficult when facing a hierarchy. An employee's location in the hierarchy can encourage silence. Organizations interested in safety must take steps to guarantee that all relevant information is presented to decision-makers. This did not happen in the meetings during the Columbia mission (see Chapter 6). For instance, e-mails from engineers at Johnson and Langley conveyed the depth of their concern about the foam strike, the questions they had about its implications, and the actions they wanted to take as a follow-up. However, these e-mails did not reach the Mission Management Team. The failure to convey the urgency of engineering concerns was caused, at least in part, by organizational structure and spheres of authority. The Langley e-mails were circulated among co-workers at Johnson who explored the possible effects of the foam strike and its consequences for landing. Yet, like Debris Assessment Team Co-Chair Rodney Rocha, they kept their concerns within local channels and did not forward them to the Mission Management Team. They were separated from the decision-making process by distance and rank. Similarly, Mission Management Team participants felt pressured to remain quiet unless discussion turned to their particular area of technological or system expertise, and, even then, to be brief. The initial damage assessment briefing prepared for the Mission Evaluation Room was cut down considerably in order to make it "fit" the schedule. Even so, it took 40 minutes. It was cut down further to a three-minute discussion topic at the Mission Management Team. Tapes of STS-107 Mission Management Team sessions reveal a noticeable "rush" by the meeting's leader to the preconceived bottom line that there was "no safety-of-flight" issue (see Chapter 6). Program managers created huge barriers against dissenting opinions by stating preconceived conclusions based on subjective knowledge and experience, rather than
Report Volume I
on solid data. Managers demonstrated little concern for mission safety. Organizations with strong safety cultures generally acknowledge that a leader's best response to unanimous consent is to play devil's advocate and encourage an exhaustive debate. Mission Management Team leaders failed to seek out such minority opinions. Imagine the difference if any Shuttle manager had simply asked, "Prove to me that Columbia has not been harmed." Similarly, organizations committed to effective communication seek avenues through which unidentified concerns and dissenting insights can be raised, so that weak signals are not lost in background noise. Common methods of bringing minority opinions to the fore include hazard reports, suggestion programs, and empowering employees to call "time out" (Chapter 10). For these methods to be effective, they must mitigate the fear of retribution, and management and technical staff must pay attention. Shuttle Program hazard reporting is seldom used, safety time outs are at times disregarded, and informal efforts to gain support are squelched. The very fact that engineers felt inclined to conduct simulated blown tire landings at Ames "after hours," indicates their reluctance to bring the concern up in established channels. Safety Shortcomings The Board believes that the safety organization, due to a lack of capability and resources independent of the Shuttle Program, was not an effective voice in discussing technical issues or mission operations pertaining to STS-107. The safety personnel present in the Debris Assessment Team, Mission Evaluation Room, and on the Mission Management Team were largely silent during the events leading up to the loss of Columbia. That silence was not merely a failure of safety, but a failure of the entire organization.
The evidence that supports the organizational causes also led the Board to conclude that NASA's current organization, which combines in the Shuttle Program all authority and responsibility for schedule, cost, manifest, safety, technical requirements, and waivers to technical requirements, is not an effective check and balance to achieve safety and mission assurance. Further, NASA's Office of Safety and Mission Assurance does not have the independence and authority that the Board and many outside reviews believe is necessary. Consequently, the Space Shuttle Program does not consistently demonstrate the characteristics of organizations that effectively manage high risk. Therefore, the Board offers the following Findings and Recommendations: Findings: F7.1-1 Throughout its history, NASA has consistently struggled to achieve viable safety programs and adjust them to the constraints and vagaries of changing budgets. Yet, according to multiple high level independent reviews, NASA's safety system has fallen short of the mark.
August 2003
The Associate Administrator for Safety and Mission Assurance is not responsible for safety and mission assurance execution, as intended by the Rogers Commission, but is responsible for Safety and Mission Assurance policy, advice, coordination, and budgets. This view is consistent with NASA's recent philosophy of management at a strategic level at NASA Headquarters but contrary to the Rogers' Commission recommendation. Safety and Mission Assurance organizations supporting the Shuttle Program are largely dependent upon the Program for funding, which hampers their status as independent advisors. Over the last two decades, little to no progress has been made toward attaining integrated, independent, and detailed analyses of risk to the Space Shuttle system. System safety engineering and management is separated from mainstream engineering, is not vigorous enough to have an impact on system design, and is hidden in the other safety disciplines at NASA Headquarters. Risk information and data from hazard analyses are not communicated effectively to the risk assessment and mission assurance processes. The Board could not find adequate application of a process, database, or metric analysis tool that took an integrated, systemic view of the entire Space Shuttle system. The Space Shuttle Systems Integration Office handles all Shuttle systems except the Orbiter. Therefore, it is not a true integration office. When the Integration Office convenes the Integration Control Board, the Orbiter Office usually does not send a representative, and its staff makes verbal inputs only when requested. The Integration office did not have continuous responsibility to integrate responses to bipod foam shedding from various offices. Sometimes the Orbiter Office had responsibility, sometimes the External Tank Office at Marshall Space Flight Center had responsibility, and sometime the bipod shedding did not result in any designation of an In-Flight Anomaly. Integration did not occur. NASA information databases such as The Problem Reporting and Corrective Action and the Web Program Compliance Assurance and Status System are marginally effective decision tools. Senior Safety, Reliability & Quality Assurance and element managers do not use the Lessons Learned Information System when making decisions. NASA subsequently does not have a constructive program to use past lessons to educate engineers, managers, astronauts, or safety personnel. The Space Shuttle Program has a wealth of data tucked away in multiple databases without a convenient way to integrate and use the data for management, engineering, or safety decisions. The dependence of Safety, Reliability & Quality Assurance personnel on Shuttle Program support limits their ability to oversee operations and
Report Volume I
communicate potential problems throughout the organization. There are conflicting roles, responsibilities, and guidance in the Space Shuttle safety programs. The Safety & Mission Assurance Pre-Launch Assessment Review process is not recognized by the Space Shuttle Program as a requirement that must be followed (NSTS 22778). Failure to consistently apply the Pre-Launch Assessment Review as a requirements document creates confusion about roles and responsibilities in the NASA safety organization.
Recommendations: R7.5-1 Establish an independent Technical Engineering Authority that is responsible for technical requirements and all waivers to them, and will build a disciplined, systematic approach to identifying, analyzing, and controlling hazards throughout the life cycle of the Shuttle System. The independent technical authority does the following as a minimum: • Develop and maintain technical standards for all Space Shuttle Program projects and elements • Be the sole waiver-granting authority for all technical standards • Conduct trend and risk analysis at the subsystem, system, and enterprise levels • Own the failure mode, effects analysis and hazard reporting systems • Conduct integrated hazard analysis • Decide what is and is not an anomalous event • Independently verify launch readiness • Approve the provisions of the recertification program called for in Recommendation R9.1-1 The Technical Engineering Authority should be funded directly from NASA Headquarters, and should have no connection to or responsibility for schedule or program cost. NASA Headquarters Office of Safety and Mission Assurance should have direct line authority over the entire Space Shuttle Program safety organization and should be independently resourced. Reorganize the Space Shuttle Integration Office to make it capable of integrating all elements of the Space Shuttle Program, including the Orbiter.
August 2003
Dupont Corporation; Dr. M. Sam Mannan, Texas A&M University; and Mr. Alan C. McMillan, President and Chief Executive Officer, National Safety Council.
17 18
The citations that contain a reference to "CAIB document" with CAB or CTF followed by seven to eleven digits, such as CAB001-0010, refer to a document in the Columbia Accident Investigation Board database maintained by the Department of Justice and archived at the National Archives.
Dr. David Woods of Ohio State University speaking to the Board on HindSight Bias. April 28, 2003. Sagan, The Limits of Safety, p.258. LaPorte and Consolini, "Working In Practice." Notes from "NASA/Navy Benchmarking Exchange (NNBE), Interim Report, Observations & Opportunities Concerning Navy Submarine Program Safety Assurance," Joint NASA and Naval Sea Systems Command NNBE Interim Report, December 20, 2002. Theodore Rockwell, The Rickover Effect, How One Man Made a Difference. (Annapolis, Maryland: Naval Institute Press, 1992), p. 318. Rockwell, Rickover, p. 320. For more information, see Dr. Diane Vaughn, The Challenger Launch Decision, Risky Technology, Culture, and Deviance at NASA (Chicago: University of Chicago Press, 1996). Presentation to the Board by Admiral Walter Cantrell, Aerospace Advisory Panel member, April 7, 2003. Presentation to the Board by Admiral Walter Cantrell, Aerospace Advisory Panel member, April 7, 2003. Aerospace's Launch Verification Process and its Contribution to Titan Risk Management, Briefing given to Board, May 21, 2003, Mr. Ken Holden, General Manager, Launch Verification Division. Joe Tomei, "ELV Launch Risk Assessment Briefing," 3rd Government/ Industry Mission Assurance Forum, Aerospace Corporation, September 24, 2002. NASA Policy Directive 8700.1A, "NASA Policy for Safety and Mission Success", Para 1.b, 5.b(1), 5.e(1), and 5.f(1). Charles B. Perrow. Normal Accidents (New York: Basic Books, 1984). A. Shenhar, "Project management style and the space shuttle program (part 2): A retrospective look," Project Management Journal, 23 (1), pp. 32-37. Harry McDonald, "SIAT Space Shuttle Independent Assessment Team Report." Ibid. "Post Challenger Evaluation of Space Shuttle Risk Assessment and Management Report, National Academy Press 1988," section 5.1, pg. 40. Harry McDonald, "SIAT Space Shuttle Independent Assessment Team Report." NSTS-22254 Rev B. Ibid. GAO Report, "Survey of NASA Lessons Learned," GAO-01-1015R, September 5, 2001. E. Tufte, Beautiful Evidence (Cheshire, CT: Graphics Press). [in press.] Ibid., Edward R. Tufte, "The Cognitive Style of PowerPoint," (Cheshire, CT: Graphics Press, May 2003). Ibid.
Sylvia Kramer, "History of NASA Safety Office from 1958-1980's," NASA History Division Record Collection, 1986, p. 1. CAIB document CAB065-0358. Ralph M. Miles Jr. "Introduction." In Ralph M. Miles Jr., editor, System Concepts: Lectures on Contemporary Approaches to Systems, p. 1-12 (New York: John F. Wiley & Sons, 1973). "The Aerospace Safety Advisory Panel, " NASA History Office, July 1, 1987, p. 1. On Rodney's appointment, see NASA Management Instruction 1103.39, July 3, 1986, and NASA News July 8, 1986. NASA Facts, "Brief Overview, Office of Safety, Reliability, Maintainability and Quality Assurance," circa 1987. "Space Program Safety: Funding for NASA's Safety Organizations Should Be Centralized," General Accounting Office Report, NSIAD-90187, 1990. "Aerospace Safety Advisory Panel Annual Report," 1996. The quotes are from the Executive Summary of National Aeronautics and Space Administration Space Shuttle Independent Assessment Team, "Report to Associate Administrator, Office of Space Flight," OctoberDecember 1999. CAIB document CTF017-0169. Harry McDonald, "SIAT Space Shuttle Independent Assessment Team Report." NASA Chief Engineer and NASA Integrated Action Team, "Enhancing Mission Success – A Framework for the Future," December 21, 2000. The information in this section is derived from a briefing titled, "Draft Final Report of the Space Shuttle Competitive Source Task Force," July 12, 2002. Mr. Liam Sarsfield briefed this report to NASA Headquarters. Dr. Karl Weick, University of Michigan; Dr. Karlene Roberts, University of California-Berkley; Dr. Howard McCurdy, American University; and Dr. Diane Vaughan, Boston College. Dr. David Woods, Ohio State University; Dr. Nancy G. Leveson, Massachusetts Institute of Technology; Mr. James Wick, Intel Corporation; Ms. Deborah L. Grubbe, DuPont Corporation; Dr. M. Sam Mannan, Texas A&M University; Douglas A. Wiegmann, University of Illinois at Urbana-Champaign; and Mr. Alan C. McMillan, President and Chief Executive Officer, National Safety Council. Todd R. La Porte and Paula M. Consolini, "Working in Practice but Not in Theory," Journal of Public Administration Research and Theory, 1 (1991) pp. 19-47. Scott Sagan, The Limits of Safety (Princeton: Princeton University Press, 1995). Dr. Diane Vaughan, Boston College; Dr. David Woods, Ohio State University; Dr. Howard E. McCurdy, American University; Dr. Karl E. Weick, University of Michigan; Dr. Karlene H. Roberts; Dr. M. Elisabeth Paté-Cornell; Dr. Douglas A. Wiegmann, University of Illinois at Urbana-Champaign; Dr. Nancy G. Leveson, Massachusetts Institute of Technology; Mr. James Wick, Intel Corporation; Ms. Deborah L. Grubbe, Report Volume I
3 4 5 6
21 22 23
24 25 26
7 8
9 10 11
28 29 30
31 32 33
34 35 36 37 38 39 40
15 16
August 2003
History As Cause: Columbia and Challenger
The Board began its investigation with two central questions about NASA decisions. Why did NASA continue to fly with known foam debris problems in the years preceding the Columbia launch, and why did NASA managers conclude that the foam debris strike 81.9 seconds into Columbia's flight was not a threat to the safety of the mission, despite the concerns of their engineers? engineers' decision-making, the three worked together. Each is a critical link in the causal chain. This chapter shows that both accidents were "failures of foresight" in which history played a prominent role.1 First, the history of engineering decisions on foam and O-ring incidents had identical trajectories that "normalized" these anomalies, so that flying with these flaws became routine and acceptable. Second, NASA history had an effect. In response to White House and Congressional mandates, NASA leaders took actions that created systemic organizational flaws at the time of Challenger that were also present for Columbia. The final section compares the two critical decision sequences immediately before the loss of both Orbiters – the pre-launch teleconference for Challenger and the post-launch foam strike discussions for Columbia. It shows history again at work: how past definitions of risk combined with systemic problems in the NASA organization caused both accidents. Connecting the parts of NASA's organizational system and drawing the parallels with Challenger demonstrate three things. First, despite all the post-Challenger changes at NASA and the agency's notable achievements since, the causes of the institutional failure responsible for Challenger have not been fixed. Second, the Board strongly believes that if these persistent, systemic flaws are not resolved, the scene is set for another accident. Therefore, the recommendations for change are not only for fixing the Shuttle's technical system, but also for fixing each part of the organizational system that produced Columbia's failure. Third, the Board's focus on the context in which decision making occurred does not mean that individuals are not responsible and accountable. To the contrary, individuals always must assume responsibility for their actions. What it does mean is that NASA's problems cannot be solved simply by retirements, resignations, or transferring personnel.2 The constraints under which the agency has operated throughout the Shuttle Program have contributed to both
August 2003
As the investigation progressed, Board member Dr. Sally Ride, who also served on the Rogers Commission, observed that there were "echoes" of Challenger in Columbia. Ironically, the Rogers Commission investigation into Challenger started with two remarkably similar central questions: Why did NASA continue to fly with known O-ring erosion problems in the years before the Challenger launch, and why, on the eve of the Challenger launch, did NASA managers decide that launching the mission in such cold temperatures was an acceptable risk, despite the concerns of their engineers? The echoes did not stop there. The foam debris hit was not the single cause of the Columbia accident, just as the failure of the joint seal that permitted O-ring erosion was not the single cause of Challenger. Both Columbia and Challenger were lost also because of the failure of NASA's organizational system. Part Two of this report cites failures of the three parts of NASA's organizational system. This chapter shows how previous political, budgetary, and policy decisions by leaders at the White House, Congress, and NASA (Chapter 5) impacted the Space Shuttle Program's structure, culture, and safety system (Chapter 7), and how these in turn resulted in flawed decision-making (Chapter 6) for both accidents. The explanation is about system effects: how actions taken in one layer of NASA's organizational system impact other layers. History is not just a backdrop or a scene-setter. History is cause. History set the Columbia and Challenger accidents in motion. Although Part Two is separated into chapters and sections to make clear what happened in the political environment, the organization, and managers' and
Report Volume I
Shuttle accidents. Although NASA leaders have played an important role, these constraints were not entirely of NASA's own making. The White House and Congress must recognize the role of their decisions in this accident and take responsibility for safety in the future.
Foam loss may have occurred on all missions, and left bipod ramp foam loss occurred on 10 percent of the flights for which visible evidence exists. The Board had a hard time understanding how, after the bitter lessons of Challenger, NASA could have failed to identify a similar trend. Rather than view the foam decision only in hindsight, the Board tried to see the foam incidents as NASA engineers and managers saw them as they made their decisions. This section gives an insider perspective: how NASA defined risk and how those definitions changed over time for both foam debris hits and O-ring erosion. In both cases, engineers and managers conducting risk assessments continually normalized the technical deviations they found.3 In all official engineering analyses and launch recommendations prior to the accidents, evidence that the design was not performing as expected was reinterpreted as acceptable and non-deviant, which diminished perceptions of risk throughout the agency. The initial Shuttle design predicted neither foam debris problems nor poor sealing action of the Solid Rocket Booster joints. To experience either on a mission was a violation of design specifications. These anomalies were signals of potential danger, not something to be tolerated, but in both cases after the first incident the engineering analysis concluded that the design could tolerate the damage. These engineers decided to implement a temporary fix and/or accept the risk, and fly. For both O-rings and foam, that first decision was a turning point. It established a precedent for accepting, rather than eliminating, these technical deviations. As a result of this new classification, subsequent incidents of O-ring erosion or foam debris strikes were not defined as signals of danger, but as evidence that the design was now acting as predicted. Engineers and managers incorporated worsening anomalies into the engineering experience base, which functioned as an elastic waistband, expanding to hold larger deviations from the original design. Anomalies that did not lead to catastrophic failure were treated as a source of valid engineering data that justified further flights. These anomalies were translated into a safety margin that was extremely influential, allowing engineers and managers to add incrementally to the amount and seriousness of damage that was acceptable. Both O-ring erosion and foam debris events were repeatedly "addressed" in NASA's Flight Readiness Reviews but never fully resolved. In both cases, the engineering analysis was incomplete and inadequate. Engineers understood what was happening, but they never understood why. NASA continued to implement a series of small corrective actions, living with the problems until it was too late.4 NASA documents show how official classifications of risk were downgraded over time.5 Program managers designated both the foam problems and O-ring erosion as "acceptable
Report Volume I
risks" in Flight Readiness Reviews. NASA managers also assigned each bipod foam event In-Flight Anomaly status, and then removed the designation as corrective actions were implemented. But when major bipod foam-shedding occurred on STS-112 in October 2002, Program management did not assign an In-Flight Anomaly. Instead, it downgraded the problem to the lower status of an "action" item. Before Challenger, the problematic Solid Rocket Booster joint had been elevated to a Criticality 1 item on NASA's Critical Items List, which ranked Shuttle components by failure consequences and noted why each was an acceptable risk. The joint was later demoted to a Criticality 1-R (redundant), and then in the month before Challenger's launch was "closed out" of the problem-reporting system. Prior to both accidents, this demotion from high-risk item to low-risk item was very similar, but with some important differences. Damaging the Orbiter's Thermal Protection System, especially its fragile tiles, was normalized even before Shuttle launches began: it was expected due to forces at launch, orbit, and re-entry.6 So normal was replacement of Thermal Protection System materials that NASA managers budgeted for tile cost and turnaround maintenance time from the start. It was a small and logical next step for the discovery of foam debris damage to the tiles to be viewed by NASA as part of an already existing maintenance problem, an assessment based on experience, not on a thorough hazard analysis. Foam debris anomalies came to be categorized by the reassuring term "in-family," a formal classification indicating that new occurrences of an anomaly were within the engineering experience base. "In-family" was a strange term indeed for a violation of system requirements. Although "in-family" was a designation introduced post-Challenger to separate problems by seriousness so that "out-of-family" problems got more attention, by definition the problems that were shifted into the lesser "in-family" category got less attention. The Board's investigation uncovered no paper trail showing escalating concern about the foam problem like the one that Solid Rocket Booster engineers left prior to Challenger.7 So ingrained was the agency's belief that foam debris was not a threat to flight safety that in press briefings after the Columbia accident, the Space Shuttle Program Manager still discounted the foam as a probable cause, saying that Shuttle managers were "comfortable" with their previous risk assessments. From the beginning, NASA's belief about both these problems was affected by the fact that engineers were evaluating them in a work environment where technical problems were normal. Although management treated the Shuttle as operational, it was in reality an experimental vehicle. Many anomalies were expected on each mission. Against this backdrop, an anomaly was not in itself a warning sign of impending catastrophe. Another contributing factor was that both foam debris strikes and O-ring erosion events were examined separately, one at a time. Individual incidents were not read by engineers as strong signals of danger. What NASA engineers and managers saw were pieces of illstructured problems.8 An incident of O-ring erosion or foam bipod debris would be followed by several launches where the machine behaved properly, so that signals of danger
August 2003
were followed by all-clear signals – in other words, NASA managers and engineers were receiving mixed signals.9 Some signals defined as weak at the time were, in retrospect, warnings of danger. Foam debris damaged tile was assumed (erroneously) not to pose a danger to the wing. If a primary O-ring failed, the secondary was assumed (erroneously) to provide a backup. Finally, because foam debris strikes were occurring frequently, like O-ring erosion in the years before Challenger, foam anomalies became routine signals – a normal part of Shuttle operations, not signals of danger. Other anomalies gave signals that were strong, like wiring malfunctions or the cracked balls in Ball Strut Tie Rod Assemblies, which had a clear relationship to a "loss of mission." On those occasions, NASA stood down from launch, sometimes for months, while the problems were corrected. In contrast, foam debris and eroding O-rings were defined as nagging issues of seemingly little consequence. Their significance became clear only in retrospect, after lives had been lost. History became cause as the repeating pattern of anomalies was ratified as safe in Flight Readiness Reviews. The official definitions of risk assigned to each anomaly in Flight Readiness Reviews limited the actions taken and the resources spent on these problems. Two examples of the road not taken and the devastating implications for the future occurred close in time to both accidents. On the October 2002 launch of STS-112, a large piece of bipod ramp foam hit and damaged the External Tank Attachment ring on the Solid Rocket Booster skirt, a strong signal of danger 10 years after the last known bipod ramp foam event. Prior to Challenger, there was a comparable surprise. After a January 1985 launch, for which the Shuttle sat on the launch pad for three consecutive nights of unprecedented cold temperatures, engineers discovered upon the Orbiter's return that hot gases had eroded the primary and reached the secondary O-ring, blackening the putty in between – an indication that the joint nearly failed. But accidents are not always preceded by a wake-up call.10 In 1985, engineers realized they needed data on the relationship between cold temperatures and O-ring erosion. However, the task of getting better temperature data stayed on the back burner because of the definition of risk: the primary erosion was within the experience base; the secondary O-ring (thought to be redundant) was not damaged and, significantly, there was a low probability that such cold Florida temperatures would recur.11 The scorched putty, initially a strong signal, was redefined after analysis as weak. On the eve of the Challenger launch, when cold temperature became a concern, engineers had no test data on the effect of cold temperatures on O-ring erosion. Before Columbia, engineers concluded that the damage from the STS-112 foam hit in October 2002 was not a threat to flight safety. The logic was that, yes, the foam piece was large and there was damage, but no serious consequences followed. Further, a hit this size, like cold temperature, was a low-probability event. After analysis, the biggest foam hit to date was redefined as a weak signal. Similar self-defeating actions and inactions followed. Engineers were again dealing with the poor quality of tracking camera images of strikes during ascent. Yet NASA took no steps to improve imagery and took no immediate action to reduce the risk of bipod ramp
Report Volume I
foam shedding and potential damage to the Orbiter before Columbia. Furthermore, NASA performed no tests on what would happen if a wing leading edge were struck by bipod foam, even though foam had repeatedly separated from the External Tank. During the Challenger investigation, Rogers Commission member Dr. Richard Feynman famously compared launching Shuttles with known problems to playing Russian roulette.12 But that characterization is only possible in hindsight. It is not how NASA personnel perceived the risks as they were being assessed, one launch at a time. Playing Russian roulette implies that the pistol-holder realizes that death might be imminent and still takes the risk. For both foam debris and O-ring erosion, fixes were in the works at the time of the accidents, but there was no rush to complete them because neither problem was defined as a show-stopper. Each time an incident occurred, the Flight Readiness process declared it safe to continue flying. Taken one at a time, each decision seemed correct. The agency allocated attention and resources to these two problems accordingly. The consequences of living with both of these anomalies were, in its view, minor. Not all engineers agreed in the months immediately preceding Challenger, but the dominant view at NASA – the managerial view – was, as one manager put it, "we were just eroding rubber O-rings," which was a low-cost problem.13 The financial consequences of foam debris also were relatively low: replacing tiles extended the turnaround time between launches. In both cases, NASA was comfortable with its analyses. Prior to each accident, the agency saw no greater consequences on the horizon.
The series of engineering decisions that normalized technical deviations shows one way that history became cause in both accidents. But NASA's own history encouraged this pattern of flying with known flaws. Seventeen years separated the two accidents. NASA Administrators, Congresses, and political administrations changed. However, NASA's political and budgetary situation remained the same in principle as it had been since the inception of the Shuttle Program. NASA remained a politicized and vulnerable agency, dependent on key political players who accepted NASA's ambitious proposals and then imposed strict budget limits. Post-Challenger policy decisions made by the White House, Congress, and NASA leadership resulted in the agency reproducing many of the failings identified by the Rogers Commission. Policy constraints affected the Shuttle Program's organization culture, its structure, and the structure of the safety system. The three combined to keep NASA on its slippery slope toward Challenger and Columbia. NASA culture allowed flying with flaws when problems were defined as normal and routine; the structure of NASA's Shuttle Program blocked the flow of critical information up the hierarchy, so definitions of risk continued unaltered. Finally, a perennially weakened safety system, unable to critically analyze and intervene, had no choice but to ratify the existing risk assessments on these two problems. The following comparison shows that these system effects persisted through time, and affected engineering decisions in the years leading up to both accidents.
August 2003
The Board found that dangerous aspects of NASA's 1986 culture, identified by the Rogers Commission, remained unchanged. The Space Shuttle Program had been built on compromises hammered out by the White House and NASA headquarters.14 As a result, NASA was transformed from a research and development agency to more of a business, with schedules, production pressures, deadlines, and cost efficiency goals elevated to the level of technical innovation and safety goals.15 The Rogers Commission dedicated an entire chapter of its report to production pressures.16 Moreover, the Rogers Commission, as well as the 1990 Augustine Committee and the 1999 Shuttle Independent Assessment Team, criticized NASA for treating the Shuttle as if it were an operational vehicle. Launching on a tight schedule, which the agency had pursued as part of its initial bargain with the White House, was not the way to operate what was in fact an experimental vehicle. The Board found that prior to Columbia, a budget-limited Space Shuttle Program, forced again and again to refashion itself into an efficiency model because of repeated government cutbacks, was beset by these same ills. The harmful effects of schedule pressure identified in previous reports had returned. Prior to both accidents, NASA was scrambling to keep up. Not only were schedule pressures impacting the people who worked most closely with the technology – technicians, mission operators, flight crews, and vehicle processors – engineering decisions also were affected.17 For foam debris and O-ring erosion, the definition of risk established during the Flight Readiness process determined actions taken and not taken, but the schedule and shoestring budget were equally influential. NASA was cutting corners. Launches proceeded with incomplete engineering work on these flaws. Challenger-era engineers were working on a permanent fix for the booster joints while launches continued.18 After the major foam bipod hit on STS-112, management made the deadline for corrective action on the foam problem after the next launch, STS-113, and then slipped it again until after the flight of STS-107. Delays for flowliner and Ball Strut Tie Rod Assembly problems left no margin in the schedule between February 2003 and the managementimposed February 2004 launch date for the International Space Station Node 2. Available resources – including time out of the schedule for research and hardware modifications – went to the problems that were designated as serious – those most likely to bring down a Shuttle. The NASA culture encouraged flying with flaws because the schedule could not be held up for routine problems that were not defined as a threat to mission safety.19 The question the Board had to answer was why, since the foam debris anomalies went on for so long, had no one recognized the trend and intervened? The O-ring history prior to Challenger had followed the same pattern. This question pointed the Board's attention toward the NASA organization structure and the structure of its safety system. Safetyoriented organizations often build in checks and balances to identify and monitor signals of potential danger. If these checks and balances were in place in the Shuttle Program, they weren't working. Again, past policy decisions produced system effects with implications for both Challenger and Columbia.
Report Volume I
Prior to Challenger, Shuttle Program structure had hindered information flows, leading the Rogers Commission to conclude that critical information about technical problems was not conveyed effectively through the hierarchy.20 The Space Shuttle Program had altered its structure by outsourcing to contractors, which added to communication problems. The Commission recommended many changes to remedy these problems, and NASA made many of them. However, the Board found that those post-Challenger changes were undone over time by management actions.21 NASA administrators, reacting to government pressures, transferred more functions and responsibilities to the private sector. The change was cost-efficient, but personnel cuts reduced oversight of contractors at the same time that the agency's dependence upon contractor engineering judgment increased. When high-risk technology is the product and lives are at stake, safety, oversight, and communication flows are critical. The Board found that the Shuttle Program's normal chain of command and matrix system did not perform a check-and-balance function on either foam or O-rings. The Flight Readiness Review process might have reversed the disastrous trend of normalizing O-ring erosion and foam debris hits, but it didn't. In fact, the Rogers Commission found that the Flight Readiness process only affirmed the pre-Challenger engineering risk assessments.22 Equally troubling, the Board found that the Flight Readiness process, which is built on consensus verified by signatures of all responsible parties, in effect renders no one accountable. Although the process was altered after Challenger, these changes did not erase the basic problems that were built into the structure of the Flight Readiness Review.23 Managers at the top were dependent on engineers at the bottom for their engineering analysis and risk assessments. Information was lost as engineering risk analyses moved through the process. At succeeding stages, management awareness of anomalies, and therefore risks, was reduced either because of the need to be increasingly brief and concise as all the parts of the system came together, or because of the need to produce consensus decisions at each level. The Flight Readiness process was designed to assess hardware and take corrective actions that would transform known problems into acceptable flight risks, and that is precisely what it did. The 1986 House Committee on Science and Technology concluded during its investigation into Challenger that Flight Readiness Reviews had performed exactly as they were designed, but that they could not be expected to replace engineering analysis, and therefore they "cannot be expected to prevent a flight because of a design flaw that Project management had already determined an acceptable risk."24 Those words, true for the history of O-ring erosion, also hold true for the history of foam debris. The last line of defense against errors is usually a safety system. But the previous policy decisions by leaders described in Chapter 5 also impacted the safety structure and contributed to both accidents. Neither in the O-ring erosion nor the foam debris problems did NASA's safety system attempt to reverse the course of events. In 1986, the Rogers Commission called it "The Silent Safety System."25 Pre-Challenger budget shortages resulted in safety personnel cutbacks. Without clout or independence, the
August 2003
safety personnel who remained were ineffective. In the case of Columbia, the Board found the same problems were reproduced and for an identical reason: when pressed for cost reduction, NASA attacked its own safety system. The faulty assumption that supported this strategy prior to Columbia was that a reduction in safety staff would not result in a reduction of safety, because contractors would assume greater safety responsibility. The effectiveness of those remaining staff safety engineers was blocked by their dependence on the very Program they were charged to supervise. Also, the Board found many safety units with unclear roles and responsibilities that left crucial gaps. Post-Challenger NASA still had no systematic procedure for identifying and monitoring trends. The Board was surprised at how long it took NASA to put together trend data in response to Board requests for information. Problem reporting and tracking systems were still overloaded or underused, which undermined their very purpose. Multiple job titles disguised the true extent of safety personnel shortages. The Board found cases in which the same person was occupying more than one safety position – and in one instance at least three positions – which compromised any possibility of safety organization independence because the jobs were established with built-in conflicts of interest.
treating it as if it were.29 The Board found that even after the loss of Challenger, NASA was guilty of treating an experimental vehicle as if it were operational and of not listening to outside experts. In a repeat of the pre-Challenger warning, the 1999 Shuttle Independent Assessment Team report reiterated that "the Shuttle was not an ʻoperational' vehicle in the usual meaning of the term."30 Engineers and program planners were also affected by "Can-Do," which, when taken too far, can create a reluctance to say that something cannot be done. How could the lessons of Challenger have been forgotten so quickly? Again, history was a factor. First, if success is measured by launches and landings,31 the machine appeared to be working successfully prior to both accidents. Challenger was the 25th launch. Seventeen years and 87 missions passed without major incident. Second, previous policy decisions again had an impact. NASA's Apollo-era research and development culture and its prized deference to the technical expertise of its working engineers was overridden in the Space Shuttle era by "bureaucratic accountability" – an allegiance to hierarchy, procedure, and following the chain of command.32 Prior to Challenger, the can-do culture was a result not just of years of apparently successful launches, but of the cultural belief that the Shuttle Program's many structures, rigorous procedures, and detailed system of rules were responsible for those successes.33 The Board noted that the pre-Challenger layers of processes, boards, and panels that had produced a false sense of confidence in the system and its level of safety returned in full force prior to Columbia. NASA made many changes to the Space Shuttle Program structure after Challenger. The fact that many changes had been made supported a belief in the safety of the system, the invincibility of organizational and technical systems, and ultimately, a sense that the foam problem was understood.
A number of changes to the Space Shuttle Program structure made in response to policy decisions had the unintended effect of perpetuating dangerous aspects of pre-Challenger culture and continued the pattern of normalizing things that were not supposed to happen. At the same time that NASA leaders were emphasizing the importance of safety, their personnel cutbacks sent other signals. Streamlining and downsizing, which scarcely go unnoticed by employees, convey a message that efficiency is an important goal. The Shuttle/Space Station partnership affected both programs. Working evenings and weekends just to meet the International Space Station Node 2 deadline sent a signal to employees that schedule is important. When paired with the "faster, better, cheaper" NASA motto of the 1990s and cuts that dramatically decreased safety personnel, efficiency becomes a strong signal and safety a weak one. This kind of doublespeak by top administrators affects people's decisions and actions without them even realizing it.26 Changes in Space Shuttle Program structure contributed to the accident in a second important way. Despite the constraints that the agency was under, prior to both accidents NASA appeared to be immersed in a culture of invincibility, in stark contradiction to post-accident reality. The Rogers Commission found a NASA blinded by its "Can-Do" attitude,27 a cultural artifact of the Apollo era that was inappropriate in a Space Shuttle Program so strapped by schedule pressures and shortages that spare parts had to be cannibalized from one vehicle to launch another.28 This can-do attitude bolstered administrators' belief in an achievable launch rate, the belief that they had an operational system, and an unwillingness to listen to outside experts. The Aerospace Safety and Advisory Panel in a 1985 report told NASA that the vehicle was not operational and NASA should stop
Report Volume I
Risk, uncertainty, and history came together when unprecedented circumstances arose prior to both accidents. For Challenger, the weather prediction for launch time the next day was for cold temperatures that were out of the engineering experience base. For Columbia, a large foam hit – also outside the experience base – was discovered after launch. For the first case, all the discussion was pre-launch; for the second, it was post-launch. This initial difference determined the shape these two decision sequences took, the number of people who had information about the problem, and the locations of the involved parties. For Challenger, engineers at Morton-Thiokol,34 the Solid Rocket Motor contractor in Utah, were concerned about the effect of the unprecedented cold temperatures on the rubber O-rings.35 Because launch was scheduled for the next morning, the new condition required a reassessment of the engineering analysis presented at the Flight Readiness Review two weeks prior. A teleconference began at 8:45 p.m. Eastern Standard Time (EST) that included 34 people in three locations: Morton-Thiokol in Utah, Marshall, and Kennedy. Thiokol engineers were recommending a launch delay. A reconsideration of a Flight Readiness Review risk
August 2003
assessment the night before a launch was as unprecedented as the predicted cold temperatures. With no ground rules or procedures to guide their discussion, the participants automatically reverted to the centralized, hierarchical, tightly structured, and procedure-bound model used in Flight Readiness Reviews. The entire discussion and decision to launch began and ended with this group of 34 engineers. The phone conference linking them together concluded at 11:15 p.m. EST after a decision to accept the risk and fly. For Columbia, information about the foam debris hit was widely distributed the day after launch. Time allowed for videos of the strike, initial assessments of the size and speed of the foam, and the approximate location of the impact to be dispersed throughout the agency. This was the first debris impact of this magnitude. Engineers at the Marshall, Johnson, Kennedy, and Langley centers showed initiative and jumped on the problem without direction from above. Working groups and e-mail groups formed spontaneously. The size of Johnson's Debris Assessment Team alone neared and in some instances exceeded the total number of participants in the 1986 Challenger teleconference. Rather than a tightly constructed exchange of information completed in a few hours, time allowed for the development of ideas and free-wheeling discussion among the engineering ranks. The early post-launch discussion among engineers and all later decision-making at management levels were decentralized, loosely organized, and with little form. While the spontaneous and decentralized exchanging of information was evidence that NASA's original technical culture was alive and well, the diffuse form and lack of structure in the rest of the proceedings would have several negative consequences. In both situations, all new information was weighed and interpreted against past experience. Formal categories and cultural beliefs provide a consistent frame of reference in which people view and interpret information and experiences.36 Pre-existing definitions of risk shaped the actions taken and not taken. Worried engineers in 1986 and again in 2003 found it impossible to reverse the Flight Readiness Review risk assessments that foam and O-rings did not pose safety-of-flight concerns. These engineers could not prove that foam strikes and cold temperatures were unsafe, even though the previous analyses that declared them safe had been incomplete and were based on insufficient data and testing. Engineers' failed attempts were not just a matter of psychological frames and interpretations. The obstacles these engineers faced were political and organizational. They were rooted in NASA history and the decisions of leaders that had altered NASA culture, structure, and the structure of the safety system and affected the social context of decision-making for both accidents. In the following comparison of these critical decision scenarios for Columbia and Challenger, the systemic problems in the NASA organization are in italics, with the system effects on decisionmaking following. NASA had conflicting goals of cost, schedule, and safety. Safety lost out as the mandates of an "operational system" increased the schedule pressure. Scarce resources went to problems that were defined as more serious, rather than to foam strikes or O-ring erosion.
Report Volume I
In both situations, upper-level managers and engineering teams working the O-ring and foam strike problems held opposing definitions of risk. This was demonstrated immediately, as engineers reacted with urgency to the immediate safety implications: Thiokol engineers scrambled to put together an engineering assessment for the teleconference, Langley Research Center engineers initiated simulations of landings that were run after hours at Ames Research Center, and Boeing analysts worked through the weekend on the debris impact analysis. But key managers were responding to additional demands of cost and schedule, which competed with their safety concerns. NASA's conflicting goals put engineers at a disadvantage before these new situations even arose. In neither case did they have good data as a basis for decision-making. Because both problems had been previously normalized, resources sufficient for testing or hardware were not dedicated. The Space Shuttle Program had not produced good data on the correlation between cold temperature and O-ring resilience or good data on the potential effect of bipod ramp foam debris hits.37 Cultural beliefs about the low risk O-rings and foam debris posed, backed by years of Flight Readiness Review decisions and successful missions, provided a frame of reference against which the engineering analyses were judged. When confronted with the engineering risk assessments, top Shuttle Program managers held to the previous Flight Readiness Review assessments. In the Challenger teleconference, where engineers were recommending that NASA delay the launch, the Marshall Solid Rocket Booster Project manager, Lawrence Mulloy, repeatedly challenged the contractor's risk assessment and restated Thiokol's engineering rationale for previous flights.38 STS-107 Mission Management Team Chair Linda Ham made many statements in meetings reiterating her understanding that foam was a maintenance problem and a turnaround issue, not a safety-of-flight issue. The effects of working as a manager in a culture with a cost/ efficiency/safety conflict showed in managerial responses. In both cases, managers' techniques focused on the information that tended to support the expected or desired result at that time. In both cases, believing the safety of the mission was not at risk, managers drew conclusions that minimized the risk of delay.39 At one point, Marshall's Mulloy, believing in the previous Flight Readiness Review assessments, unconvinced by the engineering analysis, and concerned about the schedule implications of the 53-degree temperature limit on launch the engineers proposed, said, "My God, Thiokol, when do you want me to launch, next April?"40 Reflecting the overall goal of keeping to the Node 2 launch schedule, Ham's priority was to avoid the delay of STS–114, the next mission after STS-107. Ham was slated as Manager of Launch Integration for STS-114 – a dual role promoting a conflict of interest and a single-point failure, a situation that should be avoided in all organizational as well as technical systems. NASA's culture of bureaucratic accountability emphasized chain of command, procedure, following the rules, and going by the book. While rules and procedures were essential for coordination, they had an unintended but negative effect. Allegiance to hierarchy and procedure had replaced deference to NASA engineers' technical expertise.
August 2003
In both cases, engineers initially presented concerns as well as possible solutions – a request for images, a recommendation to place temperature constraints on launch. Management did not listen to what their engineers were telling them. Instead, rules and procedures took priority. For Columbia, program managers turned off the Kennedy engineers' initial request for Department of Defense imagery, with apologies to Defense Department representatives for not having followed "proper channels." In addition, NASA administrators asked for and promised corrective action to prevent such a violation of protocol from recurring. Debris Assessment Team analysts at Johnson were asked by managers to demonstrate a "mandatory need" for their imagery request, but were not told how to do that. Both Challenger and Columbia engineering teams were held to the usual quantitative standard of proof. But it was a reverse of the usual circumstance: instead of having to prove it was safe to fly, they were asked to prove that it was unsafe to fly. In the Challenger teleconference, a key engineering chart presented a qualitative argument about the relationship between cold temperatures and O-ring erosion that engineers were asked to prove. Thiokol's Roger Boisjoly said, "I had no data to quantify it. But I did say I knew it was away from goodness in the current data base."41 Similarly, the Debris Assessment Team was asked to prove that the foam hit was a threat to flight safety, a determination that only the imagery they were requesting could help them make. Ignored by management was the qualitative data that the engineering teams did have: both instances were outside the experience base. In stark contrast to the requirement that engineers adhere to protocol and hierarchy was management's failure to apply this criterion to their own activities. The Mission Management Team did not meet on a regular schedule during the mission, proceeded in a loose format that allowed informal influence and status differences to shape their decisions, and allowed unchallenged opinions and assumptions to prevail, all the while holding the engineers who were making risk assessments to higher standards. In highly uncertain circumstances, when lives were immediately at risk, management failed to defer to its engineers and failed to recognize that different data standards – qualitative, subjective, and intuitive – and different processes – democratic rather than protocol and chain of command – were more appropriate. The organizational structure and hierarchy blocked effective communication of technical problems. Signals were overlooked, people were silenced, and useful information and dissenting views on technical issues did not surface at higher levels. What was communicated to parts of the organization was that O-ring erosion and foam debris were not problems. Structure and hierarchy represent power and status. For both Challenger and Columbia, employees' positions in the organization determined the weight given to their information, by their own judgment and in the eyes of others. As a result, many signals of danger were missed. Relevant information that could have altered the course of events was available but was not presented. Early in the Challenger teleconference, some engineers who had important information did not speak up. They did not
Report Volume I
define themselves as qualified because of their position: they were not in an appropriate specialization, had not recently worked the O-ring problem, or did not have access to the "good data" that they assumed others more involved in key discussions would have.42 Geographic locations also resulted in missing signals. At one point, in light of Marshall's objections, Thiokol managers in Utah requested an "off-line caucus" to discuss their data. No consensus was reached, so a "management risk decision" was made. Managers voted and engineers did not. Thiokol managers came back on line, saying they had reversed their earlier NO-GO recommendation, decided to accept risk, and would send new engineering charts to back their reversal. When a Marshall administrator asked, "Does anyone have anything to add to this?," no one spoke. Engineers at Thiokol who still objected to the decision later testified that they were intimidated by management authority, were accustomed to turning their analysis over to managers and letting them decide, and did not have the quantitative data that would empower them to object further.43 In the more decentralized decision process prior to Columbia's re-entry, structure and hierarchy again were responsible for an absence of signals. The initial request for imagery came from the "low status" Kennedy Space Center, bypassed the Mission Management Team, and went directly to the Department of Defense separate from the all-powerful Shuttle Program. By using the Engineering Directorate avenue to request imagery, the Debris Assessment Team was working at the margins of the hierarchy. But some signals were missing even when engineers traversed the appropriate channels. The Mission Management Team Chair's position in the hierarchy governed what information she would or would not receive. Information was lost as it traveled up the hierarchy. A demoralized Debris Assessment Team did not include a slide about the need for better imagery in their presentation to the Mission Evaluation Room. Their presentation included the Crater analysis, which they reported as incomplete and uncertain. However, the Mission Evaluation Room manager perceived the Boeing analysis as rigorous and quantitative. The choice of headings, arrangement of information, and size of bullets on the key chart served to highlight what management already believed. The uncertainties and assumptions that signaled danger dropped out of the information chain when the Mission Evaluation Room manager condensed the Debris Assessment Team's formal presentation to an informal verbal brief at the Mission Management Team meeting. As what the Board calls an "informal chain of command" began to shape STS-107's outcome, location in the structure empowered some to speak and silenced others. For example, a Thermal Protection System tile expert, who was a member of the Debris Assessment Team but had an office in the more prestigious Shuttle Program, used his personal network to shape the Mission Management Team view and snuff out dissent. The informal hierarchy among and within Centers was also influential. Early identifications of problems by Marshall and Kennedy may have contributed to the Johnson-based Mission Management Team's indifference to concerns about the foam strike. The engineers and managers circulating e-mails at Langley were peripheral to the Shuttle Program, not structurally connected to the proceedings, and
August 2003
therefore of lower status. When asked in a post-accident press conference why they didn't voice their concerns to Shuttle Program management, the Langley engineers said that people "need to stick to their expertise."44 Status mattered. In its absence, numbers were the great equalizer. One striking exception: the Debris Assessment Team tile expert was so influential that his word was taken as gospel, though he lacked the requisite expertise, data, or analysis to evaluate damage to RCC. For those with lesser standing, the requirement for data was stringent and inhibiting, which resulted in information that warned of danger not being passed up the chain. As in the teleconference, Debris Assessment Team engineers did not speak up when the Mission Management Team Chair asked if anyone else had anything to say. Not only did they not have the numbers, they also were intimidated by the Mission Management Team Chair's position in the hierarchy and the conclusions she had already made. Debris Assessment Team members signed off on the Crater analysis, even though they had trouble understanding it. They still wanted images of Columbia's left wing. In neither impending crisis did management recognize how structure and hierarchy can silence employees and follow through by polling participants, soliciting dissenting opinions, or bringing in outsiders who might have a different perspective or useful information. In perhaps the ultimate example of engineering concerns not making their way upstream, Challenger astronauts were told that the cold temperature was not a problem, and Columbia astronauts were told that the foam strike was not a problem. NASA structure changed as roles and responsibilities were transferred to contractors, which increased the dependence on the private sector for safety functions and risk assessment while simultaneously reducing the in-house capability to spot safety issues. A critical turning point in both decisions hung on the discussion of contractor risk assessments. Although both Thiokol and Boeing engineering assessments were replete with uncertainties, NASA ultimately accepted each. Thiokol's initial recommendation against the launch of Challenger was at first criticized by Marshall as flawed and unacceptable. Thiokol was recommending an unheard-of delay on the eve of a launch, with schedule ramifications and NASAcontractor relationship repercussions. In the Thiokol off-line caucus, a senior vice president who seldom participated in these engineering discussions championed the Marshall engineering rationale for flight. When he told the managers present to "Take off your engineering hat and put on your management hat," they reversed the position their own engineers had taken.45 Marshall engineers then accepted this assessment, deferring to the expertise of the contractor. NASA was dependent on Thiokol for the risk assessment, but the decision process was affected by the contractor's dependence on NASA. Not willing to be responsible for a delay, and swayed by the strength of Marshall's argument, the contractor did not act in the best interests of safety. Boeing's Crater analysis was performed in the context of the Debris Assessment Team, which was a collaborative effort that included Johnson, United Space Alliance, and Boeing. In this case, the decision process was also affected
Report Volume I
by NASA's dependence on the contractor. Unfamiliar with Crater, NASA engineers and managers had to rely on Boeing for interpretation and analysis, and did not have the training necessary to evaluate the results. They accepted Boeing engineers' use of Crater to model a debris impact 400 times outside validated limits. NASA's safety system lacked the resources, independence, personnel, and authority to successfully apply alternate perspectives to developing problems. Overlapping roles and responsibilities across multiple safety offices also undermined the possibility of a reliable system of checks and balances. NASA's "Silent Safety System" did nothing to alter the decision-making that immediately preceded both accidents. No safety representatives were present during the Challenger teleconference – no one even thought to call them.46 In the case of Columbia, safety representatives were present at Mission Evaluation Room, Mission Management Team, and Debris Assessment Team meetings. However, rather than critically question or actively participate in the analysis, the safety representatives simply listened and concurred.
The echoes of Challenger in Columbia identified in this chapter have serious implications. These repeating patterns mean that flawed practices embedded in NASA's organizational system continued for 20 years and made substantial contributions to both accidents. The Columbia Accident Investigation Board noted the same problems as the Rogers Commission. An organization system failure calls for corrective measures that address all relevant levels of the organization, but the Board's investigation shows that for all its cutting-edge technologies, "diving-catch" rescues, and imaginative plans for the technology and the future of space exploration, NASA has shown very little understanding of the inner workings of its own organization. NASA managers believed that the agency had a strong safety culture, but the Board found that the agency had the same conflicting goals that it did before Challenger, when schedule concerns, production pressure, cost-cutting and a drive for ever-greater efficiency – all the signs of an "operational" enterprise – had eroded NASA's ability to assure mission safety. The belief in a safety culture has even less credibility in light of repeated cuts of safety personnel and budgets – also conditions that existed before Challenger. NASA managers stated confidently that everyone was encouraged to speak up about safety issues and that the agency was responsive to those concerns, but the Board found evidence to the contrary in the responses to the Debris Assessment Team's request for imagery, to the initiation of the imagery request from Kennedy Space Center, and to the "we were just ʻwhat-iffing'" e-mail concerns that did not reach the Mission Management Team. NASA's bureaucratic structure kept important information from reaching engineers and managers alike. The same NASA whose engineers showed initiative and a solid working knowledge of how to get things done fast had a managerial culture with an allegiance to bureaucracy and cost-efficiency that squelched
August 2003
the engineers' efforts. When it came to managers' own actions, however, a different set of rules prevailed. The Board found that Mission Management Team decision-making operated outside the rules even as it held its engineers to a stifling protocol. Management was not able to recognize that in unprecedented conditions, when lives are on the line, flexibility and democratic process should take priority over bureaucratic response.47 During the Columbia investigation, the Board consistently searched for causal principles that would explain both the technical and organizational system failures. These principles were needed to explain Columbia and its echoes of Challenger. They were also necessary to provide guidance for NASA. The Board's analysis of organizational causes in Chapters 5, 6, and 7 supports the following principles that should govern the changes in the agency's organizational system. The Board's specific recommendations, based on these principles, are presented in Part Three. Leaders create culture. It is their responsibility to change it. Top administrators must take responsibility for risk, failure, and safety by remaining alert to the effects their decisions have on the system. Leaders are responsible for establishing the conditions that lead to their subordinates' successes or failures. The past decisions of national leaders – the White House, Congress, and NASA Headquarters – set the Columbia accident in motion by creating resource and schedule strains that compromised the principles of a high-risk technology organization. The measure of NASA's success became how much costs were reduced and how efficiently the schedule was met. But the Space Shuttle is not now, nor has it ever been, an operational vehicle. We cannot explore space on a fixed-cost basis. Nevertheless, due to International Space Station needs and scientific experiments that require particular timing and orbits, the Space Shuttle Program seems likely to continue to be schedule-driven. National leadership needs to recognize that NASA must fly only when it is ready. As the White House, Congress, and NASA Headquarters plan the future of human space flight, the goals and the resources required to achieve them safely must be aligned. Changes in organizational structure should be made only with careful consideration of their effect on the system and their possible unintended consequences. Changes that make the organization more complex may create new ways that it can fail.48 When changes are put in place, the risk of error initially increases, as old ways of doing things compete with new. Institutional memory is lost as personnel and records are moved and replaced. Changing the structure of organizations is complicated by external political and budgetary constraints, the inability of leaders to conceive of the full ramifications of their actions, the vested interests of insiders, and the failure to learn from the past.49 Nonetheless, changes must be made. The Shuttle Program's structure is a source of problems, not just because of the way it impedes the flow of information, but because it has had effects on the culture that contradict safety goals. NASA's blind spot is it believes it has a strong safety culture. Program history shows that the loss of a truly indepenReport Volume I
dent, robust capability to protect the system's fundamental requirements and specifications inevitably compromised those requirements, and therefore increased risk. The Shuttle Program's structure created power distributions that need new structuring, rules, and management training to restore deference to technical experts, empower engineers to get resources they need, and allow safety concerns to be freely aired. Strategies must increase the clarity, strength, and presence of signals that challenge assumptions about risk. Twice in NASA history, the agency embarked on a slippery slope that resulted in catastrophe. Each decision, taken by itself, seemed correct, routine, and indeed, insignificant and unremarkable. Yet in retrospect, the cumulative effect was stunning. In both pre-accident periods, events unfolded over a long time and in small increments rather than in sudden and dramatic occurrences. NASA's challenge is to design systems that maximize the clarity of signals, amplify weak signals so they can be tracked, and account for missing signals. For both accidents there were moments when management definitions of risk might have been reversed were it not for the many missing signals – an absence of trend analysis, imagery data not obtained, concerns not voiced, information overlooked or dropped from briefings. A safety team must have equal and independent representation so that managers are not again lulled into complacency by shifting definitions of risk. It is obvious but worth acknowledging that people who are marginal and powerless in organizations may have useful information or opinions that they don't express. Even when these people are encouraged to speak, they find it intimidating to contradict a leader's strategy or a group consensus. Extra effort must be made to contribute all relevant information to discussions of risk. These strategies are important for all safety aspects, but especially necessary for ill-structured problems like O-rings and foam debris. Because ill-structured problems are less visible and therefore invite the normalization of deviance, they may be the most risky of all.
Challenger launches on the ill-fated STS-33/51-L mission on January 28, 1986. The Orbiter would be destroyed 73 seconds later.
August 2003
27 28 29
The citations that contain a reference to "CAIB document" with CAB or CTF followed by seven to eleven digits, such as CAB001-0010, refer to a document in the Columbia Accident Investigation Board database maintained by the Department of Justice and archived at the National Archives.
Report of the Presidential Commission, Vol. I, pp. 171-173. Report of the Presidential Commission, Vol. I, pp. 173-174. National Aeronautics and Space Administration, Aerospace Safety Advisory Panel, "National Aeronautics and Space Administration Annual Report: Covering Calendar Year 1984," (Washington: Government Printing Office, 1985). Harry McDonald, Report of the Shuttle Independent Assessment Team. Richard J. Feynman, "Personal Observations on Reliability of the Shuttle," Report of the Presidential Commission, Appendix F:1. Howard E. McCurdy, "The Decay of NASA's Technical Culture," Space Policy (November 1989), pp. 301-10; See also Howard E. McCurdy, Inside NASA (Baltimore: Johns Hopkins University Press, 1993). Diane Vaughan, "The Trickle-Down Effect: Policy Decisions, Risky Work, and the Challenger Tragedy," California Management Review, 39, 2, Winter 1997. Morton subsequently sold its propulsion division of Alcoa, and the company is now known as ATK Thiokol Propulsion. Report of the Presidential Commission, pp. 82-118. For discussions of how frames and cultural beliefs shape perceptions, see, e.g., Lee Clarke, "The Disqualification Heuristic: When Do Organizations Misperceive Risk?" in Social Problems and Public Policy, vol. 5, ed. R. Ted Youn and William F. Freudenberg, (Greenwich, CT: JAI, 1993); William Starbuck and Frances Milliken, "Executive Perceptual Filters – What They Notice and How They Make Sense," in The Executive Effect, Donald C. Hambrick, ed. (Greenwich, CT: JAI Press, 1988); Daniel Kahneman, Paul Slovic, and Amos Tversky, eds. Judgment Under Uncertainty: Heuristics and Biases (Cambridge: Cambridge University Press, 1982); Carol A. Heimer, "Social Structure, Psychology, and the Estimation of Risk." Annual Review of Sociology 14 (1988): 491-519; Stephen J. Pfohl, Predicting Dangerousness (Lexington, MA: Lexington Books, 1978). Report of the Presidential Commission, Vol. IV: 791; Vaughan, The Challenger Launch Decision, p. 178. Report of the Presidential Commission, Vol. I, pp. 91-92; Vol. IV, p. 612. Report of the Presidential Commission, Vol. I, pp. 164-177; Chapter 6, this Report. Report of the Presidential Commission, Vol. I, p. 90. Report of the Presidential Commission, Vol. IV, pp. 791. For details of teleconference and engineering analysis, see Roger M. Boisjoly, "Ethical Decisions: Morton Thiokol and the Space Shuttle Challenger Disaster," American Society of Mechanical Engineers, (Boston: 1987), pp. 1-13. Vaughan, The Challenger Launch Decision, pp. 358-361. Report of the Presidential Commission, Vol. I, pp. 88-89, 93. Edward Wong, "E-Mail Writer Says He was Hypothesizing, Not Predicting Disaster," New York Times, 11 March 2003, Sec. A-20, Col. 1 (excerpts from press conference, Col. 3). Report of the Presidential Commission, Vol. I, pp. 92-95. Report of the Presidential Commission, Vol. I, p. 152. Weick argues that in a risky situation, people need to learn how to "drop their tools:" learn to recognize when they are in unprecedented situations in which following the rules can be disastrous. See Karl E. Weick, "The Collapse of Sensemaking in Organizations: The Mann Gulch Disaster." Administrative Science Quarterly 38, 1993, pp. 628-652. Lee Clarke, Mission Improbable: Using Fantasy Documents to Tame Disaster, (Chicago: University of Chicago Press, 1999); Charles Perrow, Normal Accidents, op. cit.; Scott Sagan, The Limits of Safety, op. cit.; Diane Vaughan, "The Dark Side of Organizations," Annual Review of Sociology, Vol. 25, 1999, pp. 271-305. Typically, after a public failure, the responsible organization makes safety the priority. They sink resources into discovering what went wrong and lessons learned are on everyone's minds. A boost in resources goes to safety to build on those lessons in order to prevent another failure. But concentrating on rebuilding, repair, and safety takes energy and resources from other goals. As the crisis ebbs and normal functioning returns, institutional memory grows short. The tendency is then to backslide, as external pressures force a return to operating goals. William R. Freudenberg, "Nothing Recedes Like Success? Risk Analysis and the Organizational Amplification of Risks," Risk: Issues in Health and Safety 3, 1: 1992, pp. 1-35; Richard H. Hall, Organizations: Structures, Processes, and Outcomes, (Prentice-Hall. 1998), pp. 184-204; James G. March, Lee S. Sproull, and Michal Tamuz, "Learning from Samples of One or Fewer," Organization Science, 2, 1: February 1991, pp. 1-13.
Turner studied 85 different accidents and disasters, noting a common pattern: each had a long incubation period in which hazards and warning signs prior to the accident were either ignored or misinterpreted. He called these "failures of foresight." Barry Turner, Man-made Disasters, (London: Wykeham, 1978); Barry Turner and Nick Pidgeon, Man-made Disasters, 2nd ed. (Oxford: Butterworth Heinneman,1997). Changing personnel is a typical response after an organization has some kind of harmful outcome. It has great symbolic value. A change in personnel points to individuals as the cause and removing them gives the false impression that the problems have been solved, leaving unresolved organizational system problems. See Scott Sagan, The Limits of Safety. Princeton: Princeton University Press, 1993. Diane Vaughan, The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA (Chicago: University of Chicago Press. 1996). William H. Starbuck and Frances J. Milliken, "Challenger: Fine-tuning the Odds until Something Breaks." Journal of Management Studies 23 (1988), pp. 319-40. Report of the Presidential Commission on the Space Shuttle Challenger Accident, (Washington: Government Printing Office, 1986), Vol. II, Appendix H. Alex Roland, "The Shuttle: Triumph or Turkey?" Discover, November 1985: pp. 29-49. Report of the Presidential Commission, Vol. I, Ch. 6. Turner, Man-made Disasters. Vaughan, The Challenger Launch Decision, pp. 243-49, 253-57, 262-64, 350-52, 356-72. Turner, Man-made Disasters. U.S. Congress, House, Investigation of the Challenger Accident, (Washington: Government Printing Office, 1986), pp. 149. Report of the Presidential Commission, Vol. I, p. 148; Vol. IV, p. 1446. Vaughan, The Challenger Launch Decision, p. 235. Report of the Presidential Commission, Vol. I, pp. 1-3. Howard E. McCurdy, "The Decay of NASA's Technical Culture," Space Policy (November 1989), pp. 301-10. Report of the Presidential Commission, Vol. I, pp. 164-177. Report of the Presidential Commission, Vol. I, Ch. VII and VIII. Report of the Presidential Commission, Vol. I, pp. 140. For background on culture in general and engineering culture in particular, see Peter Whalley and Stephen R. Barley, "Technical Work in the Division of Labor: Stalking the Wily Anomaly," in Stephen R. Barley and Julian Orr (eds.) Between Craft and Science, (Ithaca: Cornell University Press, 1997) pp. 23-53; Gideon Kunda, Engineering Culture: Control and Commitment in a High-Tech Corporation, (Philadelphia: Temple University Press, 1992); Peter Meiksins and James M. Watson, "Professional Autonomy and Organizational Constraint: The Case of Engineers," Sociological Quarterly 30 (1989), pp. 561-85; Henry Petroski, To Engineer is Human: The Role of Failure in Successful Design (New York: St. Martin's, 1985); Edgar Schein. Organization Culture and Leadership, (San Francisco: Jossey-Bass, 1985); John Van Maanen and Stephen R. Barley, "Cultural Organization," in Peter J. Frost, Larry F. Moore, Meryl Ries Louise, Craig C. Lundberg, and Joanne Martin (eds.) Organization Culture, (Beverly Hills: Sage, 1985). Report of the Presidential Commission, Vol. I, pp. 82-111. Harry McDonald, Report of the Shuttle Independent Assessment Team. Report of the Presidential Commission, Vol. I, pp. 145-148. Vaughan, The Challenger Launch Decision, pp. 257-264. U. S. Congress, House, Investigation of the Challenger Accident, (Washington: Government Printing Office, 1986), pp. 70-71. Report of the Presidential Commission, Vol. I, Ch.VII. Mary Douglas, How Institutions Think (London: Routledge and Kegan Paul, 1987); Michael Burawoy, Manufacturing Consent (Chicago: University of Chicago Press, 1979).
30 31 32
34 35 36
6 7 8 9 10 11 12 13 14 15 16 17 18 19
37 38 39 40 41
42 43 44
45 46 47
20 21 22 23 24 25 26
Report Volume I
August 2003