Challenger 1986: a risk nobody could put a number on
A risk without a number is not a risk of zero.
On the evening of 27 January 1986 the engineers of the rocket manufacturer recommended postponing the launch: there was no experience with the cold expected the next morning. Four managers decided otherwise. The decision map shows what had to be weighed and why a missing number is no reason to launch.
A known problem and a cold morning
The shuttle's solid rocket boosters are built from segments. In the joints between them, two rubber rings hold back the hot gases. From the second flight in 1981 onward, technicians kept finding rings on recovered boosters that were scorched or partly burned away [2]. In December 1982 NASA put the joint in its highest criticality class: if it fails, mission, vehicle and crew are lost [2].
The sources use degrees Fahrenheit: 32 °F is the freezing point, 53 °F is 12 °C. The coldest launch up to then was the one in January 1985, with 53 °F at the rings. It showed the worst damage: hot gas had passed the first ring at two joints, and for the first time heat had reached the second ring [2]. In a bench test a ring lost contact with the sealing surface for 2.4 seconds at 75 °F (24 °C). At 50 °F (10 °C) it did not regain contact within ten minutes [2].
On 31 July 1985 the engineer Roger Boisjoly warned his company Morton Thiokol, the maker of the boosters, in writing: the result could be a catastrophe with loss of human life [2]. Frost was forecast in Florida for the morning of 28 January 1986. From the forecast Thiokol calculated 29 °F (−2 °C) for the rings. That is 24 °F, or 13 degrees Celsius, less than on the coldest flight [5]. The decision that evening was made with this number, so the map uses it as well. By then the launch had been postponed three times and scrubbed once [7].
The night before the launch
On the evening of 27 January, 34 specialists from Thiokol and NASA joined a call from three sites. Thiokol's recommendation was on a handwritten chart: the rings must be at least 53 °F at launch [1]. NASA managers objected sharply. Thiokol asked for five minutes for an internal caucus. It lasted half an hour [1].
In that caucus the most senior manager asked the head of engineering to take off his engineering hat and put on his management hat. Four managers decided; the engineers were no longer asked [1]. At about 11:45 p.m. Thiokol signed the recommendation to launch [1].
Challenger lifted off on 28 January at 11:38 a.m. By then the air had warmed to 36 °F (2 °C) [6]. After the night of frost the boosters were colder than the air: for the spot where the joint failed, the Commission estimated 28 °F, give or take 5 °F [13]. Thiokol's calculation of the evening before was therefore accurate. A joint on the right booster failed. After 73 seconds the shuttle broke apart. All seven people on board died [6].
What was said that evening
The investigating commission asked for the sentences of that evening under oath. That the risk could not be put into a number made the evening hard, and that is normal for such a question. But the gap was not filled with arguments. Of the seven sentences on record, none names an impact of launching. Four apply pressure or change who decides. Three concern the plausibility of the risk, and they are the only thing the evening needed to talk about. Nobody said what launching that morning was worth: according to the testimony of Thiokol manager Jerald Mason, the schedule was not discussed in the internal caucus [5]. According to Boisjoly, none of the engineers spoke in favor of launching [1].
“My God, Thiokol, when do you want me to launch, next April?”
Lawrence Mulloy, NASA, as the sentence was reported. Before the Commission he gave a milder version: with that limit, launches might not be possible until April. [1]
The sentence argues against a general temperature limit. The decision was about one launch, and a warmer day would have been enough for it. The sentence does not name what waiting costs.
“I stated I was somewhat appalled”
George Hardy, NASA, about Thiokol's recommendation [1]
A reaction that carries rank. It says nothing about value, size or plausibility, and it makes every further objection more costly.
“You guys are generating new Launch Commit Criteria”
Lawrence Mulloy, NASA, as testified by Allan McDonald [1]
Whether a rule for temperature exists does not change whether the joint seals at 29 °F. The 24 flights before were all at 53 °F or warmer.
“take off his engineering hat and put on his management hat”
Jerald Mason, Thiokol, to head of engineering Robert Lund (as testified by Roger Boisjoly) [1]
The sentence changes who decides. It does not change the evidence.
“Temperature data not conclusive on predicting primary O-ring blow-by”
Thiokol's written launch recommendation [1]
Only the flights with damage were compared that evening. With all 24 flights the pattern is there: four of four below 65 °F, three of twenty above. And not conclusive means unknown, not safe.
“If the primary seal does not seat, the secondary seal will seat”
Thiokol's written launch recommendation [1]
Since December 1982 the joint had been in the criticality class without a reliable backup. On the coldest flight, heat had already reached the second ring.
“Demonstrated sealing threshold is 3 times greater than 0.038″ erosion experienced on SRM-15”
Thiokol's written launch recommendation [1]
The margin applies to erosion of a ring that is seated. The open question was whether a cold, hard ring seats in time. The bench test at 50 °F said it does not.
The question was clear, the method was missing
The question of that evening was clear: whether Challenger could launch despite the cold. It comes down to a single quantity, the probability that a joint fails to seal at 29 °F. That quantity could not be measured. Roger Boisjoly was asked to quantify his concern and said he could not. He only knew that they were moving away from the safe range of the existing data [1].
Not quantifiable does not mean unknown. Four things were established. It was the coldest launch, 24 °F below all experience. There had been thermal distress on the rings several times, the worst on the coldest flight up to then [2]. Cold rubber hardens and follows the joint more slowly, as the bench test had shown [2]. And NASA itself kept the joint in the class in which a failure costs vehicle and crew [2]. Each of these points moves the probability away from zero and above that of an ordinary flight. None of them yields a number; together they yield a direction.
What was missing was a procedure that can work with knowledge of this kind. The analysis of all flights by temperature, which shows the pattern, was not carried out by anyone before the accident. The Commission found that NASA and Thiokol were therefore unprepared to evaluate the risk of launching in conditions more extreme than any before [2]. Nor was there a shared format that evening in which the value, size and plausibility of an objection stand side by side. In such a format the question would have been whether the risk lies above what is acceptable for a flight. That was not discussed.
Where the method is missing, what decides is what people do without one. Whoever cannot prove something loses: the engineers were expected to show that launching was unsafe [1]. Rank settles it: in the end four managers voted [1]. And habit reassures: 24 flights had gone well although the rings kept showing damage. The physicist Richard Feynman, a member of the Commission, called this a kind of Russian roulette in which every flight survived lowers the standard [2]. These are not the failings of individuals. This is how people decide under uncertainty when nothing forces them into a different format.
A standard for weighing asks of every sentence which row of the map it changes. “Next April”, being appalled and the management hat change none and drop out. What remains is a single question: whether the risk at 29 °F lies above what is acceptable for a flight. That question is hard because the situation was hard. The evening could have concentrated on it, and it had an answer even without an exact number.
The Commission did not see the risk only on that evening. In its judgment the damage history presented to NASA Headquarters in August 1985 already required corrective action before the next flight [2]. The cold of 28 January came on top of a risk that was already too large.
The decision as a map
The map puts two futures side by side. In one, Challenger launches on the morning of 28 January with a calculated 29 °F at the rings. In the other, NASA waits until the rings are at least 53 °F: the coldest value the system had ever flown with.
The first state shows how the evening did the sums. Robert Lund later testified that Thiokol had to prove it was not ready to launch [1]. A risk that could not be proven therefore counted as zero. The second state scores the same row with the evidence of that evening.
In this row plausibility does not measure whether the joint fails. Nobody would launch at a probability of 1 in 10; the standard for a safe launch is far stricter. It measures how well supported it is that the launch lies above the risk NASA itself accepts for a flight. The risk itself is in the size of the row.
F1: launch in the morning, with a calculated 29 °F at the O-rings. F0: wait until the rings are at least 53 °F.
How the evening did the sums
Nobody voiced the reasons for launching. The risk counted as unproven and dropped out of the calculation.
Size a few days, two weeks in the unfavorable case
How it is scored
- V 6
- A reliable schedule was a declared goal: 15 flights were planned for 1986, 24 a year for 1990. That carries weight, but less than lives and the vehicle.
- I 1
- The launch slips by a few days, by two weeks in the unfavorable case, and the following flights move back. Nobody is harmed and no hardware is lost. Against the stand-down after a loss (623 days after the Apollo fire in 1967), two weeks are one fortieth, and the stand-down is only part of the loss.
- P 9
- Certain. Launching in the morning saves the days.
Size a one-year wait for two missions if May is missed
How it is scored
- V 6
- The same class of value as the schedule: missions that fly as planned.
- I 2
- If the two probes miss their window in May, two of about 15 flights that year wait a year for the next one. The probes are kept and fly later. That is more than a few days of delay and still a small fraction of the whole program.
- P 1
- Low. Four months and several flights lie between a few days of delay in January and a missed window in May. Nobody showed that the delay would carry that far.
Size the relationship with the client in a contract year
How it is scored
- V 6
- A real good for the manufacturer: in January 1986 NASA had announced its intention to establish a second supplier for the rocket motors.
- I 2
- The client is displeased, and Thiokol enters the year in which NASA is looking for a second supplier with a disagreement on record. No contract is at stake that evening. Measured against the loss of an orbiter, which hits the manufacturer as well, that is small.
- P 3
- Weakly supported. Nothing shows that a technically reasoned recommendation costs the contract. NASA manager George Hardy said that evening that he would not recommend launching over Thiokol's objections.
Size seven lives, one of four orbiters with its payload, the fleet grounded, at a risk of loss of about 1 in 10
How it is scored
- V 10
- Human lives are at the top of the scale. Mission success and the continuation of the program come on top of that.
- I 10
- If the joint fails, the seven people on board die. The orbiter is lost, one of four, and with it the payload, including a NASA communications satellite. The remaining orbiters stay on the ground until the cause is found and fixed: after the Apollo fire in 1967 that took 623 days. Added to that is the damage to trust in a program that had planned 15 flights for 1986. NASA itself had kept the joint since December 1982 in the class in which a failure costs mission, vehicle and crew. This full loss is a hundred times as large as two weeks of delay. It enters the map with the risk the cold adds: the calculation uses 1 in 10, that is one tenth of the full loss. That is an assumption; with 1 in 25 waiting is still ahead. The row remains the largest difference on the table, which is why it stands at 10.
- P 0
- This is how the evening did the sums: the engineers could not prove that the joint would fail at 29 °F. So the row dropped out of the calculation as if the risk had been ruled out.
With the evidence of that evening
The same four rows. The risk now carries the value that the data of that evening support at the least.
Size a few days, two weeks in the unfavorable case
How it is scored
- V 6
- A reliable schedule was a declared goal: 15 flights were planned for 1986, 24 a year for 1990. That carries weight, but less than lives and the vehicle.
- I 1
- The launch slips by a few days, by two weeks in the unfavorable case, and the following flights move back. Nobody is harmed and no hardware is lost. Against the stand-down after a loss (623 days after the Apollo fire in 1967), two weeks are one fortieth, and the stand-down is only part of the loss.
- P 9
- Certain. Launching in the morning saves the days.
Size a one-year wait for two missions if May is missed
How it is scored
- V 6
- The same class of value as the schedule: missions that fly as planned.
- I 2
- If the two probes miss their window in May, two of about 15 flights that year wait a year for the next one. The probes are kept and fly later. That is more than a few days of delay and still a small fraction of the whole program.
- P 1
- Low. Four months and several flights lie between a few days of delay in January and a missed window in May. Nobody showed that the delay would carry that far.
Size the relationship with the client in a contract year
How it is scored
- V 6
- A real good for the manufacturer: in January 1986 NASA had announced its intention to establish a second supplier for the rocket motors.
- I 2
- The client is displeased, and Thiokol enters the year in which NASA is looking for a second supplier with a disagreement on record. No contract is at stake that evening. Measured against the loss of an orbiter, which hits the manufacturer as well, that is small.
- P 3
- Weakly supported. Nothing shows that a technically reasoned recommendation costs the contract. NASA manager George Hardy said that evening that he would not recommend launching over Thiokol's objections.
Size seven lives, one of four orbiters with its payload, the fleet grounded, at a risk of loss of about 1 in 10
How it is scored
- V 10
- Human lives are at the top of the scale. Mission success and the continuation of the program come on top of that.
- I 10
- If the joint fails, the seven people on board die. The orbiter is lost, one of four, and with it the payload, including a NASA communications satellite. The remaining orbiters stay on the ground until the cause is found and fixed: after the Apollo fire in 1967 that took 623 days. Added to that is the damage to trust in a program that had planned 15 flights for 1986. NASA itself had kept the joint since December 1982 in the class in which a failure costs mission, vehicle and crew. This full loss is a hundred times as large as two weeks of delay. It enters the map with the risk the cold adds: the calculation uses 1 in 10, that is one tenth of the full loss. That is an assumption; with 1 in 25 waiting is still ahead. The row remains the largest difference on the table, which is why it stands at 10.
- P 5
- The question for this row is whether the launch lies above the risk NASA accepts for a flight. Its engineers put the risk of loss on an ordinary flight at about 1 in 100, management at 1 in 100,000. There is a good case that the cold leads beyond that: thermal distress on all four flights below 65 °F, the worst on the coldest; a bench test in which the ring found no contact for ten minutes at 50 °F; a launch 24 °F below all experience. There was no measurement for 29 °F, and part of the data could be read differently. Hence 5: quite possible, not established. Given what is at stake in this row, that is enough.
Decision map: two states of the same decision. The numbers in the four rows are the same, except for the plausibility of the risk.
How to read the map
Each row names exactly one impact: one difference between the two futures. V is the weight of the good at stake (0 to 10), I is the size of the difference (10 is the largest on the table, all others are in proportion to it), P is the plausibility given the evidence on the day of the decision (0 to 10). The numbers are assumptions of this reconstruction, not measurements. Each one carries its reason so that it can be challenged.
The scoring uses what was known on the day of the decision. What became known later is in the last section and enters no number on the map.
The missing number was no reason to launch
Flight with thermal distress on an O-ring Flight without damage
Data as a table
| Flight | Joint | Thermal distress |
|---|---|---|
| STS-1 | 66 °F / 19 °C | no |
| STS-2 | 70 °F / 21 °C | yes |
| STS-3 | 80 °F / 27 °C | no |
| STS-4 | 80 °F / 27 °C | no |
| STS-5 | 68 °F / 20 °C | no |
| STS-6 | 67 °F / 19 °C | no |
| STS-7 | 72 °F / 22 °C | no |
| STS-8 | 73 °F / 23 °C | no |
| STS-9 | 70 °F / 21 °C | no |
| 41-B | 57 °F / 14 °C | yes |
| 41-C | 63 °F / 17 °C | yes |
| 41-D | 70 °F / 21 °C | yes |
| 41-G | 67 °F / 19 °C | no |
| 51-A | 67 °F / 19 °C | no |
| 51-C | 53 °F / 12 °C | yes |
| 51-D | 67 °F / 19 °C | no |
| 51-B | 75 °F / 24 °C | no |
| 51-G | 70 °F / 21 °C | no |
| 51-F | 81 °F / 27 °C | no |
| 51-I | 76 °F / 24 °C | no |
| 51-J | 79 °F / 26 °C | no |
| 61-A | 75 °F / 24 °C | yes |
| 61-B | 76 °F / 24 °C | no |
| 61-C | 58 °F / 14 °C | yes |
Fig. 1: All four flights below 65 °F had thermal distress on the O-rings; above 65 °F it was three of twenty. The launch was 24 °F below the coldest one so far. Data: Rogers Commission 1986, Volume 1, Chapter VI.
Fig. 2: Launching only leads if the cold raises the risk of loss by less than 1 in 50. The axis is logarithmic: each step is a factor of ten.
The data in the first figure were on hand on the evening of 27 January. They did not exist as a picture: nobody plotted them in this form before the accident. Only the flights with damage were compared, and in that selection temperature looks unremarkable: there was damage at 53 °F and at 75 °F [2]. The pattern only appears once the flights without damage are included.
Nobody could say how likely a failure at 29 °F was. The map does not need that number. It needs the answer to a simpler question: on which side of the tipping point the truth lies. The tipping point is at about 1 in 50. The added risk from the cold would have had to be that small for the days gained to outweigh it.
For comparison: NASA engineers put the risk of losing vehicle and crew on a normal flight at about 1 in 100, management at 1 in 100,000 [4]. So the cold could at most triple the risk of a normal flight. Against that stood four of four cold flights with damage, a bench test, and 24 °F of distance from all experience.
A statistical recalculation from 1989, using the flight data from before the accident, arrives at a probability of at least 13 in 100 [8]. It works with 31 °F and is thus close to the 29 °F of that evening. That number did not exist that evening. It does not enter the map and only shows where the truth lay: far above the tipping point.
What would have changed the decision
The evening apparently lacked a procedure for weighing a risk without an exact number instead of setting it to zero. With such a procedure, three things change.
First, the burden of proof. Whoever wants to launch at the highest value and the largest impact has to show that the risk is small enough. That evening it was the other way round: Roger Boisjoly testified that the engineers were expected to prove beyond any doubt that launching was not safe [1]. On the map no side wins because the other cannot prove something. Every row needs its own reason.
Second, the empty side. No argument for launching is on record from that evening that names an impact of this one launch. Written down, it would have been days in the schedule, set against seven lives and one of four orbiters. Nobody voiced that comparison, so nobody had to object to it.
Third, a language across ranks. Being appalled and management hats are not entries on a map. An engineer who gives reasons for the plausibility and a manager who sets it lower are talking about the same row. The room sees whose reason holds.
Waiting had a price: more days of delay after four failed dates, in a year with 15 planned flights [3]. The map does not hide it. It shows how small it is next to what was on the other side.
What happened afterwards
The commission chaired by William Rogers found that Thiokol's management had reversed its position at the urging of NASA and against the views of its own engineers, in order to accommodate a major customer [1]. It found no evidence of pressure from outside NASA [3].
The shuttles stayed on the ground for 975 days, until 29 September 1988 [10]. After the Apollo 1 fire the pause had lasted 623 days [11][12]. The physicist Richard Feynman wrote in his appendix to the report that a successful technology must put reality above public relations, because nature cannot be fooled [4].
A risk without a number is not a risk of zero.
A risk that cannot be quantified counts as zero, and those who warn have to provide the proof: the investigation reports on major accidents describe this pattern again and again.
- 2003
Columbia
At launch a piece of insulating foam struck the wing. Engineers asked three times for images of the orbiter in space; program management cancelled the request. The investigation board writes that NASA inverted the burden of proof [14].
- 2010
Deepwater Horizon
A pressure test on the well showed an anomalous reading, and the crew explained it away. According to the national commission's report, important decisions were made without any formal risk analysis. Eleven people died [15].
- 2011
Fukushima Daiichi
According to the Japanese parliament's report, the operator and the regulator knew that flooding would cause a total loss of power. The operator called an assessment based on probabilities technically too uncertain and used that to postpone countermeasures [16].
Sources
- Rogers Commission (1986), Volume 1, Chapter V: The Contributing Cause of the Accident
- Rogers Commission (1986), Volume 1, Chapter VI: An Accident Rooted in History
- Rogers Commission (1986), Volume 1, Chapter VIII: Pressures on the System
- Richard Feynman: Personal Observations on the Reliability of the Shuttle. Rogers Commission, Volume 2, Appendix F
- Rogers Commission (1986), Volume 4: hearing of 25 February 1986
- Rogers Commission (1986), Volume 1, Chapter III: The Accident
- Rogers Commission (1986), Volume 1, Chapter II: Events Leading Up to the Challenger Mission
- Dalal, Fowlkes, Hoadley (1989): Risk Analysis of the Space Shuttle: Pre-Challenger Prediction of Failure. Journal of the American Statistical Association 84 (408), 945–957
- US Government Accountability Office (1986): NASA's plan to establish a second source for the Shuttle solid rocket motor, NSIAD-86-194
- NASA: mission STS-26, launched 29 September 1988
- NASA: Apollo 1, fire on 27 January 1967
- NASA: Apollo 7, launched 11 October 1968
- Rogers Commission (1986), Volume 1, Chapter IV: The Cause of the Accident
- Columbia Accident Investigation Board: Report, Volume 1 (August 2003)
- National Commission on the BP Deepwater Horizon Oil Spill and Offshore Drilling: Deep Water. Report to the President (January 2011)
- Fukushima Nuclear Accident Independent Investigation Commission of the National Diet of Japan (NAIIC): Executive Summary (2012)
Decisions over time Kodak, 1981 to 2003
A good decision has an expiry date.
With what was known in 1981, staying with film was right. The reasoning had an expiry date. When it was reached, the decision stayed in force.
Read the case →
Transparency of a decision Netflix and Qwikster, 2011
What is not open inside cannot be explained outside.
The customers were not told why. Inside the company nobody voiced the doubts. A decision that is not transparent inside cannot be explained outside.
Read the case →
Deciding by one's own values Volkswagen, 2006 to 2015
Values count only when they are part of the decision.
The objections were on the table, the values were on paper. Under pressure of time and cost, neither counted. As a row in a decision, a value is harder to pass over in silence or to talk down.
Read the case → The method behind the mapsThe full derivation of V × I × P is on the page Reason Based Decision Making for Business.
To the derivation →