We’re glued to the Demo-2 livestream today—and treating today like a snow day for space nerds.
While NASA makes history today, we want to throw back to a jaw-dropping story from its archives. From our story in the new issue of Post-Incident Review
, published yesterday:
Something’s going wrong.
The Incident Commander is sounding the alarm. They need eyes. Is this the worst case scenario we’ve been dreading? All the monitoring data is gone, and the graphs are showing an unintelligible pattern.
You glance at the pattern, and for some reason it clicks in your brain. A year ago, you’d been observing a test run and noticed the same seemingly indecipherable pattern. Back then, you figured why not play around with it to discover how to recreate it.
You write to the IC an obscure suggestion. It’s so weird that for a second the IC replies with, “What?!” But, you repeat it, sure of what you are seeing. Time is ticking, and there are no other viable options, so the IC backs your suggestion.
Immediately, everything returns to normal. The mission to the moon continues as planned.
This isn’t a story from a tech company in 2020. This is Apollo 12. It was in 1969 at NASA, when lightning struck twice.
We look at the role of curiosity in Post-Incident Review, but there’s another lesson: trusting your people. A crucial part of incident response is listening to the SMEs you’ve assembled, and having faith in their experience.
On the rising Saturn V, 36.5 seconds into lift off and then again at 52 seconds, hitting the spacecraft and causing pandemonium. The lightning had caused a power surge and inadvertently disconnected the fuel cells, leading to a voltage drop.
NASA engineer John Aaron gave the obscure suggestion to “try SCE to Aux,” after recognizing the telemetry pattern from an anomaly he’d witnessed a year before during a test at the Kennedy Space Center.
When Aaron gave an unexpected solution, it would have been understandable for Flight Director Gerry Griffin to not take up the suggestion. From a NASA recollection of the situation
GRIFFIN (FLIGHT) - How’s it looking, EECOM? (…) EECOM, what do you see?
AARON (EECOM) - Flight, EECOM. Try SCE to AUX.
GRIFFIN (FLIGHT) - Say again? SCE to AUX?
AARON (EECOM) - AUX.
GRIFFIN (FLIGHT) - AUX.
AARON (EECOM) - Auxiliary, FLIGHT.
GRIFFIN (FLIGHT) - SCE to AUX. CAPCOM. What panel? EECOM?
Although sounding confounded by the suggestion himself, Griffin recognized the urgency of the situation and passed the action to CapCom Gerald Carr.
Similarly, CapCom Gerald Carr could have ignored the escalation. However, he trusted his people.
Astronaut Carr activated his microphone to open the thankfully still functioning radio link to the spacecraft, and spoke as clearly and firmly as possible.
000:01:36 Carr: Apollo 12, Houston. Try SCE to Auxiliary. Over.
000:01:39 Conrad: Try SCE to Auxiliary.
Shifting the signal conditioning electronics, or SCE, system to its auxiliary setting allowed it to operate in low-voltage conditions, restoring the telemetry.
The lightning strike was a black swan event: “when lightning strikes Apollo 12, I mean, we had never simulated that before,” noted Aaron in an interview
. The power of trusting the incident command structure helped NASA figuratively and literally weather the storm.