View profile

How Apollo 12 Was a Test in Trusting Your People

We're glued to the Demo-2 livestream today—and treating today like a snow day for space nerds. While
The Morning Mind-Meld
How Apollo 12 Was a Test in Trusting Your People
By Jaime Woo and Emil Stolarsky • Issue #8 • View online
We’re glued to the Demo-2 livestream today—and treating today like a snow day for space nerds.
While NASA makes history today, we want to throw back to a jaw-dropping story from its archives. From our story in the new issue of Post-Incident Review, published yesterday:
Something’s going wrong.
The Incident Commander is sounding the alarm. They need eyes. Is this the worst case scenario we’ve been dreading? All the monitoring data is gone, and the graphs are showing an unintelligible pattern.
You glance at the pattern, and for some reason it clicks in your brain. A year ago, you’d been observing a test run and noticed the same seemingly indecipherable pattern. Back then, you figured why not play around with it to discover how to recreate it.
You write to the IC an obscure suggestion. It’s so weird that for a second the IC replies with, “What?!” But, you repeat it, sure of what you are seeing. Time is ticking, and there are no other viable options, so the IC backs your suggestion.
Immediately, everything returns to normal. The mission to the moon continues as planned.
This isn’t a story from a tech company in 2020. This is Apollo 12. It was in 1969 at NASA, when lightning struck twice.
We look at the role of curiosity in Post-Incident Review, but there’s another lesson: trusting your people. A crucial part of incident response is listening to the SMEs you’ve assembled, and having faith in their experience.
On the rising Saturn V, 36.5 seconds into lift off and then again at 52 seconds, hitting the spacecraft and causing pandemonium. The lightning had caused a power surge and inadvertently disconnected the fuel cells, leading to a voltage drop.
NASA engineer John Aaron gave the obscure suggestion to “try SCE to Aux,” after recognizing the telemetry pattern from an anomaly he’d witnessed a year before during a test at the Kennedy Space Center.
When Aaron gave an unexpected solution, it would have been understandable for Flight Director Gerry Griffin to not take up the suggestion. From a NASA recollection of the situation:
GRIFFIN (FLIGHT) - How’s it looking, EECOM? (…) EECOM, what do you see?
AARON (EECOM) - Flight, EECOM. Try SCE to AUX.
GRIFFIN (FLIGHT) - Say again? SCE to AUX?
AARON (EECOM) - AUX.
GRIFFIN (FLIGHT) - AUX.
AARON (EECOM) - Auxiliary, FLIGHT.
GRIFFIN (FLIGHT) - SCE to AUX. CAPCOM. What panel? EECOM?
Although sounding confounded by the suggestion himself, Griffin recognized the urgency of the situation and passed the action to CapCom Gerald Carr.
Similarly, CapCom Gerald Carr could have ignored the escalation. However, he trusted his people.
Astronaut Carr activated his microphone to open the thankfully still functioning radio link to the spacecraft, and spoke as clearly and firmly as possible.
000:01:36 Carr: Apollo 12, Houston. Try SCE to Auxiliary. Over.
000:01:39 Conrad: Try SCE to Auxiliary.
Shifting the signal conditioning electronics, or SCE, system to its auxiliary setting allowed it to operate in low-voltage conditions, restoring the telemetry.
The lightning strike was a black swan event: “when lightning strikes Apollo 12, I mean, we had never simulated that before,” noted Aaron in an interview. The power of trusting the incident command structure helped NASA figuratively and literally weather the storm.

This is an Incident Labs project. We’re interested in figuring out the best practices for incident management for software companies.
We’re building Ovvy Insights, which combines your team’s expertise with PagerDuty data to help you reduce toil and regain engineering capacity, now in private beta.
We also produce the Post-Incident Review, a zine focused on outages.
Did you enjoy this issue?
Jaime Woo and Emil Stolarsky

The Morning Mind-Meld is a chance to build context between the conversations happening around DevOps and SRE, and to hopefully create some inspiration, even—or, especially!—during a hectic week.

If you don't want these updates anymore, please unsubscribe here.
If you were forwarded this newsletter and you like it, you can subscribe here.
Powered by Revue