View profile

A Bowl of Alphabet Soup: Liz Fong-Jones on Managing Complexity For Better DevOps

“Anyone trying to sell you DevOps in a box, it's not going to happen,” says researcher Nicole Forsgre
The Morning Mind-Meld
A Bowl of Alphabet Soup: Liz Fong-Jones on Managing Complexity For Better DevOps
By Jaime Woo and Emil Stolarsky • Issue #2 • View online
“Anyone trying to sell you DevOps in a box, it’s not going to happen,” says researcher Nicole Forsgren in her talk with Jez Humble, “Tools Won’t Fix Your Broken DevOps.” The Accelerate co-authors are hardly alone in suggesting that you can’t “buy” DevOps—a Google search brings up eleven hundred results for the phrase. We inherently understand that tools aren’t magical, but that doesn’t stop us from hoping that with the right tool we might get to where we want to go. 
Yet that magical thinking around tools can leave us swimming in an alphabet soup. “When you order the alphabet soup, you instead end up with chaos,” says Honeycomb’s Liz Fong-Jones, in her talk “Cultivating Production Excellence.” Instead of solving developer problems, Fong-Jones argues that tools applied haphazardly, without consideration for the ecosystem they exist within, needlessly increase complexity, which leads to unexpected consequences: Terraform and PagerDuty are useful tools, but can also accidentally shut down your entire AWS environment with a single command or exhaust engineers with endlessly noisy alerts, respectively.
Teams can struggle to hold on under this state of increasing complexity. As a way out, Fong-Jones advocates for the art of production excellence, which considers the people, processes, and culture of a company: “What we’ve failed to account for is that our systems are not just technical systems; instead, they are socio-technical systems.” Production excellence occurs when developers understand when their systems are too broken; debug the problem in a cross-functional, collaborative manner; and, then, reduce unnecessary complexity to close the feedback loop.
The last point, on complexity, merits some elaboration. In systems thinking, briefly: 
  • A simple problem—say, making a cup of coffee—has a knowable outcome, can be successfully accomplished approximately following a recipe, and requires a low level of expertise. 
  • A complicated problem—like, creating your financial statements—still has a knowable outcome, but requires strict adherence to process, and requires a certain level of expertise.
  • A complex problem—for example, presenting in front of a client—unlike simple and complicated problems, has no certain outcome, can’t be solved just by following a set of rules, and replicating a series of actions doesn’t guarantee the same result. 
This delineation is important because “the solutions to complicated problems don’t work as well with complex problems,” writes the MIT Sloan Management Review, and confusing the two is a source of failure. “Complex problems involve too many unknowns and too many interrelated factors to reduce to rules and processes.” That unknowability is frightening, which might explain the faith we place on tools to solve problems: that magical thinking arms us with a sense of control against situations that appear uncontrollable.
So what can be done? Rick Nason, an associate professor of finance at Dalhousie University, suggests in his book It’s Not Complicated first adopting a new mindset toward managing, rather than solving, complex problems, although the shift can cause discomfort. He writes:
“Manage, not solve” may be a humbling strategy to use but a lack of humility might be one of the reasons why managers default to complicated thinking. “Manage, not solve” is based on a strategy of thinking and making relatively spontaneous decisions under uncertainty.
Then, Nason suggests adopting the approach of “trying, learning, and adapting” to tackle complex problems:
In a complex environment it is truly rare that a grand plan or strategy will work as intended. Successful managers, however, are not discouraged by this. They learn from their missteps and use their learning to move forward with a new angle on the problem. They expect to learn as they go. 
Complicated thinkers tend to get too intellectually invested in an idea and refuse to let go, despite sometimes overwhelming evidence that the plan is not working. Complexity thinkers have the humility and flexibility not to get trapped into this low-probability strategy. With a try, learn, and adapt approach, organizations have to allow for mistakes to be made and for risks to be taken.
Bringing the ideas together, we understand why Fong-Jones is focused on the people within the system: people are arguably the most complex component of any systems they are part of—and, so, prioritizing tools over people can’t work. So how do you switch from a tools-first approach to one that is people-first?
In Mairead O'Connor’s “People Are More Complex Than Computers,” she outlines how consultancy Equal Experts approaches problem solving. In a complex system, where no one can wholly understand the system, it matters who gets to make decisions—is unnecessary complexity being added by having those further from the problem make the decision? What if those closest to the problem were the ones deciding? 
Enter the advice process, advocated by Dennis Bakke, cofounder of Fortune 500 company AES Corporation. O’Connor describes it as:
You state your intention: What are you trying to achieve? What do you want to do? You make an active effort to collect feedback from the people that you think are relevant and broaden that as much as you can and then you make your decision. The key point is that it’s your decision—it’s not that you’re making a business case and taking it for approval. You seek the right feedback so that you are making an informed decision and then you’re accountable for that decision going forward.
The advice process works by “creating an environment where the impact of failure is small and reversible,” she notes. You might notice parallels to the try, learn, and adapt approach recommended by Nason: unlike a top-down approach, the advice process allows for adaptation. But it can’t work without trust, although O’Connor asks: “If you can’t trust your people to make good decisions, why are you working with them?”
You can’t ignore the people in your system. Liz Fong-Jones believes you can’t accomplish production excellence without collaboration. And, while you can’t buy DevOps (or SRE) in a box, Nicole Forsgren shares that when you combine “technology plus lean management practices plus culture, then you can effect real change.” 
The learnings from approaching complex problems do not end when we take off our work hats. In fact, they are even more important when we are done our nine-to-fives. As Georgia O’Keeffe wisely noted: “Whether you succeed or not is irrelevant—there is no such thing. Making your unknown known is the important thing—and keeping the unknown always beyond you.” 
Too often we see the complexity around us and attempt to solve it as if it were complicated. But in treating life as complicated, we can end up missing out on learning about what’s truly interesting in life.

Thanks to Andrew Louis for a gut-check.

This is an Incident Labs project, with new issues every two weeks. We’re interested in figuring out the best practices for incident management for software companies. We also produce the Post-Incident Review, a zine focused on outages. If you use PagerDuty and Slack, our software project Ovvy simplifies scheduling and overrides, and is currently in private beta and free to use.
Did you enjoy this issue?
Jaime Woo and Emil Stolarsky

Life is busy, and there are always those conference talks or long-form articles you should read but never seem to find the time.


The Morning Mind-Meld is a chance to build context between the conversations happening around DevOps and SRE, and to hopefully create some inspiration, even—or, especially!—during a hectic week.

If you don't want these updates anymore, please unsubscribe here.
If you were forwarded this newsletter and you like it, you can subscribe here.
Powered by Revue