Reducing Complexity With System Automation With Austin Parker

Page it to the Limit

The importance of designing simpler systems

Austin uses the example of a platform team managing a large portfolio of applications that all need to share a common layer, like SSO. Sounds simple, but complexity only gets shifted around depending on your approach.

“The role of this platform team is to impose some simplicity onto this complex design that has been foisted upon them. We started coming up with some very complicated solutions to how to do different tasks for different systems. Until we stepped back and instead focused on what these systems had in common. Once we reframed into looking at the similarities instead of the differences, we were able to make a lot of progress very quickly.”

Simple and easy are not the same thing

Simplicity is being able to reason about what is happening, even if what is happening is a very difficult and involved process. Just because something is easy, that doesn’t mean it’s simple. Just because something is simple, that doesn’t mean it’s not difficult to do. Simplicity can be very hard to implement when you see it that way.

“Breathing sounds very simple, but it’s not very easy if you think about what is happening in your body to make you breathe. It’s an incredibly complex series of physical biological systems: I’m not a biologist or doctor, so I don’t even understand them all.”

Are abstractions actually making our lives better?

The microservices vs. monolith debate keeps popping up. Austin argues that microservices were never about managing technical complexity, they were about managing organizational complexity. The abstractions for microservice platforms are managing our organizational problems but at a high technical cost. It’s how we try overcoming that cost that might be making lives more difficult.

“On the technical side, that now means you have to be able to have a deeper understanding of your software: you have to be able to introspect the behavior of your application at every single point that it is used … At some point, we’ve covered it with thousands of probes … so much so that no one can even step back to see the application anymore, because we’ve lost sight of what it actually is.”

Finding the signal in the noise

People are at the heart of these problems because they’re the ones who need to constantly re-evaluate system state. Austin makes the point that people usually fail to do that hard part of system design because it’s seen as a Day 2 problem.

“We make decisions about system normalcy at the tail end of design, using what we’ve known up to that point. Most people aren’t thinking about how they write logging statements to understand application performance at that point.”

It always comes back to Sydney Dekker

We talk about “The Field Guide to Understanding Human Error” for the first time on this show! How did that take so long??? When it comes to understanding systems, it’s better to be clear than to be concise.

“The designers of an airplane cockpit changed the airspeed indicator from a round dial to a linear tape, figuring that it would be easier to quickly read airspeed if your target was in the middle of the tape… What they found was that it actually made it harder to understand and it took cycles away from pilots and added more response time whenever things went wrong.”

Simplicity is hard

Experience is thinking about how humans interpret signals. It’s important to know when to be clever and complex and when it’s better to optimize for reasonability. Austin gives some practical tips to simplify choices when it comes to designing simpler systems.

“While simplicity and easy aren’t necessarily connected, simple and good kind of are.”

Recurring Questions

Austin wishes he would have known how to write good logs sooner. Very rarely do people stop to consider how to make metrics useful to other people. It’s less about how than why.

Austin is also glad we didn’t ask him about running production software on Windows based systems. The problem wasn’t Windows, it was dealing with heterogeneity.

References

• PagerDuty Home Page
• The Field Guide to Understanding Human Error - Sydney Dekker
• The Open Telemetry Project
• Lightstep Sandbox
• Your System is Deeper Than You Think - Austin Parker
• Episode transcribed by Rev