DevRelin’ 😎 with Austin Parker & Ted Young of Lightstep

On-Call Me Maybe

Content provided by Austin Parker, Ana Margarita Medina, and Adriana Villela. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Austin Parker, Ana Margarita Medina, and Adriana Villela or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.

1+ y ago 41:02

MP3•Episode home

Archived series ("Inactive feed" status)

When? This feed was archived on November 30, 2023 00:38 (5M ago). Last successful fetch was on September 18, 2023 15:43 (7M ago)

Why? Inactive feed status. Our servers were unable to retrieve a valid podcast feed for a sustained period.

What now? You might be able to find a more up-to-date version using the search function. This series will no longer be checked for updates. If you believe this to be in error, please check if the publisher's feed link below is valid and contact support to request the feed be restored or if you have any other concerns about this.

About the guests:

Austin Parker is the Head of Developer Relations at Lightstep and has been creating problems with computers for most of his life. He’s a maintainer of the OpenTelemetry project, the host of several podcasts, the organizer of Deserted Island DevOps, an infrequent Twitch streamer, a conference speaker, and more. When he’s not working, you can find him posting on Twitter, cooking, and parenting. His most recent book is Distributed Tracing in Practice, published by O’Reilly Media.

Ted Young is the Director of Open Source Development at Lightstep and one of the core maintainers of the OpenTracing project. Ted has spent the last 15 years building distributed systems in a variety of environments: computer animation, national elections, and elastic computing platforms.

Find our guest on:

Find us on:

Show Links:

Additional Links:

Transcript:

ADRIANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Adriana Villela, with my awesome co-host...

ANA: Ana Margarita Medina.

ADRIANA: And today, we're talking to the whole DevRel team at Lightstep, woo-hoo.

ANA: Woo!

ADRIANA: And it's taken us how long to get us all together on this? I think this is our third attempt.

ANA: Yeah, it's actually third attempt. It's taken five months in the making.

ADRIANA: [laughs]

ANA: But it's here. We're finally getting this episode out the door to our listeners.

ADRIANA: Yay. So we start with...our first question is always, what are y'all drinking?

AUSTIN: Red Bull, sugar-free

TED: Coffee.

AUSTIN: I had an espresso earlier.

ANA: Y'all are kind of crazy to still be doing caffeine at this time, like, East Coast 4:00 p.m., Pacific time 1:00 p.m. I have a strict cut-off of no more caffeine at 12:00. I just finished drinking my yerba mate, 160 milligrams.

AUSTIN: Yeah, I drink caffeine. I don't know; I don't really have a ton of problems with caffeine. So earlier this year, when we did Deserted Island DevOps in Michigan, I was like, okay, as the organizer of this, I can finally make unreasonable craft services requests. And so I was like, we got to have sugar-free Red Bulls because I like energy drinks. I like sugar-free Red Bull.

So as it was requested, there was a, you know, they would fill it up. And that was actually maybe a mistake because I think I drank like three or four of them, and I was too nervous to eat. So I think I just kept drinking sugar-free Red Bulls. And about noon on the first day, I felt like I was vibrating on my skin.

ADRIANA: Oh my God.

AUSTIN: So I did find my limit of sugar-free Red Bull, and it's about four in two hours.

ANA: [laughs]

ADRIANA: Yikes. When I was in university, I used to have a can of Coke usually for a 9:00 or 10:00 a.m. class because I don't drink coffee. [laughs] I remember experiencing the vibrating in my skin sensation back then, and then I'm like, yeah, maybe I shouldn't do this anymore.

TED: Caffeine has no discernible effect on me anymore, which in and of itself is probably bad.

[laughter]

ANA: It might mean something.

AUSTIN: It's a sign. It's a signal. [laughter] It's some kind of signal, at least.

[laughter]

TED: The adrenal glands are just like, we give up. We give up.

AUSTIN: So we've stopped trying. This guy is going to do whatever.

[laughter]

ANA: I've switched over to water for the rest of the day. So, what about you, Adriana?

ADRIANA: I've got water. I do plan on having a lovely mug of green tea later, though. Caffeine doesn't; at least green tea doesn't affect me if I have it late in the day. I'll have homemade bubble tea at like 9:00 o'clock at night, and I'll be fine, so yay.

TED: With your own tapioca boba?

ADRIANA: Yeah, yeah.

TED: Whaaat?

ADRIANA: Yeah, I get it on Amazon, yeah. And I can control the amount because I like it, but I find it too filling. So I'll have like just a few.

ANA: Two pearls.

ADRIANA: Yeah. [laughs] Sometimes five if I'm feeling super adventurous. But yeah, it boils in like one to two minutes, and then it's done. So that's my plug for homemade bubble tea.

TED: Whoa.

ADRIANA: Mind-blown. [laughs]

AUSTIN: I've not had boba in so long. I miss it. I used to get them when I came up to San Francisco when I would travel, back before the pandemic and everything. I would always really get into some boba.

ANA: That was the case when I worked at the San Francisco Uber office. We would always take walks around the block, and we would end up at Boba Guys or Asha Tea House. And it was just like the perfect 45-minute break, nerd out, get some sun, get boba. Sometimes it was a work expense because it was a work meeting, and I was like, this is nice tech privilege.

ADRIANA: Yes, yes. I would totally do that, too, at one of my old jobs. It was like the daily bubble tea round. [laughs] And I used to work in Downtown Toronto, so there's like tons of places now. So never wanting for bubble tea.

[laughter]

TED: They always dunk way too much sugar into it for my liking.

ADRIANA: Yes.

TED: So the idea of homemade boba sounds really enticing.

ADRIANA: Yeah, that's why I like it too because I find they do put too much sugar, and I can control the amount of sweetness because I can't even take the amount of sweetness that Starbucks puts in its hot chocolates, like, it's sugar overload for me.

ANA: When I request bubble tea, I'm always like the 75%-80% sweetness person, so I'm the complete opposite clearly.

ADRIANA: [laughs] I mean, you do enjoy condensed milk in your drinks, too, right?

ANA: And I cannot do sugar-free Red Bulls, like, no. Sorry, y'all.

[laughter]

AUSTIN: I mean, it's fine.

ANA: I mean, you're caring about your health which is better than what I do, so I think --

AUSTIN: I mean, I like sugar in things, but I just don't like it in drinks these days. I don't know; I've been trying to reduce sugar intake in general.

TED: I mean, it's not like regular Red Bull tastes good, so you might as well get the sugar-free, right?

ADRIANA: [laughs]

TED: Like, it's not like you're losing something.

AUSTIN: Well, that's true. I mean, there are not a lot of energy drinks that taste good. Is this just like an energy drink review podcast now?

[laughter]

ADRIANA: I guess so.

AUSTIN: All right.

ADRIANA: Apparently, apparently.

[laughter]

ANA: I still think Green Monster tasted really good growing up. That was definitely a go-to.

AUSTIN: Hmmm. See, Monster Zero, I think, tastes better than the regular ones. And then, for a while, I went for the Bang energy drinks. Those are a lot.

ANA: I just get turned off by the marketing.

AUSTIN: Well, the marketing is also bad. I feel like we need a new kind of energy drink revolution or energy drink model or concept, like something --

ANA: There is actually something new that I just discovered. And I discovered it because somehow I was going through Twitter, and I found a founder who follows me on Twitter who's also Costa Rican, lives in San Francisco, and is building a company on making coffee without coffee beans.

ADRIANA: That's cool.

AUSTIN: Interesting.

ANA: I don't know the name of it.

TED: What does it mean?

ADRIANA: [laughs]

ANA: It's using, like, cocoa nodes. I don't know how they're not processing the cocoa beans, but you're still extracting caffeine. And you still get the same flavor profiles that a lot of coffee lovers are able to get out of it. At least, that's what I gathered from reading her tweets or company's tweets.

AUSTIN: Is it like a sustainability thing or?

ANA: Yes, yes, yes. It's a sustainability angle of it, too, where it's like with the way that climate change is going, we're not going to be able to keep up with the coffee demands, so we need to look for alternative routes.

AUSTIN: I like people when they try to head off, or they look at the dystopia, and then they just kind of like say, "You know what? We're going to just jump one step ahead of that.

ADRIANA: [laughs]

AUSTIN: We aren't at the collapse quite yet but let's go ahead and see how much VC cash we can wring out of monetizing the rot."

ANA: I mean, I'm super biased as being from Costa Rica, but they're one of the biggest exporters who are known for their beans that I'm like, to be a founder...that it's like, as a country, we stand for taking care of our environment and sustainability always being part of the way that you grow up, like, to then carry that on in Silicon Valley, I was just clapping hands, just excited. [laughs]

AUSTIN: I know it's a cool idea. Don't get me wrong on this.

ANA: Well, I will take us to our technology podcast.

ADRIANA: Technology. [laughs] Yes, let's get into the technology meat of it. So I guess, for starters, Ted was my co-host on one of the On-Call Me Maybe episodes where we had Luiz Aoqui talking about Instrumenting Nomad with OpenTelemetry. Let's use this as an opportunity to get to know Ted and Austin better.

AUSTIN: I nominate Ted to go first.

ADRIANA: Awesome, awesome. [laughs] I feel like when we met in person at KubeCon, you striked me as way more interesting than me. [laughs] So I feel like you've got really cool stories to tell, like how you got into tech and how you got into OpenTelemetry. [laughs]

TED: I have definitely lived a very non-linear life, which leads to, like [laughter], oh, there's the one time I was racing an electric car in Denmark kind of thing or whatever. Tech, I initially moved to San Francisco to be an animator. So I guess I got started in the animation industry but in 3D animation. And I did have a computer science degree, so it was very techy. Besides animation, I was really interested in what's called rigging, which is where you set up the armature, like all the controls you use to move the characters around and make all the muscles squish up and stuff like that, and that was a lot of fun.

But I actually got into, like, internet stuff through environmental activism, especially climate change stuff. It's kind of hard to remember, but back in the day, it was not something people really had a big opinion about. It was very hard to get news stories written about this stuff if you weren't connected to a university campus or in a particular subculture. Like, climate change was just this super weird, vague concept. So we started using this fun thing called the internet to start doing mass mobilizations and media stuff, particularly with this guy named Bill McKibben, who turned it into this called 350.org, and eventually, that turned into the Sunrise Movement. Oh, I'm out, like, too old. [laughs]

So I moved on and then ended up getting into distributed operating systems stuff which was super nerdy. But basically, we didn't have very good platforms to build this scalable tech on. And there was Heroku, but that was about it. So I was working on an open-source version of a Heroku-like platform called Cloud Foundry. And then through that, got really, really annoyed at how difficult it was to observe what these things were doing and recreate it, especially because it was an operating system that we're shipping to all these big companies, and they're operating it, and then they're having problems. And I'm needing to recreate what it is that they're doing, and I can't just stand up their version of this system.

So I really became obsessive about stuff like tracing, which was not a big, popular concept at the time, and the ability for all of these different users who were running this thing who had their own collection systems, and they were using their own observability tools but being able to pump into all of those things. And that combo led to me becoming really interested in OpenTracing because it was like this perfect combo of, like, oh, it's distributed tracing, but it's just the interfaces. You can connect this up to any back end, and it really scratched an itch for me.

And the open-source projects I was working on around containers and stuff like that were kind of like a tire fire as far as the way the corporate backers of those companies were treating each other. It was also an opportunity to help build a new open-source community that sucked less. And so, for all those reasons, Ben Sigelman convinced me to hop over just this newfangled thing called Lightstep that existed for like a couple of months. So I joined the Lightstep train. I think I was lucky 13 when I got hired.

ADRIANA: Nice.

ANA: Did you join OpenTracing project? Because that's what was going on then, right? This is pre-OpenTelemetry.

TED: Yeah. So before OpenTelemetry, there was a smaller project called OpenTracing that was much more narrow in scope. It was just focusing on tracing, and it was also just focused on instrumentation and interfaces. Like, the most annoying part of all of this, when you get into tracing instead of just logs and metrics, is that you have to propagate this context. And that is this huge lift for anyone who gets interested in this concept and just tries to do it on their own. They discover it's just this huge lift to do this context propagation, both serializing and deserializing this stuff and then shoveling it through the runtime of your program.

And so OpenTracing was designed to really just solve that problem for everyone without getting up in anyone's business about what they were going to do with that data. The scope of the project was a lot smaller. But while I was working on that, I became really obsessed with the idea that we needed to unify all of these different signals. It wasn't just about distributed tracing; it was like logs and metrics, and you could generate metrics out of your traces and all this stuff.

So I got really interested in this idea of having a unified observability platform because that was actually this other problem that I had, which was all of these data sources being disconnected from each other. And in order to do that, you really need to have a unified telemetry system. It works much better if all the data is coming out in an integrated fashion. And around that time, another project called OpenCensus had kind of come out of Google as well that was more focused on solving that problem. And the merging of these two things led to OpenTelemetry.

ANA: I remember being at the KubeCon where they announced the merge and OpenTelemetry is a thing. And I was just like, this all sounds really cool. I hadn't really been too exposed to the space. I knew about the problems being in SRE, but I was like, okay, this all sounds really cool.

TED: Yeah, I was really proud of everyone to be willing to merge all this stuff and create a standard. We needed to merge because having multiple projects in this space actually was working at cross purposes because this is this cross-cutting concern that has to go everywhere. So it's not like you can have different databases, and you can have your database, and I can have my database with instrumentation and all of this stuff. It really seemed to work a lot better if we joined forces.

And also, both of these projects were solving one half, like; OpenTracing really got it right around keeping this stuff stable and having the interfaces decoupled from the implementation. And it was kind of designed correctly in my mind for something that was going to be cross-cutting concerns, you know, sort of like how Log4j and some of these things work. Whereas OpenCensus was this like big ball of code, I was concerned about the way that was going to lead to maintainability and breaking changes.

[laughter]

Don't get me started on gRPC. And I felt like it might be going that same direction. But I really appreciated that they were doing the work of trying to build this client that sucked in all of these different data types. And it was also actually confusing to people using OpenTracing that we didn't supply a standard implementation. So it was really...it was like peanut butter and jelly. And I'm really appreciative of everyone who's working on both projects to be willing to come together. But I feel like we lost a year. [laughs] But at this point, it's all water under the bridge.

AUSTIN: Yeah, what's a year between [inaudible 16:01]

[laughter]

ANA: What about you, Austin? How did you get started in technology and working in the observability space?

AUSTIN: How far back do you want me to go? I was a very bullied child. [laughter] No. So I guess similar to Ted, I also...my shadow does not cast a straight line upon the earth. I've gone through a lot of different sort of career aspirations and interests and things like that. And then eventually, when I was in my late 20s, I guess I decided that at some point, you need to actually make money to survive and got into technology more professionally.

I went into IT and did a lot of stuff with telecom and server break/fix kind of general IT stuff. But I always, you know, I grew up with computers. And I grew up learning how to, you know, write little programs in BASIC and being fascinated by how the computer worked and that you could make it do what you wanted it to do. And at some point, I went back, finished college, got into CS, and wound up at a startup called Apprenda. And Apprenda did platform as a service for .NET. And this was like early, you know, this is before Docker. This was before containerization really kind of took over.

I do very distinctly remember going to one of the early DockerCons and looking at the presentation and turning to my co-worker and being like, "Oh, we're so fucked," [laughter] because this was like what we did but better. So they, for a variety of reasons, went out of business. And in that transitory period, I tweeted out, "Hey, I'm looking for something in open source. I'm looking for something in this cloud space." Because one reason people might have heard of Apprenda is because we had the insight to purchase a small company called Kismatic, and Kismatic was the company that put on the first KubeCon.

ADRIANA: No way.

AUSTIN: Before they donated it to the CNCF.

ANA: Whoa.

AUSTIN: And so there are actually very special editions of KubeCons like the one year where we ran, or we were part of the KubeCon organizing experience, I guess...there's a KubeCon t-shirt that has an Apprenda logo on the sleeve before they went away from putting sponsor logos on the shirts. But anyway, because we kind of were in that early cloud-native community, Ben Sigelman saw this tweet. And actually, that same day, I think I told Ted like, "Hey, Ted, go talk to this guy." [laughter]

And I remember sitting at home and getting on Zoom with Ted, and Ted pitched me on Lightstep and OpenTracing at the time, obviously. And my motivating factor there is really that, you know, I think you can't work in ops, in any kind of operational capacity with computers without just wanting to destroy all computers [laughter] because computers are inscrutable to our frail human senses.

And I had so many late nights and weekends spent just trying to figure out what the hell was going on; why is this working? You know, not only why is this not working, but why is this working? Here are my expectations, and then here's reality. And I got to know all this arcane stuff, and I have to manually piece together the execution of this distributed system across however many nodes, and it sucked. It absolutely sucked to do. So Lightstep was very appealing to me from that perspective.

I joined Lightstep. I was actually the first full-time remote hire. There was one other remote employee at the time, but she had started in person and then turned into remote. So I was the first fully remote. I think I was employee 40-something.

ADRIANA: Nice.

AUSTIN: I flew out to California, met everyone. It was great. And then I've been working with OpenTracing and then OpenTelemetry ever since. I think from the OpenTelemetry perspective; I come to it from a slightly different position than Ted. To me, it's less about the unification and more about the standards-making that really got me into this. Because one of the biggest challenges from an end user perspective is all of the toil and all of the duplicate of work that goes into just creating the ability to monitor a system, that's because there is no standard.

Every piece of software you use, unless you are in a very tightly vertically integrated stack like you're in the .NET world or something where there are a lot of conventions around this, and there is this provided monitoring libraries and everything, you're basically forced to go to some proprietary commercial solution, or some sort of community supported thing and ask them like, "Hey, can you please give me the ability to understand what I'm doing?" And I think that sucks. That sucks for the end users, and it also sucks for the implementers of those open-source projects, or those frameworks, or whatever, because they also don't have a clear answer. They can't serve their users well.

So what really has kind of attracted me to OpenTelemetry and one of the reasons that I work so much on the community side of OpenTelemetry is to push forth that standards-making approach to getting telemetry...making it ubiquitous, making it easy to use, so on and so forth.

ADRIANA: And what have both of you seen since you started working in OpenTelemetry in terms of uptake, adoption, enthusiasm? I mean, obviously, we know all the major observability vendors are all in on it, but what about the people who are actually instrumenting their code? Are people psyched about it?

AUSTIN: Psyched is maybe understating it slightly. I think people are ridiculously excited. I think it's actually almost to a degree that it harms the project in some ways because people are coming in so early and trying to use it in anger.

ADRIANA: [laughs]

AUSTIN: And it's very similar to if you were around like in the early days of Kubernetes, and CoreOS, and Rocket and like that whole thing. We're at the beginning of the journey, not the end. And we're certainly not even, you know, we're maybe approaching a plateau in the next 12 months with OpenTelemetry where there is kind of a...the rate of change should somewhat decrease. But we're still in the super, super early days. And if you were around for Kubernetes and around for how Kubernetes developed, you know, comparing the Kubernetes of like 2012 to the Kubernetes of 2022 or whatever, maybe not 2012 but --

TED: Vaporware.

[laughter]

AUSTIN: But it was a thing where people were jumping into this with two feet, and they were jumping into an empty pool and getting their legs broken. [laughter] In a lot of ways, I feel like that's what's happening now with some people in OpenTelemetry is they're jumping in before the pools fill all the way up. And sometimes you'll be really successful with that, and sometimes you won't. It really depends on, like, there are too many qualifiers needed. Are you just using this one signal? Are you in this particular stack? Do you have a high level of operational maturity in order to build this stuff yourself?

Like, so it's exciting, but it's also terrifying because now we have a user base. Now we have people that are using this and relying on it. So that makes it much more difficult in some ways to iterate because you have to bring all those people on with you, too, right?

ADRIANA: Yeah, it's interesting you mentioned that because I remember last year when I was working at Tucows, I was trying to get everyone on board with OpenTelemetry. And we actually had Ted and Liz come to talk to a bunch of the Tucows folks because they're like, "Well, you're saying we should use this, but we don't know if OpenTelemetry is ready." And so Ted and Liz came to basically do a Q&A to help, I guess, ease some of those concerns. But definitely, I can say that we're very excited to be using it and couldn't wait for it to move faster. And I can definitely see being on the other side of things how that can be extremely stressful.

TED: Yeah, I was part of a really great end-user conversation today, which is a shout-out, by the way, to anyone who's listening to this. OpenTelemetry has a monthly end-user discussion group. You can find it on the OpenTelemetry calendar, which you can find, I think, on our website, and certainly in our community repo on GitHub.

But it is interesting to see how far ahead the users are of everyone and everything else. The biggest stuff we were discussing was people trying to do really complicated fleet management of their collectors because they're using their collectors to do just a crazy amount of work. I was kind of surprised at what people were using the collectors to do. People are building fairly advanced tail sampling pipelines and things, sort of just bolting stuff onto the front of whatever vendor observability tool they're using because those guys haven't built the features yet.

And there was this general complaint of, like, why is no one offering a unified observability platform? Why is there no place I can shove all this OTLP data into and get all of this cross-signal analysis and stuff? And I was like, well, you know OpenTelemetry is not even fully stable yet, and logging finally has just become stable. So we're basically just becoming stable. And we still really need to stabilize the data schemas. And prior to OTEL, there was no, as far as I'm aware of, no project, even like a proprietary project, that tried to collect all of this data and provide some kind of unified workflow across metrics, logs, and traces.

Like, how fast do you expect this product department to move in these companies? My prediction is two years. Two years from now will be enough time for at least some of these companies to produce a strong enough unified observability platform that the other companies go like, "Oh, crap," and kind of like wheel around to they're trying to do the same thing. But it's just interesting to see that people are really champing at the bit, whereas a couple of years ago, I feel like people were not even thinking about it, yeah.

AUSTIN: I think it's interesting because the things that people are asking for, like, the tail sampling one's actually a really good example because I see a lot of people coming into the OpenTelemetry channels and the CNCF Slack, and they're like, "Hey, I'm trying to set this up. I'm trying to set this up. How do you do this?" And people aren't, you know, some of us are forced to respond. It's like, well, you've discovered why people pay money for this, right? [laughter] Like, you've cracked the code. Like, this is so hard, and like, yeah, it is. That's why people pay someone else to do it for them.

Because you're basically saying, "Hey, how do I set up this very complicated distributed consensus algorithm that is lockless and also can scale up to millions of QPS?" I think people have not a difficult time understanding it but a difficult time considering that observability problems as a space are really just as big and thorny and complicated as anything else. Like, it's a specialized sub-discipline that really, there's a countable number of people on the face of the planet that are like really, really super deep into this and really care about it. And most of those people are working on this for commercial interests.

So I don't know if we're going to see a move towards more things being pushed into the open-source world. I do kind of think that's the case. I feel like so much of the stuff that is right now proprietary and commercial is going to get pushed out. And speaking with just my OpenTelemetry hat on and not with my Lightstep hat, I think that's a good thing because the barrier to entry to create an observability product is actually very, very low right now.

I just saw, like, in the past year or so, there's been like two or three new ones that have come out that are just like, yeah, here's something where you throw your metrics in, your logs in, your traces in. And it runs on ClickHouse or some other sort of column store, and we slapped Grafana in front of it.

ANA: [laughs]

AUSTIN: And you can do all this stuff. And it's true; you can; you can do all this stuff. And three years ago, that wasn't possible. So OpenTelemetry has really opened the door for a lot of the...perhaps the charitable way to describe this is innovation. Innovation in this space is possible now because we have moved that point of integration and shifted it left and shifted it more towards the providers of software or this open-source project to provide instrumentation and da-da-da-da.

The flip side of that, though, is we're in this awkward teenage phase, I think, on the analysis side. Everyone's trying out different things. Everyone's trying different clothes. And one week, you're wearing Hot Topic, and the next, you're wearing J.Crew.

ANA: Oh my God, I love it.

[laughter]

AUSTIN: But with time, and as that shift-left continues to happen, then what you'll see, hopefully, is more commercial and open-source solutions that really narrow in on the analysis piece because that's the thing that I think we just don't do a good job of.

A buddy of mine works for a very small startup. And he's like one of the only support engineers they have. He was complaining earlier today in a Slack I'm in. And he's like, you know, so and so is having a problem, and one of the engineers says, "Oh, just go find the dashboard for it." And then he shows this video he recorded of him just scrolling through thousands of Grafana dashboards in a list. And there's like zero, you know, that's not helpful.

Go look at the dashboard is not a helpful solution to I'm having this problem. But that is what we treat as like, oh, this is what observability gives you. Now you can make those dashboards, and before, you couldn't make those dashboards. And I don't think that's the case. I think observability has to be more than you can make the dashboard now. It has to be about tell me your problem. And now, we have tools that can solve that problem for you or help you to solve that problem in a way that you couldn't before.

ADRIANA: And I think that's why, like, it's equally important to promote OpenTelemetry but also those observability practices because otherwise, you fall into that trap of the wallow dashboards and relying only on logs for troubleshooting or what whatever other anti-patterns that so many organizations practice because that's all they know, right?

TED: But there really is room for the analysis tools to do a better job of improving our workflows. Like, I'm sitting in front of like five million dollars worth of computer hardware, but I'm trying to find correlations by looking at squiggly lines with my eyeballs. And I can't get around that really stupefying workflow until someone actually builds a computer program for me to just find the correlations and be like, here.

ANA: I mean, we're starting to see that. I think those types of companies are starting to come up, and a portion of it is scary, a portion of it is cool. Like, it is exciting to see AIOps movement and just see a lot more of the automations on operations, reliability, and debugging.

TED: I'm really curious to see people from other domains start to tackle this too. Like, there's this whole world of data science out there and big data analysis tools that are all about trying to find signals in the noise when you have 18 bajillion different dimensions on your data. That's like a whole practice that just has nothing to do with observability directly the way we practice it, but it basically does. And I'm curious; I'm waiting to see people take that big data scientists' toolkit and start piping OTLP data into it.

ANA: Or even a way that the software is asking questions, and it's just pinging the answers to the collector or whatever you're sending OTLP to. Well, as we're getting ready to wrap up, we wanted to leave some fun things for our listeners to learn. We are all pretty distributed as a team. What are one or two things that you can't live without, whether it's a workflow in your work week or something you have at your desk at your home that just makes your day-to-day a little easier?

TED: I'd say I can't live without sleep, but I've proven that wrong.

[laughter]

ADRIANA: Very fair point.

AUSTIN: I think the thing I can't live without is my hats. It's very important to have a strong visual brand, and for me, that is hats.

ANA: [laughs]

ADRIANA: So, what's today's hat?

AUSTIN: Podcasting, famously a visual medium, I'm holding up one of my signature black baseball caps, New Era size eight fitted. I actually buy these. I buy like four every year.

ANA: You're not sponsored? [laughs]

AUSTIN: I know. Actually, yeah. So hey, New Era, if you're hearing this, hit me up.

ADRIANA: [laughs]

AUSTIN: But actually, strike the name. They aren't paying me to put their name on, to advertise them. [laughter] But, yeah, I actually go through about four of these a year because I am really bad about remembering to bring raincoats or umbrellas or anything. So invariably, I will get caught in the rain with them, and then they get wet, and then they shrink, or the bill gets a little floppy, or I get sweaty in it, or whatever. It always looks like I have a new one because I probably do always have a new one. [laughter] Like, if you go two or three months without seeing me in person, then the next time, it'll be a fresh hat, so let's go with the hats.

ADRIANA: Nice.

ANA: Do you have anything, Adriana?

ADRIANA: My creature comfort for working; I guess I have a couple. I have a lava lamp at my desk. It's very relaxing to watch it do its thing. I have a little folding treadmill in my home office. So whenever I need to blow off steam and especially when it's cold out, and I don't feel like going outside like today because it's near freezing and there's snow, I hop on my treadmill, and I chill and watch some Star Trek: Voyager while I'm on my treadmill, and that relaxes me. [laughs] So that's my Star Trek shout-out, my super nerd out. [laughs] How about you, Ana?

ANA: For me, it is my bright and colorful yellow Hydro Flask that keeps me hydrated throughout the day. But at the bottom, it has this cute dodge on a freaking piece of wood typing on a computer sticker that got given to me by Denise Yu. So every time I'm on calls and drinking from it when I see my face, like, it just makes me happy, which is always like a nice, little thing to have in your work there.

ADRIANA: Aww. [laughs]

ANA: And I hope I brighten people's days, too, with a cute sticker. [laughs] And then my blanket, just, like, I'm always cold. So I just kind of always seek a little comfort while I'm at my desk.

ADRIANA: That's awesome.

ANA: And I have two more questions that maybe we'll get through. So we all recently got to meet at KubeCon. But for some reason, even though we're a DevRel team, no one thought about taking their phone out and snapping a picture. So how did it feel to finally meet your team in person, as we never all got to be in the same place until now?

AUSTIN: Everyone except Ted is shorter than I thought they were.

[laughter]

ANA: Thank you.

AUSTIN: Sorry.

ANA: We have actually said it on this podcast. Adriana and I are really proud to be short Latinos.

[laughter]

AUSTIN: In fairness, I had met everyone in person except Adriana before. So it's also one of those things where I'm the tallest person on the team. It's a little like, oh, I'm the tall person again.

ANA: [laughs]

ADRIANA: You're quite tall. I didn't expect you to be so tall. [laughs]

AUSTIN: I think a lot of us, I mean, literally all of it is the perception you get of people when they're on webcam, right?

ADRIANA: Yeah, it's true.

AUSTIN: And I have my camera set up explicitly at eye level. So if I'm doing, again, podcasting, famously a visual medium, but like, if I sit up in my chair and I'm looking, like, I am eye-level with the lens. I don't know what perception that gives people about my height. There's no angle, right?

ADRIANA: Yeah, yeah.

AUSTIN: But a lot of other people, it's either camera pointed up, or camera pointed down or something. And so you just get like this thing it's like your mental model of someone's height is completely based on that.

ANA: You used mental model in a non-architecture way. And my mental model is breaking.

[laughter]

AUSTIN: That's my value add. Obviously, the reason we didn't get a chance to take a picture is because who would have held the camera?

ANA: That's true.

ADRIANA: I know. We could have done a selfie.

AUSTIN: We could have done a selfie. I have long arms, too, so.

TED: One of the 8,000 people walking around [crosstalk 37:39]

AUSTIN: Also true.

ADRIANA: It never even occurred to me to take a picture, or it was always the thought of, like, we'll get to it tomorrow.

AUSTIN: We were all pretty busy.

ANA: Yeah, that is true.

ADRIANA: Yes, yes. Very much.

ANA: Well, for the next one, we'll snap some photos and post them for our podcast listeners to get a chance to see some of the brains behind the podcast.

AUSTIN: Next year in Amsterdam.

ADRIANA: That's right.

ANA: Yes.

ADRIANA: Yes, woo-hoo.

ANA: Last question before we wrap up the episode. If you were to quit technology, what would you do?

TED: I would start making more short films. I'd get back into cinematography.

ANA: Nice.

ADRIANA: That's awesome.

ANA: Austin.

AUSTIN: Hmm, I don't know. Is there a job where you get to destroy computers? I guess I'd be e-waste recycling; there you go. Destroying computers all day. [laughter] It would be very cathartic.

ANA: What about you, Adriana?

ADRIANA: Oh, man, I'm trying to think because I kind of did that a few years ago where I quit tech temporarily to become a professional photographer and realized that that wasn't for me, so now I don't know. I think I'd just...I don't know if I could leave tech because I feel like I'm unproductive if I'm neither digging into tech nor writing about it. So I don't even know what I would do with myself.

ANA: [laughs]

ADRIANA: Maybe I'd be rock climbing a lot, [laughs] maybe that's it. Find me on the rock. Well, maybe I'll actually go outdoor rock climbing at some point. [laughs]

AUSTIN: I actually have a serious answer for this question, by the way, that I just recalled. My serious answer is I'd probably go back into academia.

ADRIANA: Oooh. That's cool.

AUSTIN: I enjoy doing research. I enjoy writing. I enjoy teaching, so.

ANA: Nice.

ADRIANA: That's awesome.

ANA: I feel like, for me, it would be like a travel photographer, but someone that does only ocean stuff, like, just being on the water will call my attention the most, yeah, because that would be leaving tech. If I got a chance to stay in tech, I would do my own startup, but that's a whole other thing.

TED: Nice.

ANA: Sweet. Well, that gets us to wrapping up our episode. Thank you, Ted and Austin, for joining us today in the episode of getting a chance to meet the Lightstep DevRel team. Thank you to all of our listeners for joining today's episode. We loved talking a little bit more about OpenTelemetry, getting a chance to talk about caffeine, and just about where we see observability going.

Don't forget to subscribe and give us a shout-out on Twitter and other social media platforms @oncallmemaybe. Be sure to check out the show notes on oncallmemaybe.com for all the links and resources we've talked about on today's podcast. And don't forget to connect with us and our other guests on social media. For On-Call Me Maybe, we're your hosts, Ana Margarita Medina.

ADRIANA: And Adriana Villela with...

Ted and Austin: Peace, love, and code.

ADRIANA: [laughs] Love it.

AUSTIN: Great job, everyone.

ADRIANA: Woo-hoo.

TED: Aloha.

31 episodes

#Tech #Austin Parker #Monitoring #Distributed Tracing #Oncall #Software #Software Development #Technology #Open Source #Microservices #Adriana Villela #Ana Margarita Medina #Trace #SRE #Call #DevOps #Reliability #Site Reliability Engineering