Scaling Log Management with Renaud Boutet


Manage episode 230112918 series 1437556
By Discovered by Player FM and our community — copyright is owned by the publisher, not Player FM, and audio streamed directly from their servers.

Upcoming events:

A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019

FindCollabs Hackathon at App Academy on April 6, 2019

Log management requires the processing and indexing of high volumes of semi-structured data. A log management service takes log data and puts it in a cloud-hosted application so that application operators can access those logs to troubleshoot issues.

A large tech company will produce terabytes of logs. Those logs are produced on the host where a service is running. A logging agent on that host will transfer the logs to the log management service in the cloud. Once the logs are in the cloud, they are parsed, indexed, and stored in a way that is easy to query.

In 2014, Renaud Boutet co-founded Logmatic, a log management service that eventually became a leading provider. Logmatic was acquired by Datadog, and Renaud now works as a vice president at Datadog. In today’s episode, Renaud joins the show to talk about the architecture of a log management service. We talk about storage tiers, scalability requirements, failover strategies, and logging for serverless functions.

Full disclosure: Datadog is a sponsor of Software Engineering Daily.

The post Scaling Log Management with Renaud Boutet appeared first on Software Engineering Daily.

140 episodes available. A new episode about every 7 days averaging 57 mins duration .