Managing Meta's millions of machines

Ship It! SRE, Platform Engineering, DevOps

Anita Zhang is here to tell us how Meta manages millions of bare metal Linux hosts and containers. We also discuss the Twine white paper and how AI is changing their requirements.

Leave us a comment

Changelog++ members save 8 minutes on this episode because they made the ads disappear. Join today!

Sponsors:

• FireHydrant – The alerting and on-call tool designed for humans, not systems. Signals puts teams at the center, giving you ultimate control over rules, policies, and schedules. No need to configure your services or do wonky work-arounds. Signals filters out the noise, alerting you only on what matters. Manage coverage requests and on-call notifications effortlessly within Slack. But here’s the game-changer…Signals natively integrates with FireHydrant’s full incident management suite, so as soon as you’re alerted you can seamlessly kickoff and manage your entire incident inside a single platform. Learn more or switch today at firehydrant.com/signals
• Sentry – Code breaks, fix it faster. Don’t just observe. Take action. Sentry is the only app monitoring platform built for developers that gets to the root cause for every issue. 90,000+ growing teams use sentry to find problems fast. Use the code CHANGELOG when you sign up to get $100 OFF the team plan.

Featuring:

• Anita Zhang – Mastodon, Twitter, GitHub, LinkedIn
• Justin Garrison – Twitter, GitHub, LinkedIn
• Autumn Nash – Twitter, GitHub, LinkedIn

Show Notes:

Links of the week

• Decoder podcast with Drew Houston
• Twine: A Unified Cluster Management System for Shared Infrastructure

Faux or fo sho

• Attention is all you need
• Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo
• Causally Abstracted Multi-armed Bandits

Something missing or broken? PRs welcome!

Chapters

1. This is Ship It! (00:00:00)

2. Sponsor: FireHydrant (00:00:52)

https://firehydrant.com/signals

3. The opener (00:03:15)

4. Welcome Anita Zhang (00:16:28)

5. Meta's infrastructure (00:17:19)

6. Provisioning OS (00:18:34)

7. Fedora ELN & CentOS stream (00:20:00)

8. In-house automation (00:21:13)

9. What is Twshared? (00:22:54)

10. JournalD inside a container (00:24:44)

11. Host profiles (00:25:47)

12. Coolest sweatshirt ever (00:27:23)

13. Meta & open source (00:28:01)

14. Frequent releases and 1M hosts?!? (00:29:35)

15. Meta's AI fleet (00:30:48)

16. Production engineer vs Production engineer (00:31:43)

17. Other internal services (00:32:34)

18. OS challenges (00:35:05)

19. One size fits all? (00:36:07)

20. Meta's AI adoption (00:37:20)

21. Cost optimization (00:38:09)

22. Lots of abstraction (00:40:07)

23. Upcoming projects? (00:41:39)

24. Immutable file system (00:43:55)

25. Thanks for joining us! (00:45:36)

26. Sponsor: Sentry (00:48:37)

https://sentry.io/

27. The closer (00:52:34)

28. Faux or Fo Sho? (00:52:48)

29. Outro (01:02:04)