Manage episode 229714640 series 1437556
Twitter’s early engineers faced scalability problems that caused infrastructure failures on a regular basis. The infamous “fail whale” could happen as a result of problems in the application servers, the network, or the database layer.
When Twitter was scaling in its early days, the cloud providers were still immature. Engineers did not have access to the autoscaling cloud infrastructure that is available today. The early Twitter architecture was a combination of open source tools and internally created infrastructure custom built for Twitter’s workloads.
Evan Weaver was an early engineer at Twitter, and he saw the deficiencies of the data tools that the company had access to. Twitter engineers wanted access to a truly reusable data platform that would fit Twitter’s requirements: high availability, globally replicated, and transactionally consistent.
By 2012, Evan had left Twitter and started consulting for other technology companies. He found that databases across the industry were lacking the same properties that Twitter wanted, and the ideas for FaunaDB began to percolate. Around this time, there were two relevant papers about distributed databases that had come out: the Spanner paper from Google and the Calvin paper, a distributed systems paper from Yale.
With inspiration from the literature, his time at Twitter, and his knowledge from consulting, Evan started FaunaDB. Seven years later, FaunaDB is a fully fledged company with a cloud service offering. Fauna is an OLTP database used by companies like Nvidia, Nextdoor, and Capital One.
Evan joins the show to talk about his time spent scaling Twitter and the architecture of FaunaDB.
140 episodes available. A new episode about every 7 days averaging 57 mins duration .