Slack Data Platform with Josh Wills

1:21:43
 
Share
 

Manage episode 249929492 series 2396472
By Software Engineering Daily. Discovered by Player FM and our community — copyright is owned by the publisher, not Player FM, and audio is streamed directly from their servers. Hit the Subscribe button to track updates in Player FM, or paste the feed URL into other podcast apps.

Slack is a messaging platform for organizations. Since its creation in 2013, Slack has quickly become a core piece of technology used by a wide variety of technology companies, groups, and small teams.

The messages that are sent on Slack are generated at a very high volume, and are extremely sensitive. These messages must be stored on Slack’s servers in a way that does not risk a message from one company accidentally being accessible to another company. The messages must be highly available, and they also must be indexed for search.

When Slack was scaling, the company started to encounter limitations in its data infrastructure that the company was unsure how to solve. During this time, Josh Wills was the director of data engineering at Slack, and he joins the show to retell the history of his time at Slack, and why the problem of searching messages was so hard.

Josh also provides a great deal of industry context around how engineers from Facebook and Google differ from one another. When Slack was starting to become popular, the company quickly began to attract engineers from both of those companies. Facebook and Google have distinct solutions for how they have tackled the problems of data engineering.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Announcements

  • We are hiring a software engineer who can work across both mobile and web. This role will work on SoftwareDaily.com, our iOS app, and our Android application. We are looking for someone who learns very quickly and can produce high quality code at a fast pace. If you are interested in working with us, send an email to jeff@softwareengineeringdaily.com.
  • If you are planning a hackathon, check out FindCollabs Hackathons. Whether you are running an internal hackathon for your company, or you are running an open hackathon so that users can try out your product, FindCollabs Hackathons are a tool for people to build projects and collaborate with each other. FindCollabs is a company I started to allow people to find collaborators for their software projects, and our new hackathon product allows you to organize your hackathon participants to make your hackathon as productive as possible. Check it out at FindCollabs.com.

The post Slack Data Platform with Josh Wills appeared first on Software Engineering Daily.

510 episodes