Manage episode 205416794 series 1437556
Cloud computing lowered the cost and improved accessibility to tools for storing large volumes of data. In the early 2000s, Hadoop caused a revolution in large scale batch processing. Since then, companies have been building ways to store and access their data faster and more efficiently.
At the same time, the sheer volume of data has increased and machine learning has given rise to methods of extracting signal from seemingly inconsequential data points. This confluence of factors gave rise to the role of the data engineer. A data engineer defines the data pipeline and supports data scientists and machine learning engineers.
Tobias Macey hosts the “Data Engineering Podcast,” where he covers the fast moving world of data engineering–including databases, cloud providers, and open source tools. Tobias and I covered a range of topics in the data engineering space and also spent significant time discussing the world of software engineering podcasting.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
135 episodes available. A new episode about every 7 days averaging 58 mins duration .