26: Building Data Engineering Pipelines at Scale (with Data Warehouse, Spark and Airflow)

The Data Life Podcast

Content provided by Sanket Gupta. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Sanket Gupta or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.

3y ago 39:30

M4A•Episode home

Imagine you are at a beach and you are hanging out and seeing all the waves come and go and all the shells on the beach. And you get an idea. How about you collect these shells and make necklaces to sell? Well how would you go about doing this? Maybe you’d collect a few shells and make a small necklace and try to show to your friend. This is where we begin our journey on learning about data engineering pipelines.

Using an example of running a necklace business from shells - we learn about the following data engineering concepts:

1. ETL - Extract Transform Load vs ELT - Extract Load Transform concepts. Why Data Warehouses are great for analytics.

2. Spark for large data processing and hosting / running

3. Data orchestration using Airflow

My blog on Towards Data Science about moving from Pandas to Spark: https://towardsdatascience.com/moving-from-pandas-to-spark-7b0b7d956adb

Great book to learn about Spark: https://www.amazon.com/dp/1492050040/?tag=omnilence-20

Tools covered in the episode:

dbt: https://www.getdbt.com/

Databricks: https://databricks.com/

EMR: https://aws.amazon.com/emr/

AWS Redshift: https://aws.amazon.com/redshift/

Snowflake: https://www.snowflake.com/

Delta Lake: https://databricks.com/product/delta-lake-on-databricks

--- Send in a voice message: https://podcasters.spotify.com/pod/show/the-data-life-podcast/message Support this podcast: https://podcasters.spotify.com/pod/show/the-data-life-podcast/support

27 episodes

#Tech #Sanket Gupta #Data Science