Go offline with the Player FM app!
175: The Parts, Pieces, and Future of Composable Data Systems, Featuring Wes McKinney, Pedro Pedreira, Chris Riccomini, and Ryan Blue
Manage episode 398509692 series 3264623
Highlights from this week’s conversation include:
- Introduction of the panel (0:05)
- Defining composable data stack (5:22)
- Components of a composable data stack (7:49)
- Challenges and incentives for composable components (10:37)
- Specialization and modularity in data workloads (13:05)
- Organic evolution of composable systems (17:50)
- Efficiency and common layers in data management systems (22:09)
- The IR and Data Computation (23:00)
- Components of the Storage Layer (26:16)
- Decoupling Language and Execution (29:42)
- Apache Calcite and Modular Frontend (36:46)
- Data Types and Coercion (39:27)
- Describing Data Sets and Schema (42:00)
- Open Standards and Frontiers (46:22)
- Challenges of standardizing APIs (48:15)
- Trade-offs in building composable systems (54:04)
- Evolution of data system composability (56:32)
- Exciting new projects in data systems (1:01:57)
- Final thoughts and takeaways (1:17:25)
The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.
RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.
375 episodes
Manage episode 398509692 series 3264623
Highlights from this week’s conversation include:
- Introduction of the panel (0:05)
- Defining composable data stack (5:22)
- Components of a composable data stack (7:49)
- Challenges and incentives for composable components (10:37)
- Specialization and modularity in data workloads (13:05)
- Organic evolution of composable systems (17:50)
- Efficiency and common layers in data management systems (22:09)
- The IR and Data Computation (23:00)
- Components of the Storage Layer (26:16)
- Decoupling Language and Execution (29:42)
- Apache Calcite and Modular Frontend (36:46)
- Data Types and Coercion (39:27)
- Describing Data Sets and Schema (42:00)
- Open Standards and Frontiers (46:22)
- Challenges of standardizing APIs (48:15)
- Trade-offs in building composable systems (54:04)
- Evolution of data system composability (56:32)
- Exciting new projects in data systems (1:01:57)
- Final thoughts and takeaways (1:17:25)
The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.
RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.
375 episodes
All episodes
×Welcome to Player FM!
Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.