Welcome to The Data Flowcast: Mastering Airflow for Data Engineering & AI — the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward. Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems. Podcast Webpage: https://www.astronomer.io/podcast/
…
continue reading
1
How Laurel Uses Airflow To Enhance Machine Learning Pipelines with Vincent La and Jim Howard
23:58
23:58
Play later
Play later
Lists
Like
Liked
23:58
The world of timekeeping for knowledge workers is transforming through the use of AI and machine learning. Understanding how to leverage these technologies is crucial for improving efficiency and productivity.In this episode, we’re joined by Vincent La, Principal Data Scientist at Laurel, and Jim Howard, Principal Machine Learning Engineer at Laure…
…
continue reading
1
How Vibrant Planet's Self-Healing Pipelines Revolutionize Data Processing
23:52
23:52
Play later
Play later
Lists
Like
Liked
23:52
Discover the cutting-edge methods Vibrant Planet uses to revolutionize geospatial data processing and resource management.In this episode, we delve into the intricacies of scaling geospatial data processing and resource allocation with experts from Vibrant Planet. Joining us are Cyrus Dukart, Engineering Lead, and David Sacerdote, Staff Software En…
…
continue reading
1
The Future of AI in Data Engineering With Astronomer’s Julian LaNeve and David Xue
23:36
23:36
Play later
Play later
Lists
Like
Liked
23:36
The world of data orchestration and machine learning is rapidly evolving, and tools like Apache Airflow are at the forefront of these changes. Understanding how to effectively utilize these tools can significantly enhance data processing and AI model deployment.This episode features Julian LaNeve, CTO at Astronomer, and David Xue, Machine Learning …
…
continue reading
1
The Power of Airflow in Modern Data Environments at Wynn Las Vegas with Siva Krishna Yetukuri
24:31
24:31
Play later
Play later
Lists
Like
Liked
24:31
Understanding the critical role of data integration and management is essential for driving business success, particularly in a dynamic environment like a luxury casino resort.In this episode, we sit down with Siva Krishna Yetukuri, Cloud Data Architect at Wynn Las Vegas, to explore how Airflow and other tools are transforming data workflows and cu…
…
continue reading
1
Powering the Texas Rangers World Series Win With AI on Airflow with Alexander Booth
23:38
23:38
Play later
Play later
Lists
Like
Liked
23:38
The integration of data and AI in sports is transforming how teams strategize and perform. Understanding how to harness this technology is key to staying competitive in the rapidly evolving landscape of baseball.In this episode, we sit down with Alexander Booth, Assistant Director of Research and Development at Texas Rangers Baseball Club, to explo…
…
continue reading
1
Expanding the Data Engineering Toolkit at Reddit
45:48
45:48
Play later
Play later
Lists
Like
Liked
45:48
Welcome back to the Airflow Podcast.This week, we met up with Ben Wisegarver, a staff data scientist at Reddit who runs their data warehousing and data engineering functions.Reddit users generate petabytes of data every day that needs to be processed, stored, and analyzed by a wide breadth of backend services. Our conversation with Ben touches on e…
…
continue reading
1
GDPR, Self-Service Data, and Infrastructure Automation with Typeform
31:26
31:26
Play later
Play later
Lists
Like
Liked
31:26
Welcome back to the Airflow Podcast.This week, we met up with Albert Franzi and Carlos Escura from Typeform. Typeform is a tool that allows you to build beautiful interactive forms that you can use for a wide variety of use cases, including customer surveys, employee engagement, product feedback, and market research to name a few. In our conversati…
…
continue reading
After a bit of a break, we're back with the third official episode bundle of The Airflow Podcast. In this batch, we'll get a little bit deeper with current Airflow users and maintainers on core fundamental concepts in data engineering, architectures for operating modern data platforms at scale, and the process of maintaining and operating Airflow, …
…
continue reading
This week, we linked up with Airflow release manager, core committer, and Astronomer platform engineer Ash Berlin-Taylor to discuss the Airflow 2.0 roadmap [1]. There is some great stuff in the works around performance, autoscaling, and usability that we're excited about. In this episode, Ash lends his thoughts on the design, implementation, and va…
…
continue reading
This week, we had the pleasure of meeting up with Jarek Potiuk, Principal Software Engineer at Polidea and Apache Airflow committer, to discuss his most recent contribution to the community, Airflow Breeze. Jarek deeply values developer productivity and realized while building a team of Airflow committers that, in order to open a PR on the project,…
…
continue reading
This episode kicks off season 2 of The Airflow Podcast. In this next season, we'll focus on the future of Airflow and chat with leading members of the community to paint a picture of what's to come. We're pumped to be diving back into this project and look forward to the great conversations we have lined up.This week, we chatted with James Malone, …
…
continue reading
This week, we met up with Ash Berlin-Taylor to discuss the recent 1.10 release, what it's like to be a release manager for an open source project, Airflow's bid to graduate from incubating status, and the next phase of Airflow project development.As mentioned in our podcast intro, we at Astronomer are hiring Data Engineers who are passionate about …
…
continue reading
This time, we met up with WePay's Joy Gao to talk through her work on the RBAC components in the recent Airflow 1.10 release. We dove deep into what inspired her work and took some time to discuss what it's like to be a woman contributing to a predominately male open-source community. Hope you enjoy!If you'd like to get started using Airflow in you…
…
continue reading
In this episode, we dove into the relationship between Airflow and Kuberenetes and interviewed Daniel Imberman, Senior Software Engineer at Bloomberg (1:30), and Greg Neiheisel, CTO here at Astronomer (37:31). Daniel has done most of the work on the Kubernetes executor for Airflow and Greg plans to take on a chunk of the development going forward, …
…
continue reading
This week, we’ll examine conversations with both old guests and new to paint a comprehensive picture of Airflow’s pain points. While we still undoubtedly believe that Airflow is the future of ETL, it’s important to acknowledge that any incubating project will have issues, and bringing those issues to the forefront of the community’s attention will …
…
continue reading
On this episode, we linked up with Erik Bernhardsson (@erikbern), creator of Luigi and CTO of Better Mortgage. We chatted about everything from the motivations behind Luigi's creation and his current thoughts on Airflow- we hope you enjoy!Check out:- Erik's blog at erikbern.com- Our open-source library of Airflow plugins at github.com/airflow-plugi…
…
continue reading
In this episode, we dive into Airflow Best Practices and include longer portions of interviews with Alan Cruickshank (1:30), Business Insights and Data Manager at Tails.com, Chris Riccomini (7:27), Principal Software Engineer at WePay, and Bolke de Bruin(31:45), Head of Advanced Analytics Technology at ING. Hope you enjoy!We're still working to get…
…
continue reading
Episode 2 of The Airflow Podcast is here to discuss six specific use cases that we’ve seen for Apache Airflow. Here’s the lineup:Patrick Atwater (@patwater), Water Data Projects Manager at ARGO Labs: 2:03-5:35Maksime Pecherskiy (@mrmaksimize), CDO of San Diego: 5:35-23:06Scott Halgrim (@shalgrim), Data Engineer at Zapier: 23:06-27:27Bolke de Bruin …
…
continue reading
For the first episode of the Airflow Podcast, we met up with Maxime Beauchemin, creator of Airflow, to explore the motivations behind its creation and the problems it was designed to solve. We asked Maxime for his definition of Airflow, the design principles behind hook/operator use, and his vision for the project.Speaker list:Pete DeJoy - Product …
…
continue reading
A sneak peek at our upcoming podcast about Apache Airflow.Featured in this clip (in order of appearance):Pete DeJoy - Product Specialist at AstronomerPatrick Atwater - Water Data Projects Manager at ARGO LabsMaksime Pecherskiy - Chief Data Officer of the City of San DiegoBolke de Bruin - Head of Advanced Analytics at ING…
…
continue reading