Real-Time Data Transformation and Analytics with dbt Labs

Streaming Audio: Apache Kafka® & Real-Time Data

dbt is known as being part of the Modern Data Stack for ELT processes. Being in the MDS, dbt Labs believes in having the best of breed for every part of the stack. Oftentimes folks are using an EL tool like Fivetran to pull data from the database into the warehouse, then using dbt to manage the transformations in the warehouse. Analysts can then build dashboards on top of that data, or execute tests.
It’s possible for an analyst to adapt this process for use with a microservice application using Apache Kafka® and the same method to pull batch data out of each and every database; however, in this episode, Amy Chen (Partner Engineering Manager, dbt Labs) tells Kris about a better way forward for analysts willing to adopt the streaming mindset: Reusable pipelines using dbt models that immediately pull events into the warehouse and materialize as materialized views by default.

dbt Labs is the company that makes and maintains dbt. dbt Core is the open-source data transformation framework that allows data teams to operate with software engineering’s best practices. dbt Cloud is the fastest and most reliable way to deploy dbt.
Inside the world of event streaming, there is a push to expand data access beyond the programmers writing the code, and towards everyone involved in the business. Over at dbt Labs they’re attempting something of the reverse— to get data analysts to adopt the best practices of software engineers, and more recently, of streaming programmers. They’re improving the process of building data pipelines while empowering businesses to bring more contributors into the analytics process, with an easy to deploy, easy to maintain platform. It offers version control to analysts who traditionally don’t have access to git, along with the ability to easily automate testing, all in the same place.
In this episode, Kris and Amy explore:

• How to revolutionize testing for analysts with two of dbt’s core functionalities
• What streaming in a batch-based analytics world should look like
• What can be done to improve workflows
• How to democratize access to data for everyone in the business

EPISODE LINKS

• Learn more about dbt labs
• An Analytics Engineer’s Guide to Streaming
• Panel discussion: If Streaming Is the Answer, Why Are We Still Doing Batch?
• All Current 2022 sessions and slides
• Watch the video version of this podcast
• Kris Jenkins’ Twitter
• Streaming Audio Playlist
• Join the Confluent Community
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Intro to Event-Driven Microservices with Confluent
• Use PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)

Chapters

1. Intro (00:00:00)

2. What is MDS? (00:03:48)

3. What is dbt? (00:08:48)

4. Who uses dbt? (00:10:32)

5. How does someone get started with dbt? (00:14:30)

6. How does dbt fit into the world of streaming? (00:20:44)

7. How can you do unit testing with dbt? (00:24:04)

8. Will batch and streaming always be a part of the solution? (00:26:12)

9. What are event streamers doing wrong? (00:32:54)

10. What are some things to know about data testing with dbt? (00:37:19)

11. What should people be watching for in the industry? (00:40:41)

12. It's a wrap! (00:41:52)