AWS Glue Saves the Day and the Future of Big Data

THE DOPPLER

Our guest on the podcast this week is Jeff Thorn, CEO at Thorn Technologies.

We discuss modern-day data analytics for enterprises. We look at new and old AWS products like AWS Glue, AWS Kinesis, and Redshift. AWS Kinesis is something Thorn Technologies leveraged to create a product that allowed them to capture location data to track user behavior data at large trade shows. They could find out where users were spending time in the physical space. At first, it was hard to process the large amount of data coming in with user information and it sometimes took 24 hours to provide insights. When they migrated it to the cloud and used AWS Kinesis streaming data and Redshift data warehouse on the back-end they were able to get near real-time insight into behavior data. Kinesis is Amazon’s real-time streaming data processing service. It is good for when you have a lot of data coming in all at once and want real-time insights.

Where is big data now and where is it heading?

The top data products in the cloud change the way you can solve problems. Today, the top game changers are Hadoop and spin-offs based on the Hadoop infrastructure. Hadoop gives users more power because you do not have to pre-structure your data. It opens the door to new kinds of analytics. In the past, you needed a data warehouse that was designed specifically for the data it held and with new information you had to spend months re-designing it to fit into the existing system. With Hadoop and other distributed analytics platforms you can manipulate your data on the fly and write analytics specific to your data.

In the next five years, big data and analytics will continue to evolve. Right now doing your own analytics and making your own decisions is popular. However, IoT and AI are quickly becoming a reality. In five years it will not be you making decisions, it will be machines making decisions. We will see what machines tell us about our businesses.