Data Engineering Podcast

Help
Help

Suggested Topics
- What is Podknife?
  
  Podknife is a curated podcast information and review site designed to be accessible in your browser from any device. We’re working to build the most useful podcast information source available by providing you with as much publicly available information about each podcast in our database as we can find and keeping it as up to date as possible.
- Can I listen to podcasts on Podknife?
  
  Yeah! There's a player at the bottom of the page.
- How can I get my podcast on Podknife?
  
  We’d love to hear about your podcast and consider it for addition. You can suggest your own by logging in and selecting the “Suggest a Podcast” link from the menu. Fill out the form and we’ll take it from there.
- How can I submit a review of a podcast?
  
  To submit a review of a podcast, go to the page of the podcast you want to review and click the "Submit Review" button below the image associated with the podcast. Clicking that button brings you to a new page where you can select a rating between 1 and 5 stars and write a brief review.
- How can I submit a review of a podcast episode?
  
  To submit a review of a podcast episode, go to the page of the podcast whose episode you want to review and find the specific episode you're looking for. Clicking on any episode reveals a "View Episode" button and clicking on that brings you to an episode-specific page where you can find more information displayed about the episode along with a "Submit Review" button. Clicking that brings you to a new page where you can select a rating between 1 and 5 stars and write a brief review of the episode.
- Is there a Podknife app?
  
  Not yet - we're keeping it in the browser at least for now. No app download required to access anything on the site.
- How can I change my display name?
  
  To change your display name, log in to Podknife and select "Account Settings" in the dropdown menu. From the Account Settings page, enter your desired new display name and press the "Save" button to confirm the change.
- How can I change my password?
  
  To change your password, log in to Podknife and select "Account Settings" in the dropdown menu. From the Account Settings page, enter your desired new password in the two fields below "Update Password" and press the "Save" button to confirm the change.
- How can I keep track of all my reviews?
  
  Within the menu, click on "My Profile", you will then be able to check all your reviews for both podcast & episode. You can also check all your favorite podcasts.
- How can I keep track of all my favorited podcasts?
  
  You can find podcasts you’ve favorited displayed on the My Profile page associated with your account (https://podknife.com/profile).
  You can share your My Profile page displaying your favorite podcasts as well as any reviews you’ve written publicly with a link in the form https://podknife.com/users/[displayname] where [displayname] is replaced by your Display Name.
- How can I favorite a podcast?
  
  On each podcast page you’ll find the outline of a star above the podcast title. Clicking this star fills it in and designates the podcast as a favorite (for logged in users) or invites the user to log in or register for an account (for users who are not logged in). You can find favorited podcasts displayed on the My Profile page associated with your account (https://podknife.com/profile).
- How can I register for a Podknife account?
  
  To register, click on the “Register” button in the top right of your browser window (or visit https://podknife.com/users/sign_up). Enter a Display Name to be associated with your account on the site (alongside any reviews you contribute, for example), an Email address to associate with the account and your password (twice for confirmation). Press the “Confirm” button and you should be all set, logged in and ready to review and favorite podcasts as you wish.
- How can I get in touch?
  
  We’d love to hear from you. You can reach us using the Feedback form (https://podknife.com/feedbacks/new) which can be found either in the footer on each page (for users who are not logged in) or in the dropdown menu (for logged in users).
  
  You can also reach us on Twitter by tweeting us @podknife (https://twitter.com/podknife) or on Facebook by messaging us from our page (https://www.facebook.com/podknife/).
- Log In
- Register
- Feedback
- Help
- Privacy Policy
- Terms of Use
© Podknife 2024

Please login or sign up to post and edit reviews.

This podcast currently has no reviews.

Submit Review

Favorite Report Add all episodes to Queue

Data Engineering Podcastinactive

Publisher |: Tobias Macey

Media Type |: audio

Podknife tags |: Data Science,; Interview,; Technology

Categories Via RSS |: Technology

This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.

People |: Tobias Macey
Premiere Date |: 2017-01-08
Frequency |: Weekly
Explicit |: No

This podcast currently has no reviews.

Submit Review

373 Available Episodes (373 Total) / Average duration: 00:53:42

Date Duration Most Reviewed Highest Rated

373 Available Episodes (373 Total)Average duration: 00:53:42

Apr 24, 2023

00:45:26

Summary

Real-time capabilities have quickly become an expectation for consumers. The complexity of providing those capabilities is still high, however, making it more difficult for small teams to compete. Meroxa was created to enable teams of all sizes to deliver real-time data applications. In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudderstack
Your host is Tobias Macey and today I'm interviewing DeVaris Brown about the impact of real-time data on business opportunities and risk profiles

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what Meroxa is and the story behind it?
- How have the focus and goals of the platform and company evolved over the past 2 years?
Who are the target customers for Meroxa?
- What problems are they trying to solve when they come to your platform?
Applications powered by real-time data were the exclusive domain of large and/or sophisticated tech companies for several years due to the inherent complexities involved. What are the shifts that have made them more accessible to a wider variety of teams?
- What are some of the remaining blockers for teams who want to start using real-time data?
With the democratization of real-time data, what are the new categories of products and applications that are being unlocked?
- How are organizations thinking about the potential value that those types of apps/services can provide?
With data flowing constantly, there are new challenges around oversight and accuracy. How does real-time data change the risk profile for applications that are consuming it?
- What are some of the technical controls that are available for organizations that are risk-averse?
What skills do developers need to be able to effectively design, develop, and deploy real-time data applications?
- How does this differ when talking about internal vs. consumer/end-user facing applications?
What are the most interesting, innovative, or unexpected ways that you have seen Meroxa used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Meroxa?
When is Meroxa the wrong choice?
What do you have planned for the future of Meroxa?

Contact Info

LinkedIn
@devarispbrown on Twitter

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Sponsored By:

Rudderstack: ![Rudderstack](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/CKNV8HZ6.png) RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. RudderStack’s warehouse-first approach means it does not store sensitive information, and it allows you to leverage your existing data warehouse/data lake infrastructure to build a single source of truth for every team. RudderStack also supports real-time use cases. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again. Visit [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack) to sign up for free today, and snag a free T-Shirt just for being a Data Engineering Podcast listener.

Support Data Engineering Podcast

Apr 16, 2023

00:49:19

Summary

Business intellingence has been chasing the promise of self-serve data for decades. As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. With the availability of AI powered by large language models combined with the evolution of semantic layers, the team at Zenlytic have taken aim at this problem again. In this episode Paul Blankley and Ryan Janssen explore the power of natural language driven data exploration combined with semantic modeling that enables an intuitive way for everyone in the business to access the data that they need to succeed in their work.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudderstack
Your host is Tobias Macey and today I'm interviewing Paul Blankley and Ryan Janssen about Zenlytic, a no-code business intelligence tool focused on emerging commerce brands

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what Zenlytic is and the story behind it?
Business intelligence is a crowded market. What was your process for defining the problem you are focused on solving and the method to achieve that outcome?
Self-serve data exploration has been attempted in myriad ways over successive generations of BI and data platforms. What are the barriers that have been the most challenging to overcome in that effort?
- What are the elements that are coming together now that give you confidence in being able to deliver on that?
Can you describe how Zenlytic is implemented?
- What are the evolutions in the understanding and implementation of semantic layers that provide a sufficient substrate for operating on?
- How have the recent breakthroughs in large language models (LLMs) improved your ability to build features in Zenlytic?
- What is your process for adding domain semantics to the operational aspect of your LLM?
For someone using Zenlytic, what is the process for getting it set up and integrated with their data?
Once it is operational, can you describe some typical workflows for using Zenlytic in a business context?
- Who are the target users?
- What are the collaboration options available?
What are the most complex engineering/data challenges that you have had to address in building Zenlytic?
What are the most interesting, innovative, or unexpected ways that you have seen Zenlytic used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Zenlytic?
When is Zenlytic the wrong choice?
What do you have planned for the future of Zenlytic?

Contact Info

Paul Blankley (LinkedIn)

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Sponsored By:

Rudderstack: ![Rudderstack](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/CKNV8HZ6.png) RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. RudderStack’s warehouse-first approach means it does not store sensitive information, and it allows you to leverage your existing data warehouse/data lake infrastructure to build a single source of truth for every team. RudderStack also supports real-time use cases. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again. Visit [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack) to sign up for free today, and snag a free T-Shirt just for being a Data Engineering Podcast listener.

Support Data Engineering Podcast

Apr 10, 2023

01:11:42

Summary

The customer data platform is a category of services that was developed early in the evolution of the current era of cloud services for data processing. When it was difficult to wire together the event collection, data modeling, reporting, and activation it made sense to buy monolithic products that handled every stage of the customer data lifecycle. Now that the data warehouse has taken center stage a new approach of composable customer data platforms is emerging. In this episode Darren Haken is joined by Tejas Manohar to discuss how Autotrader UK is addressing their customer data needs by building on top of their existing data stack.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudderstack
Your host is Tobias Macey and today I'm interviewing Darren Haken and Tejas Manohar about building a composable CDP and how you can start adopting it incrementally

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what you mean by a "composable CDP"?
- What are some of the key ways that it differs from the ways that we think of a CDP today?
What are the problems that you were focused on addressing at Autotrader that are solved by a CDP?
One of the promises of the first generation CDP was an opinionated way to model your data so that non-technical teams could own this responsibility. What do you see as the risks/tradeoffs of moving CDP functionality into the same data stack as the rest of the organization?
- What about companies that don't have the capacity to run a full data infrastructure?
Beyond the core technology of the data warehouse, what are the other evolutions/innovations that allow for a CDP experience to be built on top of the core data stack?
added burden on core data teams to generate event-driven data models
When iterating toward a CDP on top of the core investment of the infrastructure to feed and manage a data warehouse, what are the typical first steps?
- What are some of the components in the ecosystem that help to speed up the time to adoption? (e.g. pre-built dbt packages for common transformations, etc.)
What are the most interesting, innovative, or unexpected ways that you have seen CDPs implemented?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on CDP related functionality?
When is a CDP (composable or monolithic) the wrong choice?
What do you have planned for the future of the CDP stack?

Contact Info

Darren
- LinkedIn
- @DarrenHaken on Twitter
Tejas
- LinkedIn
- @tejasmanohar on Twitter

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Sponsored By:

Rudderstack: ![Rudderstack](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/CKNV8HZ6.png) RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. RudderStack’s warehouse-first approach means it does not store sensitive information, and it allows you to leverage your existing data warehouse/data lake infrastructure to build a single source of truth for every team. RudderStack also supports real-time use cases. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again. Visit [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack) to sign up for free today, and snag a free T-Shirt just for being a Data Engineering Podcast listener.

Support Data Engineering Podcast

Apr 03, 2023

01:01:57

Summary The data ecosystem has been building momentum for several years now. As a venture capital investor Matt Turck has been trying to keep track of the main trends and has compiled his findings into the MAD (ML, AI, and Data) landscape reports each year. In this episode he shares his experiences building those reports and the perspective he has gained from the exercise. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Businesses that adapt well to change grow 3 times faster than the industry average. As your business adapts, so should your data. RudderStack Transformations lets you customize your event data in real-time with your own JavaScript or Python code. Join The RudderStack Transformation Challenge today for a chance to win a $1,000 cash prize just by submitting a Transformation to the open-source RudderStack Transformation library. Visit dataengineeringpodcast.com/rudderstack (https://www.dataengineeringpodcast.com/rudderstack) today to learn more Your host is Tobias Macey and today I'm interviewing Matt Turck about his annual report on the Machine Learning, AI, & Data landscape and the insights around data infrastructure that he has gained in the process Interview Introduction How did you get involved in the area of data management? Can you describe what the MAD landscape report is and the story behind it? At a high level, what is your goal in the compilation and maintenance of your landscape document? What are your guidelines for what to include in the landscape? As the data landscape matures, how have you seen that influence the types of projects/companies that are founded? What are the product categories that were only viable when capital was plentiful and easy to obtain? What are the product categories that you think will be swallowed by adjacent concerns, and which are likely to consolidate to remain competitive? The rapid growth and proliferation of data tools helped establish the "Modern Data Stack" as a de-facto architectural paradigm. As we move into this phase of contraction, what are your predictions for how the "Modern Data Stack" will evolve? Is there a different architectural paradigm that you see as growing to take its place? How has your presentation and the types of information that you collate in the MAD landscape evolved since you first started it?~~ What are the most interesting, innovative, or unexpected product and positioning approaches that you have seen while tracking data infrastructure as a VC and maintainer of the MAD landscape? What are the most interesting, unexpected, or challenging lessons that you have learned while working on the MAD landscape over the years? What do you have planned for future iterations of the MAD landscape? Contact Info Website (https://mattturck.com/) @mattturck (https://twitter.com/mattturck) on Twitter MAD Landscape Comments Email (mailto:mad2023@firstmarkcap.com) Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ (https://www.pythonpodcast.com) covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast (https://www.themachinelearningpodcast.com) helps you go from idea to production with machine learning. Visit the site (https://www.dataengineeringpodcast.com) to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com (mailto:hosts@dataengineeringpodcast.com)) with your story. To help other people find the show please leave a review on Apple Podcasts (https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557) and tell your friends and co-workers Links MAD Landscape (https://mad.firstmarkcap.com) First Mark Capital (https://firstmark.com/) Bayesian Learning (https://en.wikipedia.org/wiki/Bayesian_inference) AI Winter (https://en.wikipedia.org/wiki/AI_winter) Databricks (https://www.databricks.com/) Cloud Native Landscape (https://landscape.cncf.io/) LUMA Scape (https://lumapartners.com/lumascapes/) Hadoop Ecosystem (https://www.analyticsvidhya.com/blog/2020/10/introduction-hadoop-ecosystem/) Modern Data Stack (https://www.fivetran.com/blog/what-is-the-modern-data-stack) Reverse ETL (https://medium.com/memory-leak/reverse-etl-a-primer-4e6694dcc7fb) Generative AI (https://generativeai.net/) dbt (https://www.getdbt.com/) Transform (https://transform.co/) Podcast Episode (https://www.dataengineeringpodcast.com/transform-co-metrics-layer-episode-206/) Snowflake IPO (https://www.cnn.com/2020/09/16/investing/snowflake-ipo/index.html) Dataiku (https://www.dataiku.com/) Iceberg (https://iceberg.apache.org/) Podcast Episode (https://www.dataengineeringpodcast.com/tabular-iceberg-lakehouse-tables-episode-363) Hudi (https://hudi.apache.org/) Podcast Episode (https://www.dataengineeringpodcast.com/hudi-streaming-data-lake-episode-209/) DuckDB (https://duckdb.org/) Podcast Episode (https://www.dataengineeringpodcast.com/duckdb-in-process-olap-database-episode-270/) Trino (https://trino.io/) Y42 (https://www.y42.com/) Podcast Episode (https://www.dataengineeringpodcast.com/y42-full-stack-data-platform-episode-295) Mozart Data (https://www.mozartdata.com/) Podcast Episode (https://www.dataengineeringpodcast.com/mozart-data-modern-data-stack-episode-242/) Keboola (https://www.keboola.com/) MPP Database (https://www.techtarget.com/searchdatamanagement/definition/MPP-database-massively-parallel-processing-database) The intro and outro music is from The Hug (http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug) by The Freak Fandango Orchestra (http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/) / CC BY-SA (http://creativecommons.org/licenses/by-sa/3.0/)

Mar 25, 2023

01:13:33

Summary

The promise of streaming data is that it allows you to react to new information as it happens, rather than introducing latency by batching records together. The peril is that building a robust and scalable streaming architecture is always more complicated and error-prone than you think it's going to be. After experiencing this unfortunate reality for themselves, Abhishek Chauhan and Ashish Kumar founded Grainite so that you don't have to suffer the same pain. In this episode they explain why streaming architectures are so challenging, how they have designed Grainite to be robust and scalable, and how you can start using it today to build your streaming data applications without all of the operational headache.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Businesses that adapt well to change grow 3 times faster than the industry average. As your business adapts, so should your data. RudderStack Transformations lets you customize your event data in real-time with your own JavaScript or Python code. Join The RudderStack Transformation Challenge today for a chance to win a $1,000 cash prize just by submitting a Transformation to the open-source RudderStack Transformation library. Visit dataengineeringpodcast.com/rudderstack today to learn more
Hey there podcast listener, are you tired of dealing with the headache that is the 'Modern Data Stack'? We feel your pain. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. It ends up being anything but that. Setting it up, integrating it, maintaining it—it’s all kind of a nightmare. And let's not even get started on all the extra tools you have to buy to get it to do its thing. But don't worry, there is a better way. TimeXtender takes a holistic approach to data integration that focuses on agility rather than fragmentation. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. If you're fed up with the 'Modern Data Stack', give TimeXtender a try. Head over to dataengineeringpodcast.com/timextender where you can do two things: watch us build a data estate in 15 minutes and start for free today.
Join in with the event for the global data community, Data Council Austin. From March 28-30th 2023, they'll play host to hundreds of attendees, 100 top speakers, and dozens of startups that are advancing data science, engineering and AI. Data Council attendees are amazing founders, data scientists, lead engineers, CTOs, heads of data, investors and community organizers who are all working together to build the future of data. As a listener to the Data Engineering Podcast you can get a special discount of 20% off your ticket by using the promo code dataengpod20. Don't miss out on their only event this year! Visit: dataengineeringpodcast.com/data-council today
Your host is Tobias Macey and today I'm interviewing Ashish Kumar and Abhishek Chauhan about Grainite, a platform designed to give you a single place to build streaming data applications

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what Grainite is and the story behind it?
What are the personas that you are focused on addressing with Grainite?
What are some of the most complex aspects of building streaming data applications in the absence of something like Grainite?
- How does Grainite work to reduce that complexity?
What are some of the commonalities that you see in the teams/organizations that find their way to Grainite?
What are some of the higher-order projects that teams are able to build when they are using Grainite as a starting point vs. where they would be spending effort on a fully managed streaming architecture?
Can you describe how Grainite is architected?
- How have the design and goals of the platform changed/evolved since you first started working on it?
What does your internal build vs. buy process look like for identifying where to spend your engineering resources?
What is the process for getting Grainite set up and integrated into an organizations technical environment?
- What is your process for determining which elements of the platform to expose as end-user features and customization options vs. keeping internal to the operational aspects of the product?
Once Grainite is running, can you describe the day 0 workflow of building an application or data flow?
- What are the day 2 - N capabilities that Grainite offers for ongoing maintenance/operation/evolution of those applications?
What are the most interesting, innovative, or unexpected ways that you have seen Grainite used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Grainite?
When is Grainite the wrong choice?
What do you have planned for the future of Grainite?

Contact Info

Ashish
- LinkedIn
Abhishek
- LinkedIn

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Sponsored By:

Support Data Engineering Podcast

Mar 19, 2023

00:51:38

Summary

As with all aspects of technology, security is a critical element of data applications, and the different controls can be at cross purposes with productivity. In this episode Yoav Cohen from Satori shares his experiences as a practitioner in the space of data security and how to align with the needs of engineers and business users. He also explains why data security is distinct from application security and some methods for reducing the challenge of working across different data systems.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Join in with the event for the global data community, Data Council Austin. From March 28-30th 2023, they'll play host to hundreds of attendees, 100 top speakers, and dozens of startups that are advancing data science, engineering and AI. Data Council attendees are amazing founders, data scientists, lead engineers, CTOs, heads of data, investors and community organizers who are all working together to build the future of data. As a listener to the Data Engineering Podcast you can get a special discount of 20% off your ticket by using the promo code dataengpod20. Don't miss out on their only event this year! Visit: dataengineeringpodcast.com/data-council today
RudderStack makes it easy for data teams to build a customer data platform on their own warehouse. Use their state of the art pipelines to collect all of your data, build a complete view of your customer and sync it to every downstream tool. Sign up for free at dataengineeringpodcast.com/rudder
Hey there podcast listener, are you tired of dealing with the headache that is the 'Modern Data Stack'? We feel your pain. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. It ends up being anything but that. Setting it up, integrating it, maintaining it—it’s all kind of a nightmare. And let's not even get started on all the extra tools you have to buy to get it to do its thing. But don't worry, there is a better way. TimeXtender takes a holistic approach to data integration that focuses on agility rather than fragmentation. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. If you're fed up with the 'Modern Data Stack', give TimeXtender a try. Head over to dataengineeringpodcast.com/timextender where you can do two things: watch us build a data estate in 15 minutes and start for free today.
Your host is Tobias Macey and today I'm interviewing Yoav Cohen about the challenges that data teams face in securing their data platforms and how that impacts the productivity and adoption of data in the organization

Interview

Introduction
How did you get involved in the area of data management?
Data security is a very broad term. Can you start by enumerating some of the different concerns that are involved?
How has the scope and complexity of implementing security controls on data systems changed in recent years?
- In your experience, what is a typical number of data locations that an organization is trying to manage access/permissions within?
What are some of the main challenges that data/compliance teams face in establishing and maintaining security controls?
- How much of the problem is technical vs. procedural/organizational?
As a vendor in the space, how do you think about the broad categories/boundary lines for the different elements of data security? (e.g. masking vs. RBAC, etc.)
- What are the different layers that are best suited to managing each of those categories? (e.g. masking and encryption in storage layer, RBAC in warehouse, etc.)
What are some of the ways that data security and organizational productivity are at odds with each other?
- What are some of the shortcuts that you see teams and individuals taking to address the productivity hit from security controls?
What are some of the methods that you have found to be most effective at mitigating or even improving productivity impacts through security controls?
- How does up-front design of the security layers improve the final outcome vs. trying to bolt on security after the platform is already in use?
- How can education about the motivations for different security practices improve compliance and user experience?
What are the most interesting, innovative, or unexpected ways that you have seen data teams align data security and productivity?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on data security technology?
What are the areas of data security that still need improvements?

Contact Info

Yoav Cohen

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Sponsored By:

Support Data Engineering Podcast

Mar 10, 2023

00:49:21

Summary

With the rise of the web and digital business came the need to understand how customers are interacting with the products and services that are being sold. Product analytics has grown into its own category and brought with it several services with generational differences in how they approach the problem. NetSpring is a warehouse-native product analytics service that allows you to gain powerful insights into your customers and their needs by combining your event streams with the rest of your business data. In this episode Priyendra Deshwal explains how NetSpring is designed to empower your product and data teams to build and explore insights around your products in a streamlined and maintainable workflow.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Join in with the event for the global data community, Data Council Austin. From March 28-30th 2023, they'll play host to hundreds of attendees, 100 top speakers, and dozens of startups that are advancing data science, engineering and AI. Data Council attendees are amazing founders, data scientists, lead engineers, CTOs, heads of data, investors and community organizers who are all working together to build the future of data. As a listener to the Data Engineering Podcast you can get a special discount of 20% off your ticket by using the promo code dataengpod20. Don't miss out on their only event this year! Visit: dataengineeringpodcast.com/data-council today!
RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudder
Your host is Tobias Macey and today I'm interviewing Priyendra Deshwal about how NetSpring is using the data warehouse to deliver a more flexible and detailed view of your product analytics

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what NetSpring is and the story behind it?
- What are the activities that constitute "product analytics" and what are the roles/teams involved in those activities?
When teams first come to you, what are the common challenges that they are facing and what are the solutions that they have attempted to employ?
Can you describe some of the challenges involved in bringing product analytics into enterprise or highly regulated environments/industries?
- How does a warehouse-native approach simplify that effort?
There are many different players (both commercial and open source) in the product analytics space. Can you share your view on the role that NetSpring plays in that ecosystem?
How is the NetSpring platform implemented to be able to best take advantage of modern warehouse technologies and the associated data stacks?
- What are the pre-requisites for an organization's infrastructure/data maturity for being able to benefit from NetSpring?
- How have the goals and implementation of the NetSpring platform evolved from when you first started working on it?
Can you describe the steps involved in integrating NetSpring with an organization's existing warehouse?
- What are the signals that NetSpring uses to understand the customer journeys of different organizations?
- How do you manage the variance of the data models in the warehouse while providing a consistent experience for your users?
Given that you are a product organization, how are you using NetSpring to power NetSpring?
What are the most interesting, innovative, or unexpected ways that you have seen NetSpring used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on NetSpring?
When is NetSpring the wrong choice?
What do you have planned for the future of NetSpring?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Sponsored By:

Support Data Engineering Podcast

Mar 06, 2023

00:45:44

Summary

The ecosystem for data professionals has matured to the point that there are a large and growing number of distinct roles. With the scope and importance of data steadily increasing it is important for organizations to ensure that everyone is aligned and operating in a positive environment. To help facilitate the nascent conversation about what constitutes an effective and productive data culture, the team at Data Council have dedicated an entire conference track to the subject. In this episode Pete Soderling and Maggie Hays join the show to explore this topic and their experience preparing for the upcoming conference.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Hey there podcast listener, are you tired of dealing with the headache that is the 'Modern Data Stack'? We feel your pain. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. It ends up being anything but that. Setting it up, integrating it, maintaining it—it’s all kind of a nightmare. And let's not even get started on all the extra tools you have to buy to get it to do its thing. But don't worry, there is a better way. TimeXtender takes a holistic approach to data integration that focuses on agility rather than fragmentation. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. If you're fed up with the 'Modern Data Stack', give TimeXtender a try. Head over to dataengineeringpodcast.com/timextender where you can do two things: watch us build a data estate in 15 minutes and start for free today.
Your host is Tobias Macey and today I'm interviewing Pete Soderling and Maggie Hays about the growing importance of establishing and investing in an organization's data culture and their experience forming an entire conference track around this topic

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what your working definition of "Data Culture" is?
- In what ways is a data culture distinct from an organization's corporate culture? How are they interdependent?
- What are the elements that are most impactful in forming the data culture of an organization?
What are some of the motivations that teams/companies might have in fighting against the creation and support of an explicit data culture?
- Are there any strategies that you have found helpful in counteracting those tendencies?
In terms of the conference, what are the factors that you consider when deciding how to group the different presentations into tracks or themes?
- What are the experiences that you have had personally and in community interactions that led you to elevate data culture to be it's own track?
What are the broad challenges that practitioners are facing as they develop their own understanding of what constitutes a healthy and productive data culture?
What are some of the risks that you considered when forming this track and evaluating proposals?
What are your criteria for determining whether this track is successful?
What are the most interesting, innovative, or unexpected aspects of data culture that you have encountered through developing this track?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on selecting presentations for this year's event?
What do you have planned for the future of this topic at Data Council events?

Contact Info

Pete
- @petesoder on Twitter
- LinkedIn
Maggie
- LinkedIn

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

Links

Data Council
- Podcast Episode
Data Community Fund
DataHub
- Podcast Episode
Database Design For Mere Mortals by Michael J. Hernandez (affiliate link)
SOAP
REST
Econometrics
DBA == Database Administrator
Conway's Law
dbt
- Podcast Episode

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Sponsored By:

TimeXtender: ![TimeXtender Logo](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/35MYWp0I.png) TimeXtender is a holistic, metadata-driven solution for data integration, optimized for agility. TimeXtender provides all the features you need to build a future-proof infrastructure for ingesting, transforming, modelling, and delivering clean, reliable data in the fastest, most efficient way possible. You can't optimize for everything all at once. That's why we take a holistic approach to data integration that optimises for agility instead of fragmentation. By unifying each layer of the data stack, TimeXtender empowers you to build data solutions 10x faster while reducing costs by 70%-80%. We do this for one simple reason: because time matters. Go to [dataengineeringpodcast.com/timextender](https://www.dataengineeringpodcast.com/timextender) today to get started for free!

Support Data Engineering Podcast

Feb 27, 2023

00:46:54

Summary

There has been a lot of discussion about the practical application of data mesh and how to implement it in an organization. Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Are you tired of dealing with the headache that is the 'Modern Data Stack'? We feel your pain. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. It ends up being anything but that. Setting it up, integrating it, maintaining it—it’s all kind of a nightmare. And let's not even get started on all the extra tools you have to buy to get it to do its thing. But don't worry, there is a better way. TimeXtender takes a holistic approach to data integration that focuses on agility rather than fragmentation. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. If you're fed up with the 'Modern Data Stack', give TimeXtender a try. Head over to dataengineeringpodcast.com/timextender where you can do two things: watch us build a data estate in 15 minutes and start for free today.
Your host is Tobias Macey and today I'm interviewing Jean-Georges Perrin about his work at PayPal to implement a data mesh and the role of data contracts in making it work

Interview

Introduction
How did you get involved in the area of data management?
Can you start by describing the goals and scope of your work at PayPal to implement a data mesh?
- What are the core problems that you were addressing with this project?
- Is a data mesh ever "done"?
What was your experience engaging at the organizational level to identify the granularity and ownership of the data products that were needed in the initial iteration?
What was the impact of leading multiple teams on the design of how to implement communication/contracts throughout the mesh?
What are the technical systems that you are relying on to power the different data domains?
- What is your philosophy on enforcing uniformity in technical systems vs. relying on interface definitions as the unit of consistency?
What are the biggest challenges (technical and procedural) that you have encountered during your implementation?
How are you managing visibility/auditability across the different data domains? (e.g. observability, data quality, etc.)
What are the most interesting, innovative, or unexpected ways that you have seen PayPal's data mesh used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on data mesh?
When is a data mesh the wrong choice?
What do you have planned for the future of your data mesh at PayPal?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Sponsored By:

TimeXtender: ![TimeXtender Logo](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/35MYWp0I.png) TimeXtender is a holistic, metadata-driven solution for data integration, optimized for agility. TimeXtender provides all the features you need to build a future-proof infrastructure for ingesting, transforming, modelling, and delivering clean, reliable data in the fastest, most efficient way possible. You can't optimize for everything all at once. That's why we take a holistic approach to data integration that optimises for agility instead of fragmentation. By unifying each layer of the data stack, TimeXtender empowers you to build data solutions 10x faster while reducing costs by 70%-80%. We do this for one simple reason: because time matters. Go to [dataengineeringpodcast.com/timextender](https://www.dataengineeringpodcast.com/timextender) today to get started for free!

Support Data Engineering Podcast

Feb 19, 2023

00:55:06

Summary

Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used. Projects like Apache Iceberg provide a viable alternative in the form of data lakehouses that provide the scalability and flexibility of data lakes, combined with the ease of use and performance of data warehouses. Ryan Blue helped create the Iceberg project, and in this episode he rejoins the show to discuss how it has evolved and what he is doing in his new business Tabular to make it even easier to implement and maintain.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Hey there podcast listener, are you tired of dealing with the headache that is the 'Modern Data Stack'? We feel your pain. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. It ends up being anything but that. Setting it up, integrating it, maintaining it—it’s all kind of a nightmare. And let's not even get started on all the extra tools you have to buy to get it to do its thing. But don't worry, there is a better way. TimeXtender takes a holistic approach to data integration that focuses on agility rather than fragmentation. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. If you're fed up with the 'Modern Data Stack', give TimeXtender a try. Head over to timextender.com/dataengineering where you can do two things: watch us build a data estate in 15 minutes and start for free today.
Your host is Tobias Macey and today I'm interviewing Ryan Blue about the evolution and applications of the Iceberg table format and how he is making it more accessible at Tabular

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what Iceberg is and its position in the data lake/lakehouse ecosystem?
- Since it is a fundamentally a specification, how do you manage compatibility and consistency across implementations?
What are the notable changes in the Iceberg project and its role in the ecosystem since our last conversation October of 2018?
Around the time that Iceberg was first created at Netflix a number of alternative table formats were also being developed. What are the characteristics of Iceberg that lead teams to adopt it for their lakehouse projects?
- Given the constant evolution of the various table formats it can be difficult to determine an up-to-date comparison of their features, particularly earlier in their development. What are the aspects of this problem space that make it so challenging to establish unbiased and comprehensive comparisons?
For someone who wants to manage their data in Iceberg tables, what does the implementation look like?
- How does that change based on the type of query/processing engine being used?
Once a table has been created, what are the capabilities of Iceberg that help to support ongoing use and maintenance?
What are the most interesting, innovative, or unexpected ways that you have seen Iceberg used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Iceberg/Tabular?
When is Iceberg/Tabular the wrong choice?
What do you have planned for the future of Iceberg/Tabular?

Contact Info

LinkedIn
rdblue on GitHub

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Sponsored By:

Acryl: ![Acryl](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/2E3zCRd4.png) The modern data stack needs a reimagined metadata management platform. Acryl Data’s vision is to bring clarity to your data through its next generation multi-cloud metadata management platform. Founded by the leaders that created projects like LinkedIn DataHub and Airbnb Dataportal, Acryl Data enables delightful search and discovery, data observability, and federated governance across data ecosystems. Signup for the SaaS product today at [dataengineeringpodcast.com/acryl](https://www.dataengineeringpodcast.com/acryl)

Support Data Engineering Podcast

Apr 24 | 00:45:26

Realtime Data Applications Made Easier With Meroxa

Apr 16 | 00:49:19

Building Self Serve Business Intelligence With AI And Semantic Modeling At Zenlytic

Apr 10 | 01:11:42

An Exploration Of The Composable Customer Data Platform

Apr 03 | 01:01:57

Mapping The Data Infrastructure Landscape As A Venture Capitalist

Mar 25 | 01:13:33

Unlocking The Potential Of Streaming Data Applications Without The Operational Headache At Grainite

Mar 19 | 00:51:38

Aligning Data Security With Business Productivity To Deploy Analytics Safely And At Speed

Mar 10 | 00:49:21

Use Your Data Warehouse To Power Your Product Analytics With NetSpring

Mar 06 | 00:45:44

Exploring The Nuances Of Building An Intentional Data Culture

Feb 27 | 00:46:54

Building A Data Mesh Platform At PayPal

Feb 19 | 00:55:06

The View Below The Waterline Of Apache Iceberg And How It Fits In Your Data Lakehouse

This podcast could use a review!

This podcast could use a review! Have anything to say about it? Share your thoughts using the button below.

Submit Review