Later this month, Nathan Steiner, the Director of Field Engineering, ANZ, at Databricks, will give a presentation at the Data Engineering Summit. There he will talk about the “habits” of data-driven organisations, and the importance of an open architecture that combines the best elements of data lakes and data warehouses.

Steiner kindly appeared on this episode of the Data Futurology podcast to talk about this, and further discuss the Databricks vision for data-driven workspaces.

“Historically, you look at data engineers, data analysts, AI, machine learning and data scientists, they were focused on different types of data, so you had your data engineers focused on your siloed and disparate ADW enterprise data warehousing, relational database structured systems, and you had your data scientists looking at predominantly real time data,” he says during the wide-ranging conversation.

The solution, to Steiner’s and Databricks’ vision, is bringing those data resources together and making for a more collaborative data environment. “It’s more pragmatic and effective for these job roles to be working from a single uniform platform,” he says.

As Steiner notes during the conversation, the personalisation that is so important to modern business is driven from being able to make the data resources collaborative. He highlights the example of a financial services company that wants to be able to issue credit within five minutes from an application via a smartphone. “In the back end, it's AI, and ML that is doing the credit risk assessment frameworks of that particular individual and creating that value customer experience,” he says.

Finally, Steiner considers the governance implications of the Databricks lakehouse, and the advantages of having a uniform and unified approach when it comes to governance.

For more insights on breaking down data silos and unifying data teams, be sure to tune in to the podcast!

Enjoy the show!

Learn more about Databricks

Learn more about Nathan Steiner

Thank you to you our sponsor, Talent Insights Group!

Read the full podcast episode summary here.