Summary

Working with unstructured data has typically been a motivation for a data lake. The challenge is imposing enough order on the platform to make it useful. Kirk Marple has spent years working with data systems and the media industry, which inspired him to build a platform for automatically organizing your unstructured assets to make them more valuable. In this episode he shares the goals of the Unstruk Data Warehouse, how it is architected to extract asset metadata and build a searchable knowledge graph from the information, and the myriad ways that the system can be used. If you are wondering how to deal with all of the information that doesn’t fit in your databases or data warehouses, then this episode is for you.

Announcements

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
• Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch.
• Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
• Your host is Tobias Macey and today I’m interviewing Kirk Marple about Unstruk Data, a company that is building a data warehouse for unstructured data that ofers automated data preparation via metadata enrichment, integrated compute, and graph-based search

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you describe what Unstruk Data is and the story behind it?
• What would you classify as "unstructured data"?
  • What are some examples of industries that rely on large or varied sets of unstructured data?
  • What are the challenges for analytics that are posed by the different categories of unstructured data?
• What is the current state of the industry for working with unstructured data?
  • What are the unique capabilities that Unstruk provides and how does it integrate with the rest of the ecosystem?
  • Where does it sit in the overall landscape of data tools?
• Can you describe how the Unstruk data warehouse is implemented?
  • What are the assumptions that you had at the start of this project that have been challenged as you started working through the technical implementation and customer trials?
  • How has the design and architecture evolved or changed since you began working on it?
• How do you handle versioning of data, given the potential for individual files to be quite large?
• What are some of the considerations that users should have in mind when modeling their data in the warehouse?
• Can you talk through the workflow of ingesting and analyzing data with Unstruk?
  • How do you manage data enrichment/integration with structured data sources?
• What are the most interesting, innovative, or unexpected ways that you have seen the technology of Unstruk used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on and with the Unstruk platform?
• When is Unstruk the wrong choice?
• What do you have planned for the future of Unstruk?

Contact Info

• LinkedIn
• @KirkMarple on Twitter

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

• Unstruk Data
• TIFF
• ROSBag
• HDF5
• Media/Digital Asset Management
• Data Mesh
• SAN
• NAS
• Knowledge Graph
• Entity Extraction
• OCR (Optical Character Recognition)
• Cloud Native
• Cosmos DB
• Azure Functions
• Azure EventHub
• Azure Cognitive Search
• GraphQL
• KNative
• Schema.org
• Pinecone Vector Database
  • Podcast Episode
• Dublin Core Metadata Initiative
• Knowledge Management

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA