Summary

Kafka has become a de facto standard interface for building decoupled systems and working with streaming data. Despite its widespread popularity, there are numerous accounts of the difficulty that operators face in keeping it reliable and performant, or trying to scale an installation. To make the benefits of the Kafka ecosystem more accessible and reduce the operational burden, Alexander Gallego and his team at Vectorized created the Red Panda engine. In this episode he explains how they engineered a drop-in replacement for Kafka, replicating the numerous APIs, that can scale more easily and deliver consistently low latencies with a much lower hardware footprint. He also shares some of the areas of innovation that they have found to help foster the next wave of streaming applications while working within the constraints of the existing Kafka interfaces. This was a fascinating conversation with an energetic and enthusiastic engineer and founder about the challenges and opportunities in the realm of streaming data.

Announcements

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• What are the pieces of advice that you wish you had received early in your career of data engineering? If you hand a book to a new data engineer, what wisdom would you add to it? I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. Go to dataengineeringpodcast.com/97things to add your voice and share your hard-earned expertise.
• When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $60 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
• Are you bogged down by having to manually manage data access controls, repeatedly move and copy data, and create audit reports to prove compliance? How much time could you save if those tasks were automated across your cloud platforms? Immuta is an automated data governance solution that enables safe and easy data analytics in the cloud. Our comprehensive data-level security, auditing and de-identification features eliminate the need for time-consuming manual processes and our focus on data and compliance team collaboration empowers you to deliver quick and valuable data analytics on the most sensitive data to unlock the full potential of your cloud data platforms. Learn how we streamline and accelerate manual processes to help you derive real results from your data at dataengineeringpodcast.com/immuta.
• If you’re looking for a way to optimize your data engineering pipeline – with instant query performance – look no further than Qubz. Qubz is next-generation OLAP technology built for the scale of Big Data from UST Global, a renowned digital services provider. Qubz lets users and enterprises analyze data on the cloud and on-premise, with blazing speed, while eliminating the complex engineering required to operationalize analytics at scale. With an emphasis on visual data engineering, connectors for all major BI tools and data sources, Qubz allow users to query OLAP cubes with sub-second response times on hundreds of billions of rows. To learn more, and sign up for a free demo, visit dataengineeringpodcast.com/qubz.
• You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data platforms. For more opportunities to stay up to date, gain new skills, and learn from your peers there are a growing number of virtual events that you can attend from the comfort and safety of your home. Go to dataengineeringpodcast.com/conferences to check out the upcoming events being offered by our partners and get registered today!
• Your host is Tobias Macey and today I’m interviewing Alexander Gallego about his work at Vectorized building Red Panda as a performance optimized, drop-in replacement for Kafka

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you start by describing what Red Panda is and what motivated you to create it?
• What are the limitations of Kafka that make something like Red Panda necessary?
• What are the current strengths of the Kafka ecosystem that make it a reasonable implementation target for Red Panda?
• How is Red Panda architected?
  • How has the design or direction changed or evolved since you first began working on it?
• What are the challenges that you face in automatically optimizing the runtime to take advantage of the hardware that it is deployed on?
  • How do cloud environments contribute to that complexity?
• How are you handling the compatibility layer for the Kafka API?
  • What is your approach for managing versioning and ensuring that you maintain bug compatibility?
• Beyond performance, what other areas of innovation or improvement in the capabilities and experience do you see while adhering to the Kafka protocol?
• What are the opportunities for innovation in the streaming space that aren’t being explored yet?
• What are some of the most interesting, innovative, or unexpected ways that you have seen Redpanda being used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while building Red Panda and Vectorized?
• When is Red Panda the wrong choice?
• What do you have planned for the future of the product and business?
• What is your Hack The Planet diversity scholarship?

Contact Info

• @emaxerrno on Twitter
• LinkedIn

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

• Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
• To help other people find the show please leave a review on iTunes and tell your friends and co-workers
• Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat

Links

• Vectorized

  • Free Download Trial

  • @vectorizedio Company Twitter Accn’t

  • Community Slack

• acquires-data-processing-provider-concord-systems.jsp?utm_source=rss&utm_medium=rss">Concord alternative to Flink

• Apache Flink

  • Podcast Episode

• FAANG == Facebook, Apple, Amazon, Netflix, and Google

• Blackblaze

• Raft

• NATS

• Pulsar

  • Podcast Episode
  • StreamNative Podcast Episode

• Open Messaging Specification

• ScyllaDB

• CockroachDB

• MemSQL

• WASM == Web Assembly

• Debezium

  • Podcast Episode

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA