Using Apache Kafka as Cloud-Native Data System ft. Gwen Shapira

Gwen leads the Cloud-Native Kafka team, which focuses on developing new features to evolve Kafka to its next stage as a fully managed cloud data platform. Turning Kafka into a self-service platform is not entirely straightforward, however, Kafka’s early day investment in elasticity, scalability, and multi-tenancy to run at a company-wide scale served as the North Star in taking Kafka to its next stage— a fully managed cloud service where users will just need to send us their workloads and everything else will magically work. Through examining modern cloud-native data services, such as Aurora, Amazon S3, Snowflake, Amazon DynamoDB, and BigQuery, there are seven capabilities that you can expect to see in modern cloud data systems, including:
1. Elasticity: Adapt to workload changes to scale up and down with a click or APIs—cloud-native Kafka omits the requirement to install REST Proxy for using Kafka APIs
2. Infinite scale: Kafka has the ability to elastic scale with a behind-the-scene process for capacity planning
3. Resiliency: Ensures high availability to minimize downtown and disaster recovery
4. Multi-tenancy: Cloud-native infrastructure needs to have isolations—data, namespaces, and 3. performance, which Kafka is designed to support
5. Pay per use: Pay for resources based on usage
6. Cost-effectiveness: Cloud deployment has notably lower costs than self-managed services, which also decreases adoption time
7. Global: Connect to Kafka from around the globe and consume data locally

Building around these key requirements, a fully managed Kafka as a service provides an enhanced user experience that is scalable and flexible with reduced infrastructure management costs. Based on their experience building cloud-native Kafka, Gwen and her team published a four-part thesis that shares insights on user expectations for modern cloud data services as well as technical implementation considerations to help you develop your own cloud-native data system.

EPISODE LINKS

ABOUT CONFLUENT

#cloudnative #serverless #apachekafka #kafka #confluent