KubeCon + CloudNative North America 2022

In this talk, we’ll tell the story of how we built our very own eventually consistent system which is currently deployed in production clusters across the US, Germany, and Singapore -- including all the mistakes we made along the way. We’ll walk through how we leveraged tools like gRPC, Kubernetes, LevelDB, and Prometheus to implement two new open source projects that serve as the heart of our system. We’ll also confess all the ways we messed up during the process — from struggling to debug protocol buffer errors, to tangling up send and receive goroutines, to reasoning about the phases of replication. It won’t all be pretty, but we hope you’ll benefit from the lessons we learned, including the most important lesson — that you *can* build your own distributed system. We’ll close out by talking about why rolling our own system (in spite of all the headaches and mistakes) made sense for our use case, and why it might also make sense for you. Attendees will walk away with a hearty introduction to distributed systems concepts, as well as a to-do list of things they can investigate in their own systems to determine how they might be able to reduce concurrency-related bugs and/or consistency-related costs, improve maintenance, and reach more daily active users around the world.