KubeCon + CloudNative North America 2022

prometheus-operator is the de facto standard for running Prometheus on Kubernetes. Yet, its configuration can be complicated and baroque, making it hard to know what is being scraped, or to properly enforce RBAC. Scaling also requires careful thought. However, there are an increasing number of ways to run Prometheus as “stateless”. How can we adopt this to solve these problems? This talk introduces an alternative, operator-based approach for running stateless Prometheus instances on Kubernetes by leveraging Prometheus as a node agent. This prompted rethinking how Prometheus configuration is done today, and led to new, simpler, and more opinionated CRDs. We will discuss trade-offs in the new configuration model and the challenges of running a fleet of node-agent Prometheuses at scale. The hope is this lowers the barrier to entry of managing Prometheus infrastructure, while still supporting features and access controls for enterprise users.