KubeCon + CloudNative North America 2022

We spent over two years pouring through 800 page linux kernel performance books, tweaking obscure control plane settings, and developing detailed custom monitoring dashboards so you don’t have to! We found there is a large delta between what we learned in CKA training, and the layer upon layer of hard fought knowledge it takes run a large scale multi-tenant application in production. Join us as we take you through real world findings that took months of research to fully understand, and provide evidence that some of the things we were convinced were best practices, were the very things holding us back the most.

Securing a multi-tenant deployment for an enterprise is very challenging. Adobe built a scalable GitOps based application deployment solution for their individual teams using Argo projects. However, due to a lack of a standard solution for infrastructure automation across teams, enabling secure multi-tenant rollouts was a challenge. Adobe leveraged Crossplane in tandem with Argo to broker the provisioning of cloud resources consistently and across all teams. With this solution, Adobe and Amazon designed a layered isolation mechanism for tenant teams on top of existing shared Kubernetes clusters via a mix of technologies such as OPA Gatekeeper, ServiceAccount boundaries, IAM roles etc. This solved the non-negotiable requirements of security and multi-tenancy, which are hard to achieve natively with Crossplane and Argo. Interested? Join Adobe and Amazon engineers to hear their vision, architecture, challenges, solutions, and key takeaways.

Cluster life cycle management is a challenging task and Cluster API promises to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. With the growing adoption of CAPI in recent times, are you looking into writing your own Cluster API provider for your infrastructure (or) cloud with a unique set of requirements and constraints? We got you covered. This talk will walk you through our journey as a Cluster API provider and all the lessons learned the hard way.

- Building blocks to implement Cluster API provider, and bare essentials like CSI & CPI.
- What are the common patterns around developing and debugging workflows?
- How to enable multi-version API support via webhooks?
- How to address common problems like multi-tenancy, and user quota management in a strong multitenant cloud environment with Enterprise customers?
- How to leverage CAPI in building Kubernetes as a Service layer on your clouds.

Come and learn from the maintainers of a Cluster API infrastructure provider - "our journey around moving from handcrafted Kubernetes life cycle management to Cluster API based life cycle management in a multitenant cloud".

Kubernetes is generally considered a single-tenant container orchestrator, but as companies have been running it and realizing the benefits of the Kubernetes architecture contrasted with the nontrivial level-of-effort of managing many single tenant clusters we’ve seen a spike in use cases & projects that support the need for multi-tenant & zero-trust deployments. You can see this in the growth of “Sandboxed Runtimes” like Kata, gVisor & Firecracker. As well as tools like vCluster, Kamaji & HNC. In this talk Chris Hein & Eric Ernst will demonstrate one way hard multi-tenancy can be achieved by leveraging Cluster API Nested with VirtualCluster running inside a Kubernetes cluster with workload isolation & virtual networking being provided by the Kata runtime. Users of this architecture get the benefits of per-tenant Kubernetes control planes to use CRDs, Admission Webhooks, Cluster level RBAC, Aggregate APIServers along with workload & network segregation while reducing the overall maintenance burden. Modeled after the ICDCS paper by folks from Alibaba - https://bit.ly/3tfnWnA If you are interested in sandboxed runtimes, hard multi-tenancy, scaling Kubernetes, Cluster API or multi-cluster Kubernetes this is the talk for you.

While self-service clusters are desirable, there are many cloud resources that need to be created for a cluster. In an enterprise, these may fall under a different team’s responsibilities. So, how does a cloud or infrastructure team provide the necessary guardrails to ensure that the Kubernetes environments created by developers are compliant with the organization’s security, governance, and cost management standards? In this talk, Dolis shares an approach where Crossplane and Kyverno, both CNCF projects, can be used to provide self-service Kubernetes environments on the cloud for developers with necessary checks and restrictions in place. While Crossplane, an increasingly popular IaC orchestrator running using Kubernetes, is used to provision different infrastructure resources, Kyverno can be utilized to provide governance on what type of resources can be created, by whom, and how the resources are configured. We can automate resource provisioning with governance using Crossplane and Kyverno. In addition to deploying and managing cloud resources, you can also create Kyerno policies to ensure that the generated resources are compliant with your company’s requirements.

In 2014, namespaces were added to Kubernetes. Many tried to implement multi-tenancy on-top, with limited success. What if namespaces are just the wrong tool, and we better invest into cluster-like isolation called workspaces, built deeply into the apiserver. The kcp project explors Kubernetes - with logical cluster support to implement workspaces - with ability to scale horizontally via sharding, towards 1,000,000 clusters - with novel API service models disrupting CRDs. In contrast to other projects like vcluster or OpenClusterManager, kcp challenges years old decisions in Kubernetes by going deep into API-Machinery and apiserver. Strategically, we reduce the size of clusters to those of namespaces, and by that open up the space between workspaces for innovation, while within a workspace kcp is just Kubernetes. Outline: 1. from namespaces to workspaces 2. APIExport and APIBindings, identity based security 3. scaling up kcp to 1,000,000 workspaces.

Now that many people are deploying Kubernetes in production, they all have the same question: how do you manage multiple Kubernetes clusters? In this session, we’ll chat about the new CNCF Sandbox project Open Cluster Management (https://open-cluster-management.io) and how it can help you simplify multicluster container orchestration. Open APIs are evolving within the project for cluster registration, work distribution, dynamic placement of policies and workloads, and much more. Attendees will learn how they can use Open Cluster Management to take control of their sprawling infrastructure.

Argo Workflows and Argo CD are powerful tools, but unifying them under a multi-tenant experience is necessary to run at scale across multiple teams in any large organization. Argo Workflows and Argo CD use different approaches to RBAC and both have different security considerations and available security features. We at Ethos, the Adobe Cloud Platform, have designed an architecture to create a secure multi-tenant CI/CD experience for our developer teams. Join our talk to learn how we achieved multi-tenancy through the isolation of each component of our developer CI/CD workflows, such as building, scanning, pushing, workflow artifacts, workflow secrets, as well as the restriction of application deployment with Argo CD AppProjects and RBAC.