What Is Least Privilege in Kubernetes and Why Does It Matter for Secure Deployments?

Many things in security tend to come down to the same few concepts, one of the most important being to limit the surface area of attack. The principle of least privilege is the idea that we should design our operations to use only the minimal set of permissions required to accomplish the task. When deploying applications in Kubernetes, this usually means making use of multiple deployment identities for the deployment of different resources, each with its own scope and permission set. (This concept applies to running applications as well, but that's a discussion for a different time.)

Why Developers Over-Permission Kubernetes Pipelines

Many Kubernetes clusters require deploying prerequisites that may need to create cluster-level resources, such as CustomResourceDefinitions or perhaps even ClusterRole and ClusterRoleBinding objects. These types of resources typically require a cluster administrator (or RBAC-equivalent) account to deploy because they have a broad effect across the entire cluster. For convenience, many teams may be tempted to use this elevated permission set to deploy everything to the cluster. After all, who wants to deal with two (or more!) service principals in their pipelines? That's just too much work, when there are deadlines to be met!

The Risk of Using Elevated Permissions for All Kubernetes Deployments

The danger here is that the more things you deploy, the less certainty you'll generally have of the content. Let's consider an example.

Kubernetes Deployment Case Study: Security Pitfalls at Agnostica

Setting Up MySQL with Cluster-Level Permissions

The dev team at Agnostica has decided to run their next-gen, micro-service-based shopping platform atop Kubernetes. As part of the deployment, they've decided that they'll make use of an operator for running MySQL in Kubernetes, deploying it via its public Helm chart. Operators are a useful construct for many sorts of Kubernetes, but they generally rely on the ability to deploy CRDs and other cluster-level resources to provide their services, so you should always review these deployments to make sure they meet your security and compliance requirements before deploying them to your cluster.

Using Cluster Admin Accounts for All Deployments

The Agnostica team decide to build a pipeline in their favorite devops tool. They know they need to give cluster-level permissions to the pipeline’s identity in order to deploy the CRDs and other resources required by the MySQL operator chart, so they configure that, add their helm upgrade -i ... task to the pipeline and are off to the races. Success! They've now got a way to easily manage MySQL database instances within their cluster. Because the operator doesn't change very often, this pipeline doesn't run very much. Now, it's time to create the pipeline to deploy their application.

RBAC Shortcuts and Pipeline Reuse

Frank, the lead engineer at Agnostica, figures that there's no need to create another service account to deploy the application. After all, they've already configured the first one, and it's got all the access (and more!) that it needs to get their application deployed. Why spend the time configuring another? Instead, he creates the second pipeline to deploy their app using the same identity as the prerequisites pipeline.

Helm Charts, Stack Overflow, and a Whole Lot of YAML

Because their app consists of a few dozen microservices, the Helm charts they use for it consist of quite a few files. There are even a few other Helm charts that they use from other sources to deploy some additional dependencies.

There is So. Much. YAML.

Frank and team continue over the following weeks to create all the pipeline steps, kubectl apply -f-ing and helm upgrade -i-ing all sorts of things, copying examples from StackOverflow and using AI to generate code they don’t fully understand.

Finally, they get the application successfully deployed. Hurrah! The new platform is a success. Profits are up 250%. The team revels in their herculean accomplishment.

How an Insider Threat Exploited Kubernetes Secrets

A New Engineer With Deep Kubernetes Knowledge

But now there's more work to do. Features to develop, reports to create; a developer's job is never done. So the team hires a few more engineers to tackle the work, including a particularly promising engineer named Scott who's got 20 years of production Kubernetes experience under his belt[1]. These engineers are given access to the code & repository for just the application deploy, since that's where their job focus is.

Malicious Webhook Injection via PR

Scott, as it turns out, isn't the most trustworthy guy around. But he does know a thing or two about Kubernetes that his less erudite teammates do. So, while he's creating a pull request to handle some configuration changes in the various Helm charts for the application, he decides to slip the following file into the mix.

Security Labels That Obscure Risk

Frank reviews Scott's PR. At first, he questions the ValidatingWebhookConfiguration resource, having only seen things like Deployment and Service resources and the like. But Scott's the Kubernetes expert, and to a novice, anything labeled as "security" must be a good thing, right?[2] So the PR gets approved, and later deployed to production via pipeline.

Kubernetes Security Breach: What Went Wrong

A Clean Exit and a Quiet Attack

A few weeks later, Scott gets an offer from another company and leaves the team. Jim, the project manager, throws a farewell party as a thanks for all Scott has contributed to the team. They'd never have made it to version 2.0 of the platform without his expertise in Kubernetes. Cake is eaten, handshakes and high-fives are exchanged, and Scott drives off at the end of the day.

Secrets Exfiltrated via Webhook

...you probably see where this is heading.

A few weeks later, the team notices that some of their payment processor accounts have some discrepancies. Customers are calling in with complaints. Some of their SaaS based accounts seem to be compromised. In short, everything has gone wrong. So what happened?

As it turns out, Scott was the owner of the validateya.ml domain in the ValidatingWebhookConfiguration, and that resource has the effect of sending the entire contents of any Secret object created or updated in the cluster to the https://agnostica.validateya.ml/validator URL. All he had to do was sit back and wait for the secrets to all manner of resources came rolling in, at which point he could compromise services at his leisure, wreaking all sorts of havoc against his former employer.

Understanding ValidatingWebhookConfiguration in Kubernetes

ValidatingWebhookConfiguration resources are not intrinsically unsafe. Most often, they're used to validate policy against incoming resources and are usually deployed targeting resources running inside the cluster rather than external resources like we've shown here. OPA Gatekeeper and Kyverno are two such tools that make use of these configuration resources, and to great effect.

Kubernetes Supply Chain Attacks: Risks of Third-Party Helm Charts

The danger of insider threat is not the only way such a vulnerability could manifest either. Are you certain that you can trust any externally sourced applications or Helm charts that you deploy alongside your app? (Think databases, logging utilities, service meshes, etc.) What if they got unknowingly compromised? Supply chain attacks are certainly an additional risk here, which is why I'll reiterate my stance that "the more things you deploy, the less certainty you'll generally have about the content", especially when using accounts with elevated privileges to do the deployments.

Try It Yourself: Demonstrating Secret Exfiltration in Kubernetes

If you'd like to try something similar for yourself[3], head on over to PipeDream's RequestBin to create a unique URL that you can use to view incoming requests via your browser. Then change lines 6 and 17 in the YAML above to include that custom URL. Once done, use the following command to deploy a sample secret to your cluster (or create your own).

kubectl create secret generic db-user-pass --from-literal=username=admin --from-literal=password='S!B\*d$zDsb='

Viewing and Decoding the Exfiltrated Data

You should be able to see the request come in, which will look something like this:

Decode the two data objects from base64 to roundtrip the contents of the secret.

echo 'YWRtaW4=' | base64 -d  # returns admin

echo 'UyFCXCpkJHpEc2I9' | base64 -d  # returns S!B\*d$zDsb=

How to Secure Your Kubernetes CI/CD Pipelines

The takeaway from this story should be this: be certain to critically review anything that gets deployed using a cluster admin level account. Limit the resources that you deploy with it and use less privileged accounts with “just enough” RBAC privileges assigned to get the job done for the rest of your deployments. There are other mitigations that can assist these scenarios as well, most notably restricting network egress from the cluster using a firewall or other network security policy and monitoring for anomalous behaviors.

As with all things security related, vigilance remains paramount. As engineers, we are accountable for everything that gets deployed to production, whether created by us or not.

[1] perceptive readers will have already done the math here. Kubernetes 1.0 was released on July 21, 2015...

[2] another fun naming strategy might be to call this resource the secrets-disaster-recovery-plan

[3] …on a non-production cluster, preferably!

Sources:

What is the Principle of Least Privilege (POLP)? | Definition from TechTarget

Extend the Kubernetes API with CustomResourceDefinitions | Kubernetes

Pipedream | Connect APIs, AI, databases and more

About the Author

Brandon Sharp

Brandon Sharp is a lead Cloud Architect at CloudFit and brings over 20 years of software development, cloud architecture, and automation experience to the team. He leads the engineering efforts for our Managed Workload product group and is passionate about evangelizing best practices for securely building and deploying workloads in the cloud