RBAC That Doesn't Drive Your Team Crazy
Kubernetes RBAC is powerful and also frustrating. The API is straightforward — Roles grant permissions, RoleBindings assign Roles to subjects. But designing RBAC for an organization where different teams need different access levels, where developers need enough permissions to debug but not enough to break production, and where security auditors want to verify who can do what — that's where it gets complicated.
I've redesigned RBAC at three organizations, and the biggest lesson is: start restrictive and add permissions based on actual needs, not anticipated needs. Every permission you grant is one you'll eventually need to audit.
The Three-Tier Access Model
Most organizations need three levels of access. Platform administrators who can do everything across all namespaces — they manage the cluster itself, install operators, and handle infrastructure. Team leads who have broad access within their team's namespaces but can't affect other teams or cluster-level resources. And developers who can deploy and debug within their namespaces but can't modify RBAC, create namespaces, or access secrets directly.
Platform Admin Role
Platform admins get a ClusterRole that grants broad access. I don't use the built-in cluster-admin ClusterRole because it includes permissions that even platform admins shouldn't use casually — like impersonation and escalation. Instead, I create a custom ClusterRole:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: platform-admin
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["*"]
- apiGroups: ["apps", "batch", "networking.k8s.io", "policy"]
resources: ["*"]
verbs: ["*"]
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["roles", "rolebindings"]
verbs: ["*"]
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["clusterroles", "clusterrolebindings"]
verbs: ["get", "list", "watch"]Notice that platform admins can create namespace-scoped Roles and RoleBindings but can only read ClusterRoles and ClusterRoleBindings. Creating cluster-level RBAC resources should go through a reviewed pull request, not a kubectl command. This is a deliberate friction point that has prevented several "someone accidentally granted cluster-admin to a service account" incidents.
Team Lead Role
Team leads get a namespace-scoped Role that lets them manage workloads, view logs, and manage team-level RBAC:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: team-lead
namespace: team-alpha
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "pods/exec", "services", "configmaps", "persistentvolumeclaims", "events"]
verbs: ["*"]
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
verbs: ["*"]
- apiGroups: ["batch"]
resources: ["jobs", "cronjobs"]
verbs: ["*"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses", "networkpolicies"]
verbs: ["*"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["roles", "rolebindings"]
verbs: ["get", "list", "watch", "create", "update"]
- apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
verbs: ["*"]Team leads can create Roles and RoleBindings within their namespace. This lets them onboard new team members without filing a platform team ticket. The boundary is that they can't grant permissions they don't have — Kubernetes enforces this through the escalation prevention built into the RBAC API.
Developer Role
Developers can deploy, debug, and view resources but can't modify RBAC, access secrets directly, or exec into pods in production:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: developer
namespace: team-alpha
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "services", "configmaps", "events", "persistentvolumeclaims"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"] # allowed in dev, blocked in prod via admission control
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "statefulsets"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["apps"]
resources: ["deployments/rollback"]
verbs: ["create"]
- apiGroups: ["batch"]
resources: ["jobs", "cronjobs"]
verbs: ["get", "list", "watch", "create", "delete"]Developers can trigger rollbacks (deployments/rollback) because during an incident, you want the person closest to the code to be able to revert. They can't create Deployments from scratch (only update existing ones), which means all new services go through the GitOps workflow.
Service Account RBAC
Application service accounts should follow the principle of least privilege more strictly than human accounts. A pod that needs to read ConfigMaps shouldn't have permission to create Deployments. I create a dedicated service account and Role for each application:
apiVersion: v1
kind: ServiceAccount
metadata:
name: api-server
namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: api-server-role
namespace: production
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "watch"]
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["api-server-secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: api-server-binding
namespace: production
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: api-server-role
subjects:
- kind: ServiceAccount
name: api-server
namespace: productionNote the resourceNames field on the secrets rule. This limits the service account to reading only the specific secret it needs, not every secret in the namespace. Without this, a compromised pod could read every team's database credentials.
RBAC Audit and Compliance
Kubernetes audit logs capture every API request, including who made it and what RBAC rules allowed it. But raw audit logs are overwhelming. The practical approach is to focus on two things: who has elevated permissions, and who exercised those permissions.
To enumerate effective permissions for a user:
$ kubectl auth can-i --list --as=developer@company.com -n production
Resources Non-Resource URLs Resource Names Verbs
pods [] [] [get list watch]
pods/log [] [] [get list watch]
deployments.apps [] [] [get list watch update patch]
deployments.apps/rollback [] [] [create]
...Run this periodically for all users and diff against the previous run. Any unexpected permission change gets investigated. I script this into a weekly job that sends a Slack notification if the permission set for any user changes.
For compliance reporting, the kubectl-who-can plugin answers the inverse question — "who can delete pods in production?":
$ kubectl-who-can delete pods -n production
ROLEBINDING NAMESPACE SUBJECT TYPE
team-lead-binding production alice@company User
platform-admin-bind production platform-admins GroupThis output is what auditors want to see. It maps permissions to specific individuals, which is the basis for access reviews.
Group-Based vs Individual Bindings
Bind Roles to groups, not individuals. If you're creating a RoleBinding for every developer who joins the team, you're doing it wrong. Use group subjects that map to your identity provider's groups. When someone joins or leaves the team, you update the group membership in your identity provider (Okta, Azure AD, Google Workspace). Kubernetes RBAC doesn't change. This scales much better than individual bindings and makes access reviews straightforward - you audit group memberships, not Kubernetes RoleBindings.
Temporary Access Elevation
Developers occasionally need elevated access - debugging a production issue, inspecting secrets, running a one-time maintenance task. The worst approach is permanently granting broader permissions. The second worst is having a shared "break glass" credential.
A better approach is time-limited access elevation. Tools like kubectl-sudo or commercial platforms like Teleport provide just-in-time access that expires automatically. If you can't adopt a tool, the manual version is a separate RoleBinding with a short TTL, created and deleted via a script. It's not perfect - the cleanup is manual - but it's better than permanent elevated access. Pair it with an alert that fires if the elevated binding exists for longer than the intended duration.
Preventing Privilege Escalation
Kubernetes has built-in escalation prevention: a user can't create a RoleBinding that grants permissions they don't already have. But there are edge cases. A user with permission to create pods in a namespace can mount the namespace's service account tokens and use those tokens to make API calls with the service account's permissions.
Mitigate this by ensuring service accounts don't have unnecessary permissions. The default service account in each namespace gets no permissions by default (since Kubernetes 1.24 stopped auto-mounting tokens). But many applications create service accounts with broad permissions for convenience. Audit your service accounts periodically to find any with cluster-admin or broad permissions that aren't actively needed.
RBAC for CI/CD Service Accounts
CI/CD pipelines need Kubernetes access to deploy applications, but they shouldn't have broad cluster access. Create a dedicated service account per team with permissions scoped to only their namespace and only the resources they deploy. The CI/CD service account should be able to update existing Deployments and create Jobs (for database migrations), but it can't create new Deployments, modify RBAC, or access Secrets. New services still go through the GitOps workflow where Argo CD creates the initial resources.
Rotate CI/CD tokens regularly. Kubernetes 1.24+ supports time-limited tokens via the TokenRequest API - create a token that expires in 1 hour and regenerate it at the start of each pipeline run, rather than using a long-lived token stored in your CI system's secrets.
Namespace-Level RBAC Templates
When onboarding a new team, you shouldn't be manually creating Roles and RoleBindings. Use a namespace provisioning controller (or a simple script triggered by a Git PR) that stamps out the standard RBAC configuration for each new namespace. The template includes the three-tier Roles (platform-admin, team-lead, developer), RoleBindings to the appropriate identity provider groups, a CI/CD service account with deploy-only permissions, and the associated ResourceQuotas and LimitRanges.
Store these templates in your GitOps config repo. When a team requests a new namespace, they submit a PR that adds their namespace definition. The PR review process becomes your access control review - the platform team verifies the group names match the right people, the resource quotas are appropriate, and the RBAC doesn't include unnecessary permissions. Once merged, Argo CD creates the namespace with all its RBAC resources automatically. This makes the onboarding process auditable, repeatable, and fast.