Recently I rebuilt my desk Kubernetes cluster, and as an identity geek the first question was "how will I log in?". Back in 2021 I wrote up how I integrated Kubernetes with Okta for user RBAC on a MicroK8s cluster. It was a pile of --oidc-* flags on the API server, three curl requests to get a token by hand, and kubelogin to tidy it all up. This time I wanted to see what five years of Kubernetes had changed.
Quite a lot, as it turns out. I also found something embarrassing: my 2021 post mentioned that MicroK8s runs with --authorization-mode=AlwaysAllow by default, and I still built a new cluster without checking. Spoiler: it was on. Every pod could have been a cluster admin.
This is Part 4 of the Raspberry Co-PI-lot series, and the earlier parts (rebuilding the cluster, Flux and SOPS, workload identity) are listed at the bottom.
This post will cover fixing that, signing in to kubectl with Microsoft Entra ID, keeping a break-glass way in, and a web dashboard on the same sign-in. My AI coding agent did most of the typing, so I’ll cover how that went too. Let’s dive in!
The scary one first: RBAC was off
Kubernetes decides what you can do in two steps: authentication asks "who are you?" and authorization asks "are you allowed?". Think of a nightclub: the bouncer checks your ID, then someone else decides whether you get into the VIP room. MicroK8s ships with the second person on a long lunch break.
Every pod gets a ServiceAccount token mounted automatically, and under AlwaysAllow that token is effectively cluster-admin. I proved it against my little test app’s identity:
kubectl auth can-i '*' '*' --as=system:serviceaccount:whoami:default
It answered yes. A harmless demo app could have deleted everything in the cluster.
The fix is the MicroK8s rbac add-on, which sets --authorization-mode=RBAC,Node. Be warned: it restarts all three API servers at once. For me that was about a 10 second API blip, with workloads still running and a burst of forbidden log lines while the default roles were created. Afterwards the same check returned no, and a fresh certificate issued by cert-manager (Part 3) still went through.
Note: watch the logs afterwards. A burst of forbidden that stops is normal; one that keeps repeating is a component missing a role.
Structured authentication, not a pile of flags
In 2021, the API server learned about Okta through flags like --oidc-issuer-url and --oidc-username-prefix. Those still exist, but there is now a better way. Kubernetes introduced a file-based AuthenticationConfiguration in 1.29, and the v1.34 release notes list structured authentication configuration as stable. The Kubernetes docs describe it as supporting multiple JWT authenticators, CEL expression validation and dynamic reloading.
In English: instead of squeezing your identity provider into flags, you hand the API server a YAML file saying "trust tokens from this issuer, for this audience, and turn the claims into a username and groups like so". It is easier to review in Git and can describe more than one issuer.
Here is the file from my cluster, with names genericised:
apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
jwt:
- issuer:
url: https://login.microsoftonline.com/<tenant-id>/v2.0
audiences:
- <kubectl-app-client-id>
audienceMatchPolicy: MatchAny
claimValidationRules:
- expression: "claims.tid == '<tenant-id>'"
message: token is not from my tenant
claimMappings:
username:
claim: oid
prefix: "entra:"
groups:
claim: groups
prefix: "entra:"
extra:
- key: example.com/upn
valueExpression: "has(claims.preferred_username) ? claims.preferred_username : ''"
What it does: only tokens from my tenant, issued to my rpi-k8s-kubectl app registration, are accepted; a CEL rule rejects any token whose tid is not my tenant (belt and braces); the username comes from the oid claim and groups from groups, both prefixed entra: so they can never be confused with a certificate user or ServiceAccount; and the friendly name rides along as extra.
In 2021 I swapped the username claim to preferred_username to make things readable. This time I used the object ID for identity and parked the friendly name in extra, because names change and object IDs don’t.
On MicroK8s the file lives at /var/snap/microk8s/current/args/authentication-config.yaml, and the API server needs one extra flag. I applied it one node at a time, starting with a canary:
sudo install -m 0644 authentication-config.yaml /var/snap/microk8s/current/args/authentication-config.yaml
f=/var/snap/microk8s/current/args/kube-apiserver
grep -q authentication-config $f \
|| echo '--authentication-config=${SNAP_DATA}/args/authentication-config.yaml' | sudo tee -a $f
sudo systemctl restart snap.microk8s.daemon-kubelite
kubectl get nodes
A word from the wise: a syntax error in that file stops the API server on that node starting. That is why it goes one node at a time.
Entra ID, kubelogin and one group
On the Entra side, the setup is modest:
- A security group,
rpi-k8s-admins, with me in it. - An app registration,
rpi-k8s-kubectl, single tenant, as a public client withhttp://localhost:8000andhttp://localhost:18000redirect URIs. - The groups claim switched on (
groupMembershipClaims: SecurityGroup) and admin consent granted.
I was moving to a new Mac, so I wanted nothing tied to the machine: no client secret, no token baked into a kubeconfig. A public client using the browser, with MFA handled by Entra, fits.
On the workstation, kubelogin does the heavy lifting, as in 2021. It is a kubectl credential plugin that opens a browser, caches the tokens and refreshes them. The wiring:
kubectl config set-credentials rpi-k8s-entra --exec-api-version=client.authentication.k8s.io/v1 \
--exec-interactive-mode=IfAvailable --exec-command=kubectl \
--exec-arg=oidc-login --exec-arg=get-token \
--exec-arg=--oidc-issuer-url=https://login.microsoftonline.com/<tenant-id>/v2.0 \
--exec-arg=--oidc-client-id=<kubectl-app-client-id> \
--exec-arg=--oidc-extra-scope=profile --exec-arg=--oidc-extra-scope=offline_access
kubectl config set-context rpi-k8s --cluster=rpi-k8s --user=rpi-k8s-entra
kubectl config use-context rpi-k8s
kubectl auth whoami
Unlike 2021 there is no client secret in the kubeconfig. The first kubectl auth whoami opens the browser, I sign in (consenting on behalf of the organisation the first time), and the answer comes back as entra: plus my object ID.
Last piece: tell RBAC what the group is allowed to do. This binding is committed to Git and reconciled by Flux (Part 2):
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: entra-rpi-k8s-admins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: "entra:<rpi-k8s-admins-group-object-id>"
The group name carries the entra: prefix from the claim mapping. My 2021 binding named one Okta user; this binds a group, so adding or removing an admin is a group change in Entra with no cluster change. Removing someone stops their cached token being useful once it expires.
Note: Entra drops the groups claim for users in more than 200 groups (an overage), per Microsoft’s group claims guidance. Not an issue in my lab, but read that page before doing this in a real tenant.
Break-glass: the certificate I try not to use
What if Entra, or my internet, is the thing that is broken? I kept a way in that depends on neither: the MicroK8s admin client certificate, which authenticates as system:masters and so works even if I fumble every binding.
The downside is that Kubernetes does not support certificate revocation. A leaked admin certificate stays valid until it expires, and short of re-issuing the cluster CA I cannot take it back. So it does not live on my laptop: I fetch it over SSH from a node when needed and merge it in as a rpi-k8s-breakglass context. The last resort is microk8s kubectl on a node.
I accepted that risk knowingly. For a home lab with key-only SSH, it is a sensible trade; in a real environment I would want a shorter-lived emergency credential.
Headlamp, and the wrong-account moment
I also wanted a dashboard. The old Kubernetes Dashboard is now archived, and its README points to Headlamp. Headlamp supports OpenID Connect sign-in and forwards your own token to the API server.
That is the design I wanted: Headlamp has no authority of its own, so what you can do in it is exactly what your Entra identity can do in kubectl. Its ServiceAccount has no bindings at all (the chart’s default cluster-admin binding is off). I reused the same app registration, so the token audience is already trusted and the authentication file needed no change. The extras were a Web redirect URI and a client secret, kept only in SOPS and Key Vault.
Here is the relevant part of the Flux-managed HelmRelease values:
values:
fullnameOverride: headlamp
image:
tag: v0.46.0
config:
inCluster: true
inClusterContextName: rpi-k8s
staticPlugins:
enabled: false
watchPlugins: false
oidc:
secret:
create: false
externalSecret:
enabled: true
name: headlamp-oidc
hasScopes: true
usePKCE: true
env:
- name: OIDC_CLIENT_ID
value: <kubectl-app-client-id>
- name: OIDC_ISSUER_URL
value: https://login.microsoftonline.com/<tenant-id>/v2.0
- name: OIDC_SCOPES
value: openid profile email offline_access
- name: OIDC_CALLBACK_URL
value: https://headlamp.lab.example.com/oidc-callback
- name: OIDC_USE_PKCE
value: "true"
clusterRoleBinding:
create: false
The secret comes from a SOPS-encrypted Secret called headlamp-oidc. No plugins, a restricted Pod Security profile and a default-deny network policy (Part 6) keep it tight, and it idled at about 15 to 16 MiB when I measured, which matters with 3.8 GB per Pi.
Now the fun bit. My first look at Headlamp showed "You don’t have permissions" everywhere. I simply had not signed in, and by design an unauthenticated Headlamp can do nothing.
So I signed in and still had no rights. My work browser profile had signed me in as a guest account from my day job’s tenant, which is not in rpi-k8s-admins, so Kubernetes correctly gave it nothing. I told the agent: "ignore me, when I use the right account it works."
A lovely accidental proof the group binding works. If you juggle tenants, use a dedicated browser profile for lab sign-ins.
Working with the agent
Everything above was built by me directing an AI coding agent in my terminal. How it went:
- The finding. The agent spotted
AlwaysAllowduring its research pass (about 01:00) and proved it with thecan-icheck. I asked for RBAC on, then Entra sign-in, with the constraint that nothing be tied to my old laptop. - What it did well. It canaried the authentication file on node 1 and confirmed the API answered before touching the others, and the break-glass context stayed in my kubeconfig throughout. It also checked results (
can-ibefore and after) rather than just saying "done". - Where I was needed. The agent cannot complete a browser sign-in with MFA, so the first kubelogin run and the consent were mine. Reusing the one app registration for Headlamp was the agent’s recommendation, with the trade-off spelled out (no node change, but a client secret on a public-client app); I went with it. The break-glass risk was my call too.
- A wrong-account moment, mine. The agent could not know which browser profile I would use.
- Away from home. From somewhere without Wi-Fi I pasted a screenshot and asked it to stop a running job; it cancelled the (read-only) one.
Headlamp also paid for itself quickly: it surfaced a flapping readiness probe on the local-path provisioner, fixed with a longer timeout and more memory headroom. And in Part 6 it is how I noticed a Calico crashloop.
Wrapping Up
Five years on, the shape is the same (token from the identity provider, kubelogin, RBAC decides), but the API server config is a reviewable file, access is a group instead of a named user, and I now check RBAC is actually on. Next in Part 5: ingress, certificates and getting home through CGNAT.
As always, feel free to reach out with any questions or comments!
Up Next
- Rebuilding My Desk Kubernetes Cluster, Five Years On
- GitOps and Secrets with Flux and SOPS
- Workload Identity Without AKS
- Entra ID for kubectl, Five Years After Okta (this post)
- Ingress, Certificates and Getting Home Through CGNAT
- Hardening a Home Cluster, and How I Broke It
- Keeping It Patched Without Babysitting It
- One LLM Gateway for Everything, with a Hard $100 Cap
- What I Learnt Building It with an AI Agent
Until next time, stay cloudy!