Recently I decided my Raspberry Pi cluster should talk to Azure without a single secret living inside it. No client secrets, nothing for a future me to rotate or leak.
This is Part 3 of a series. In Part 1 I rebuilt the cluster, and in Part 2 I put it under GitOps with Flux and SOPS. All the titles are listed at the end.
On AKS that is close to a checkbox. On a desk full of Pis it is mostly plumbing, but small enough to understand end to end. Back in 2020 I got Azure Arc running on a Pi cluster; this time I wanted Azure to trust the cluster instead.
This post will cover how workload identity federation works, how I self-hosted the OIDC issuer on a public blob container, how cert-manager uses it for DNS-01 certificates, and how a systemd timer uses it for nightly dqlite backups. Let’s dive in!
In English Please?
Think of a pub that accepts your work ID as proof you are an employee, so you get the staff discount without the pub ever issuing you a card. The pub does not hold a copy of your credentials. It just trusts your employer’s stamp, and checks the stamp is real.
Kubernetes already issues every service account a signed token (a JWT). With workload identity federation, Microsoft Entra ID can be told to trust tokens from a particular issuer for a particular subject. The cluster presents a token, Entra checks the signature against the issuer’s public keys, and hands back a normal Entra access token for a managed identity. The only thing Entra needs is a public URL where it can read the issuer’s discovery document and public keys.
AKS hosts that URL for you; I host it myself. The Entra docs for federated credentials cover the Kubernetes scenario, and the Azure Workload Identity docs for self-managed clusters describe the two documents I publish below.
Self-Hosting the OIDC Issuer
Entra needs two documents: a discovery document at .well-known/openid-configuration, and a JWKS (JSON Web Key Set) at openid/v1/jwks holding the public signing keys. Neither is secret. I put them in a blob container with anonymous read on a storage account, so the issuer URL is https://st<name>oidc.blob.core.windows.net/oidc.
A small script generates both, pulling the key set from the API server itself:
#!/usr/bin/env bash
# Generate the public OIDC discovery documents for the cluster's service-account issuer.
# Usage: scripts/oidc/generate.sh https://<account>.blob.core.windows.net/oidc [outdir]
# Upload the outputs to the container with blob names:
# .well-known/openid-configuration
# openid/v1/jwks
set -euo pipefail
issuer="${1:?issuer URL required, e.g. https://<account>.blob.core.windows.net/oidc}"
issuer="${issuer%/}"
out="${2:-./oidc-out}"
mkdir -p "$out/.well-known" "$out/openid/v1"
kubectl --context rpi-k8s get --raw /openid/v1/jwks > "$out/openid/v1/jwks"
cat > "$out/.well-known/openid-configuration" <<EOF
{
"issuer": "${issuer}",
"jwks_uri": "${issuer}/openid/v1/jwks",
"response_types_supported": ["id_token"],
"subject_types_supported": ["public"],
"id_token_signing_alg_values_supported": ["RS256"]
}
EOF
echo "Wrote $out/.well-known/openid-configuration and $out/openid/v1/jwks"
I uploaded both files with those exact blob names and checked in a private browser window that both URLs returned JSON.
Next, each API server has to stamp its tokens with that issuer. In the MicroK8s args file /var/snap/microk8s/current/args/kube-apiserver I added these flags:
--service-account-issuer=https://st<name>oidc.blob.core.windows.net/oidc
--service-account-issuer=https://kubernetes.default.svc
--service-account-jwks-uri=https://st<name>oidc.blob.core.windows.net/oidc/openid/v1/jwks
The kube-apiserver reference says that when the issuer flag is given more than once, the first is used to generate tokens. That is why the blob URL goes first and the old in-cluster issuer stays second, so tokens already issued stay valid. The JWKS URI flag overrides where the discovery document points for keys. The API server’s own address is private, so Entra gets pointed at the blob instead.
I rolled this out one node at a time and restarted snap.microk8s.daemon-kubelite on each, because a typo in that file stops the API server starting on that node.
Note: all three nodes must share the same service account signing key, otherwise one JWKS cannot cover them all. On MicroK8s that is /var/snap/microk8s/current/certs/serviceaccount.key. Check it with sha256sum on each node before trusting tokens from it. If the key ever changes, rerun the script and re-upload the JWKS, or every token exchange will fail.
The last Azure piece is a user-assigned managed identity per consumer, each with a federated credential: issuer set to the blob URL exactly (no trailing slash), subject system:serviceaccount:<namespace>:<serviceaccount>, audience api://AzureADTokenExchange.
Using it without the webhook
AKS adds a mutating webhook that injects the token and environment variables into labelled pods. I did not install it. A projected token volume and four environment variables do the same job. These are the cert-manager Helm values from my Flux HelmRelease:
values:
crds:
enabled: true
# DNS-01 self-checks must use public resolvers (CNAME-delegated challenges in acme.example.com).
dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
dns01RecursiveNameserversOnly: true
serviceAccount:
name: cert-manager # must match the federated credential subject
extraEnv:
- name: AZURE_CLIENT_ID
value: <cert-manager-managed-identity-client-id>
- name: AZURE_TENANT_ID
value: <tenant-id>
- name: AZURE_FEDERATED_TOKEN_FILE
value: /var/run/secrets/azure/tokens/azure-identity-token
- name: AZURE_AUTHORITY_HOST
value: https://login.microsoftonline.com/
volumes:
- name: azure-identity-token
projected:
sources:
- serviceAccountToken:
audience: api://AzureADTokenExchange
expirationSeconds: 3600
path: azure-identity-token
volumeMounts:
- name: azure-identity-token
mountPath: /var/run/secrets/azure/tokens
readOnly: true
The cert-manager Azure DNS docs show the same variables and projected volume as the webhook’s output, so I wrote them by hand. The Azure SDK reads the token file and swaps it with Entra. The service account name must stay cert-manager, because that is what the federated credential trusts.
DNS-01 Without Giving Away My Domain
I wanted wildcard certificates from Let’s Encrypt, which means DNS-01 challenges. That needs write access to DNS, and I was not keen on giving a Pi write access to a domain that also holds my mail records.
The fix is delegation. Let’s Encrypt follows CNAMEs when it looks up the challenge record, so I created a child zone acme.example.com that holds nothing but challenge records, and gave the cert-manager identity DNS Zone Contributor on that zone only. Then I created one static CNAME per real zone, once, by hand:
_acme-challenge.example.compoints toexample-com.acme.example.com_acme-challenge.lab.example.compoints tolab-example-com.acme.example.com
cert-manager’s delegated domains support does the rest with cnameStrategy: Follow:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: <your-email>
privateKeySecretRef:
name: letsencrypt-prod-account
solvers:
- dns01:
cnameStrategy: Follow
azureDNS:
subscriptionID: <subscription-id>
resourceGroupName: rg-dns # resource group of the acme.example.com zone
hostedZoneName: acme.example.com
environment: AzurePublicCloud
managedIdentity:
clientID: <cert-manager-managed-identity-client-id>
The two wildcard certificates are then ordinary cert-manager resources:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: wildcard-example-com
namespace: ingress
spec:
secretName: wildcard-example-com-tls
issuerRef: { name: letsencrypt-prod, kind: ClusterIssuer }
dnsNames: ["*.example.com"] # apex deliberately excluded
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: wildcard-lab-example-com
namespace: ingress
spec:
secretName: wildcard-lab-example-com-tls
issuerRef: { name: letsencrypt-prod, kind: ClusterIssuer }
dnsNames: ["*.lab.example.com", "lab.example.com"]
Be warned: resourceGroupName is the resource group of the DNS zone, not of the managed identity. We got that wrong the first time (the git history has a "fix issuer RG" commit to prove it).
A word from the wise: only names with an _acme-challenge CNAME into the challenge zone can get a certificate. When I tried a certificate for a single name with no CNAME, the challenge stayed pending forever. The wildcard certificates cover everything under them, so in practice I use those. For anything else, create the CNAME first, and remember a CNAME cannot sit alongside a TXT record at the same name. The Part 5 post shows how these certificates get used for ingress.
The Backup Timer: Workload Identity on the Host
The second consumer is not a pod. Every night each node backs up the MicroK8s datastore (dqlite, which holds every Kubernetes object, Secrets included) to blob storage. A systemd timer starts a script that mints its own short-lived service account token and trades it with Entra.
The service account is deliberately boring. No pods run in that namespace and no token is mounted:
apiVersion: v1
kind: Namespace
metadata:
name: backup
labels:
pod-security.kubernetes.io/enforce: restricted
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: dqlite-backup
namespace: backup
automountServiceAccountToken: false
The script (installed at /usr/local/sbin/) uses kubectl create token to get a ten-minute token with the right audience, posts it to the Entra token endpoint as a client assertion, and uploads the archive with the resulting bearer token:
#!/usr/bin/env bash
# /usr/local/sbin/microk8s-dqlite-backup
# Back up the MicroK8s dqlite datastore and upload it to Azure Blob Storage.
set -euo pipefail
TENANT_ID=<tenant-id>
CLIENT_ID=<backup-managed-identity-client-id>
ACCOUNT=st<name>backup
CONTAINER=dqlite
LOCAL_DIR=/var/backups/microk8s
KEEP_LOCAL=3
MICROK8S=/snap/bin/microk8s
host=$(hostname)
ts=$(date -u +%Y%m%dT%H%M%SZ)
name="${host}-${ts}"
install -d -m 700 "$LOCAL_DIR"
work=$(mktemp -d "$LOCAL_DIR/.work.XXXXXX")
trap 'rm -rf "$work"' EXIT
echo "backing up dqlite to $LOCAL_DIR/$name.tar.gz"
( cd "$work" && "$MICROK8S" dbctl backup -o "$name" >/dev/null )
archive=$(find "$work" -maxdepth 1 -name "$name*.tar.gz" -print -quit)
[ -n "$archive" ] || { echo "dbctl backup produced no archive" >&2; exit 1; }
tar -tzf "$archive" >/dev/null # integrity check
mv "$archive" "$LOCAL_DIR/$name.tar.gz"
archive="$LOCAL_DIR/$name.tar.gz"
size=$(stat -c %s "$archive")
sha=$(sha256sum "$archive" | cut -d' ' -f1)
sa_token=$("$MICROK8S" kubectl create token dqlite-backup -n backup \
--audience api://AzureADTokenExchange --duration 10m)
access_token=$(curl -fsS -X POST "https://login.microsoftonline.com/${TENANT_ID}/oauth2/v2.0/token" \
--data-urlencode "client_id=${CLIENT_ID}" \
--data-urlencode "scope=https://storage.azure.com/.default" \
--data-urlencode "grant_type=client_credentials" \
--data-urlencode "client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer" \
--data-urlencode "client_assertion=${sa_token}" \
| python3 -c 'import json,sys; print(json.load(sys.stdin)["access_token"])')
url="https://${ACCOUNT}.blob.core.windows.net/${CONTAINER}/dqlite/${host}/${name}.tar.gz"
curl -fsS -X PUT "$url" -T "$archive" \
-H "Authorization: Bearer ${access_token}" \
-H "x-ms-version: 2023-11-03" \
-H "x-ms-blob-type: BlockBlob" \
-H "x-ms-blob-content-type: application/gzip" \
-H "x-ms-meta-sha256: ${sha}" \
-o /dev/null
echo "uploaded ${size} bytes sha256=${sha} to ${url}"
# Keep the newest KEEP_LOCAL archives locally.
# shellcheck disable=SC2012
ls -1t "$LOCAL_DIR"/*.tar.gz 2>/dev/null | tail -n +$((KEEP_LOCAL + 1)) | xargs -r rm -f
Each run produces an archive (about 4.3 MB here), checks it is a valid tarball, records a SHA-256 as blob metadata and keeps the newest three copies locally. The service and timer are short:
# /etc/systemd/system/microk8s-dqlite-backup.service
[Unit]
Description=Back up MicroK8s dqlite datastore to Azure Blob
Wants=network-online.target
After=network-online.target snap.microk8s.daemon-k8s-dqlite.service
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/microk8s-dqlite-backup
Nice=10
IOSchedulingClass=idle
# /etc/systemd/system/microk8s-dqlite-backup.timer
[Unit]
Description=Daily MicroK8s dqlite backup
[Timer]
OnCalendar=*-*-* 01:00:00
RandomizedDelaySec=45min
FixedRandomDelay=true
Persistent=true
[Install]
WantedBy=timers.target
The systemd timer docs explain FixedRandomDelay: the random delay is derived deterministically, so each node gets its own stable offset between 01:00 and 01:45. The nodes never back up at once, and the offsets stay clear of the Sunday snap refresh windows. Persistent=true catches up if a node was off.
The storage account has shared-key access disabled and anonymous access off, so Entra bearer tokens are the only way in. A lifecycle rule deletes blobs after 30 days. Downloading one archive to a workstation and comparing its hash against the blob metadata showed they matched.
Note: I have not yet rehearsed a restore on this cluster. A backup you have never restored is a hope, not a backup, so a throwaway single-node rebuild is on my list.
Least privilege
Each identity can do exactly one thing:
| Identity | Federated subject | Role | Scope |
|---|---|---|---|
| cert-manager managed identity | system:serviceaccount:cert-manager:cert-manager |
DNS Zone Contributor | the acme.example.com zone only |
| Backup managed identity | system:serviceaccount:backup:dqlite-backup |
Storage Blob Data Contributor | the dqlite container only |
A stolen cert-manager token could write TXT records in a zone that holds nothing else, and only for an hour (ten minutes for the backup token). The nicest property is that there is nothing to rotate: the signing key lives on the nodes, the public half lives in a blob, and Entra does the trusting.
Working With the Agent
As with the rest of this series, my co-pilot did most of the typing. This part took roughly an hour and a quarter on the evening of 15 November, from about 22:30 to 23:46.
What I asked for was simple: Azure access with no stored secrets. The agent proposed the self-hosted issuer, wrote the generator script, a portal runbook for the Azure steps (the commit is called "workload identity portal runbook") and rolled the API server flags out one node at a time, in line with the hard rules in its AGENTS.md file. It also did the less glamorous work: proving the end-to-end path by getting both wildcard certificates issued, and verifying the backup by downloading it and comparing hashes rather than just trusting a 200.
It was not flawless. The resource group slip above was a mistake (the agent’s, caught when the first issuer would not work), and the stuck single-name certificate was a constraint we discovered by hitting it, then documented. Later, as part of hardening, the OIDC storage account had its shared-key access turned off. When it came to money, it asked first: it checked whether I wanted to switch a storage account to locally redundant storage, and I said geo-redundant was fine. That was my call to make, not its.
Wrapping Up
With a public blob, two flags on the API server and a projected token, a self-managed cluster can do the same workload identity trick as AKS: no secrets, short-lived tokens and narrow roles. Next up is Part 4, where I tackle the scary finding that every token on this cluster was effectively cluster-admin, and wire Entra ID into kubectl, five years after my Okta post.
As always, feel free to reach out with any questions or comments!
Up Next
- Rebuilding My Desk Kubernetes Cluster, Five Years On
- GitOps and Secrets with Flux and SOPS
- Workload Identity Without AKS (this post)
- Entra ID for kubectl, Five Years After Okta
- Ingress, Certificates and Getting Home Through CGNAT
- Hardening a Home Cluster, and How I Broke It
- Keeping It Patched Without Babysitting It
- One LLM Gateway for Everything, with a Hard $100 Cap
- What I Learnt Building It with an AI Agent
Until next time, stay cloudy!