Recently I rebuilt my desk cluster, and by the time I had cert-manager and Traefik running, I had also run a lot of kubectl apply and helm install commands. Most of them were typed by my AI coding agent, to be fair, but the effect was the same. I could already feel the drift creeping in: what was actually running, and what would I rebuild from if an SD card died?
The answer, as always, is GitOps. This time I wanted it to cover secrets too, and I wanted guardrails, because the thing doing most of the typing is a very fast, very confident AI.
This is Part 2 of a series. In Part 1 (Rebuilding My Desk Kubernetes Cluster, Five Years On) I rebuilt the cluster from scratch. If you want the really long back story, my 2020 post on Azure Arc on an RPi cluster is where this all started.
This post will cover bootstrapping Flux on a Pi cluster, adopting Helm releases that were installed by hand, keeping Flux inside a tiny memory budget, SOPS with two recipients, and the lint pipeline and rules I use to keep the agent in its lane. Let’s dive in!
In English Please?
GitOps means Git is the source of truth for the cluster. You describe what you want in a repository, and a controller inside the cluster continually pulls that repository and makes reality match it.
Think of it like a thermostat. You set the temperature once (the commit), and Flux keeps nudging the room back to that number, even if someone opens a window (a hand-made kubectl edit).
Flux is the controller I picked. SOPS is the other half: a tool that encrypts the values inside a YAML file but leaves the structure readable, so I can commit secrets to Git and still review diffs.
Bootstrapping Flux and the repo layout
The bootstrap is one command. Flux installs its controllers, pushes its own manifests to the repo, and then manages itself from there. I only installed three controllers (source, kustomize and helm) because the notification and image automation controllers cost memory I do not have.
kubectl create ns flux-system
kubectl -n flux-system create secret generic sops-age \
--from-file=age.agekey=$HOME/.config/sops/age/rpi-k8s.agekey
flux bootstrap github \
--owner=<github-user> --repository=rpi-k8s --personal --private --branch=main \
--path=clusters/rpi-k8s \
--components=source-controller,kustomize-controller,helm-controller
The sops-age secret has to exist before anything encrypted reconciles. It is the one object I create by hand.
Flux’s bootstrap docs describe the deploy key option: the cluster gets an SSH key, and the matching GitHub deploy key is read-only by default. That is exactly what I want. The cluster can pull from the repo but can never push to it, so a compromised Pi cannot rewrite my history. The sync object that bootstrap generated uses an ssh://git@github.com/... URL, which is what I observed on my cluster.
The repository layout is deliberately boring:
clusters/rpi-k8s/
flux-system/ Flux itself (generated, plus my memory patches)
infrastructure.yaml Kustomizations: infra-controllers, then infra-configs
apps.yaml Kustomization: apps (after infra-configs)
infrastructure/
controllers/ HelmRepositories + HelmReleases (cert-manager, traefik, kured)
configs/ ClusterIssuers, Certificates, RBAC, network policy
apps/ workloads: whoami (canary), registry, dashboard, LLM gateway
The dependency chain is what makes this safe. Each stage waits for the previous one to be healthy, so CRDs and controllers exist before anything tries to use them:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: infra-configs
namespace: flux-system
spec:
dependsOn:
- name: infra-controllers
interval: 1h
retryInterval: 2m
timeout: 10m
path: ./infrastructure/configs
prune: true
wait: true
sourceRef:
kind: GitRepository
name: flux-system
decryption:
provider: sops
secretRef:
name: sops-age
dependsOn plus wait: true gives me the ordering, and prune: true means deleting a file from Git deletes the object. Note that only infra-configs and apps have the decryption block. infra-controllers does not, so secrets must never live under infrastructure/controllers/.
Adopting releases I installed by hand
Here is the awkward bit. cert-manager was installed with helm install, and Traefik arrived via the MicroK8s ingress add-on. Neither was in Git. I needed Flux to take them over without restarting anything.
The trick is that a HelmRelease can reuse the existing release name and namespace. Flux then upgrades the existing release instead of creating a new one. But "should be fine" is not evidence, so before committing I diffed what Helm would render against what was actually installed:
helm template cert-manager jetstack/cert-manager --version <pinned> \
--namespace cert-manager -f values.yaml > /tmp/wanted.yaml
helm get manifest cert-manager -n cert-manager > /tmp/live.yaml
diff /tmp/wanted.yaml /tmp/live.yaml
The only differences were Helm hook resources and whitespace, which is what I expected. Once that was true, adoption was a no-op. I did the same for Traefik.
Be warned: Traefik still found a way to bite. The first upgrade failed, because the Traefik chart bundles older Gateway API CRDs than the ones the MicroK8s add-on installed. A ValidatingAdmissionPolicy called safe-upgrades.gateway.networking.k8s.io (part of the Gateway API project’s CRD packaging, as I saw it on the cluster) refused the downgrade.
The fix is to tell Helm to leave CRDs alone:
spec:
# Gateway API CRDs (v1.5+) come from the MicroK8s add-on. The chart bundles older ones, and the
# Gateway API `safe-upgrades` ValidatingAdmissionPolicy rejects downgrades, so never touch CRDs here.
install:
crds: Skip
upgrade:
crds: Skip
Squeezing Flux into a Pi
Memory is tight on a 3.8 GB Pi. Flux’s stock manifests give each controller a 1 Gi memory limit. I measured the controllers idling at 36 to 52 Mi, so a 1 Gi ceiling is a lot of rope.
I capped them with kustomize patches in clusters/rpi-k8s/flux-system/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patches:
- target:
kind: Deployment
labelSelector: app.kubernetes.io/part-of=flux
patch: |
- op: replace
path: /spec/template/spec/containers/0/resources
value:
requests: { cpu: 25m, memory: 48Mi }
limits: { memory: 256Mi }
- target:
kind: Deployment
name: helm-controller
patch: |
- op: replace
path: /spec/template/spec/containers/0/resources/limits/memory
value: 384Mi
The patches sit in kustomization.yaml rather than the generated gotk-components.yaml, so they survive re-running flux bootstrap. The helm-controller gets 384 Mi because it renders charts.
SOPS with two recipients
This is the part I am happiest with. Flux needs to decrypt secrets unattended, so it needs a key that lives in the cluster. I, on the other hand, do not want a private key file sitting on every laptop I own.
SOPS can encrypt one file to several recipients, and any one of them can decrypt. So I use two: an age key for Flux, and an Azure Key Vault key for people. My .sops.yaml looks like this (public key material only, names genericised):
creation_rules:
- path_regex: (infrastructure|apps)/.*\.sops\.ya?ml$
encrypted_regex: ^(data|stringData)$
age: age1<public-key>
azure_keyvault: https://kv-<name>.vault.azure.net/keys/sops/<key-version>
The encrypted_regex encrypts only data and stringData, so names and metadata stay readable in pull requests (Flux’s SOPS guide notes that encrypting a Secret’s other metadata is not supported by kustomize-controller anyway). The Key Vault key is non-exportable, so decrypting needs a signed-in Azure identity: az login and MFA.
Creating a secret is then three commands:
kubectl create secret generic my-secret -n my-ns --from-literal=key=value \
--dry-run=client -o yaml > apps/my-app/my-secret.sops.yaml
sops -e -i apps/my-app/my-secret.sops.yaml
git add apps/my-app/my-secret.sops.yaml
To prove Key Vault worked on its own, I pointed the age key at a path that does not exist and decrypted anyway:
SOPS_AGE_KEY_FILE=/nonexistent sops -d apps/my-app/my-secret.sops.yaml
It decrypted, which means the Key Vault recipient was doing the work. Earlier, an end-to-end test with a throwaway secret showed it appear decrypted in the cluster, then get pruned when I removed it from Git.
A word from the wise: back up the age private key (mine is in Key Vault and a password manager). Lose every recipient key and those files are unreadable forever.
Working with the agent
I did not hand-write most of this. I asked my terminal coding agent to move everything into Flux, and it restructured the repo into infrastructure/controllers, infrastructure/configs and apps, wrote the HelmReleases, and bootstrapped Flux. In my opinion the diff-before-adopt step is the one worth copying: prove adoption is a no-op before letting Flux touch a running cert-manager, instead of hoping.
It also measured the Flux controllers rather than guessing, and wrote the memory patches. When the Traefik upgrade failed, the admission policy denial pointed at the CRD clash, and the fix was a two-line values change.
Where I had to make calls was the human side of the secrets. The agent started with an age key, which got Flux working end to end. The next morning I was about to move to a new Mac and wanted nothing tied to the laptop, so we added the Key Vault key as a second recipient. The guardrails live in a file called AGENTS.md at the root of the repo, which the agent reads before it works. A few of its hard rules that matter most for this post:
- Rule 1: Never commit plaintext secrets.
- Rule 2: Git is the source of truth. Do not
kubectl apply/edit/patchorhelm upgradeanything Flux manages. - Rule 8: Verify after every change. Do not claim success without evidence.
- Rule 11: Never print decrypted secret values into logs, PR text, issues or docs.
Rules are only words, though. CI is what actually holds the agent (and me, at midnight) to them.
The lint pipeline
Every push runs a GitHub Actions workflow with separate jobs (markdown, yamllint, Kubernetes manifests, Helm renders, secrets, shellcheck and actionlint), so a failure names the problem. The same checks run locally with scripts/ci/lint-all.sh, so the agent runs them before it pushes.
The Kubernetes job builds every path Flux reconciles with kustomize build, then validates the output with kubeconform in strict mode. The key detail is that the Kubernetes version comes from a single file, the MicroK8s track pin, so validation always matches what the cluster runs:
ci_kubeconform() {
kubeconform \
-strict \
-skip CustomResourceDefinition \
-summary \
-kubernetes-version "$KUBERNETES_VERSION" \
-schema-location default \
-schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
-schema-location "$(git rev-parse --show-toplevel)/scripts/ci/schemas/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json" \
"$@"
}
I deliberately did not use -ignore-missing-schemas. A typo in a kind should fail the build, not be skipped. The catch is that the public datree CRDs catalogue did not have schemas for Calico’s policy resources, so the checks could not validate them. The agent vendored those schemas into scripts/ci/schemas/ from the cluster’s own CRDs, and the third -schema-location picks them up.
A second job renders every HelmRelease with helm template and validates the output, catching bad values and chart versions that do not exist.
The secrets job is the one that earns its keep:
secrets:
name: Secrets
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@<pinned-sha>
with:
fetch-depth: 0 # gitleaks scans the full history
persist-credentials: false
- uses: actions/setup-python@<pinned-sha>
with:
python-version: "3.12"
- run: pip install "pyyaml==${PYYAML_VERSION}"
# *.sops.yaml are encrypted; no Secret manifest exists outside them.
- run: scripts/ci/check-sops.sh
- name: Install gitleaks
run: |
# download pinned release, verify its SHA-256, extract
- run: gitleaks git --redact --no-banner --verbose .
check-sops.sh fails the build if a *.sops.yaml file has no sops: block, if any value is not an ENC[AES256_GCM, string, or if a Secret manifest exists outside a .sops.yaml file. Then gitleaks scans the whole history for credential-shaped strings. Plaintext has to beat both.
My first real CI run went green across all seven jobs.
What I’d watch out for
- Hand edits vanish. That is the point, but it surprises you the first time.
flux suspend kustomization appswhile debugging by hand, thenflux resume. - Pods do not pick up changed secrets automatically if they read them as environment variables. Restart the workload after Flux applies the change.
- Order your recipients before your files. If you rotate the age key, apply the new in-cluster
sops-agesecret before Flux reconciles files encrypted to it, orinfra-configsandappswill fail to decrypt.
Wrapping Up
The cluster now rebuilds from a Git repo, secrets live in that repo safely, and a CI pipeline plus a short rulebook keep my agent (and me) from the classic mistakes. The agent is fast, so the guardrails matter more, not less. In Part 3 I will give the cluster access to Azure without storing a single secret in it.
As always, feel free to reach out with any questions or comments!
Up Next
- Rebuilding My Desk Kubernetes Cluster, Five Years On
- GitOps and Secrets with Flux and SOPS (this post)
- Workload Identity Without AKS
- Entra ID for kubectl, Five Years After Okta
- Ingress, Certificates and Getting Home Through CGNAT
- Hardening a Home Cluster, and How I Broke It
- Keeping It Patched Without Babysitting It
- One LLM Gateway for Everything, with a Hard $100 Cap
- What I Learnt Building It with an AI Agent
Until next time, stay cloudy!