Raspberry Co-PI-lot (Part 6): Hardening a Home Cluster, and How I Broke It

Recently I asked my co-pilot a question I already suspected the answer to. It was around nine at night, and the Calico pods on every node were sitting in a crashloop with over a hundred restarts each. I typed: "looks like calico is in a crashloop, I am fixing with another agent unless you tell me you caused it?"

The reply was four words long: "Yes, I caused it."

I have had worse confessions from humans. This one at least arrived with a fix. But what stuck with me is that everything had passed. Port scans from my Mac, pod tests, even a drain and reboot of one node. Roughly seven hours later the network layer had restarted about 118 times on each node, and none of my tests had noticed.

This is Part 6 of the Raspberry Co-PI-lot series. Earlier parts covered rebuilding the cluster, GitOps with Flux and SOPS, workload identity, Entra ID for kubectl and ingress with certificates. Each one is listed at the bottom, none of them required, but the identity and ingress posts are the ones this post leans on most.

This post will cover the hardening I put on the cluster (Pod Security Admission, network policies, a LAN guard, Secrets encryption, audit logging and a host firewall), the single line that broke it, and what hardening with an AI agent was like. Let’s dive in!

Why Bother Hardening a Home Cluster?

Three Pis on my desk, behind a router, with no public ingress. But the threat I care about is me running something dodgy in a container one day, and it then poking around my home network. Think of a block of flats: no burglars in the lobby, but every flat still gets a lock and the fire doors stay shut.

The goals were modest: make a compromised pod boring, keep it off the LAN, keep stolen backups useless, record who changed what, and shrink what each node listens on. Memory is tight here (3.8 GB per Pi), so it had to be configuration, not software.

Pod Security Admission and Network Policies

Kubernetes ships with Pod Security Admission, which checks every pod against one of three Pod Security Standards: privileged, baseline or restricted. You turn it on per namespace with labels, and I put one on every namespace. Most are restricted. Three are privileged on purpose, because Calico, kured and the local-path helper pods need host access.

For namespaces I forget to label, the API server also takes a cluster-wide default from an admission configuration file. This is the block I added on each node:

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: EventRateLimit
    path: eventconfig.yaml
  - name: PodSecurity
    configuration:
      apiVersion: pod-security.admission.config.k8s.io/v1
      kind: PodSecurityConfiguration
      defaults:
        enforce: baseline
        enforce-version: latest
        audit: restricted
        audit-version: latest
        warn: restricted
        warn-version: latest
      exemptions:
        usernames: []
        runtimeClasses: []
        namespaces: []

MicroK8s already ships this file with EventRateLimit, so I added PodSecurity beside it. An unlabelled namespace now enforces baseline and warns about anything that would fail restricted. The shape follows the Kubernetes docs for enforcing standards cluster-wide. A privileged pod in a fresh namespace was refused, which is all the testing it needed.

Enforcing restricted meant my whoami canary and the Zot registry had to run as non-root. Zot’s volume had to be recreated, because local-path volumes are plain host directories and fsGroup does not chown them.

Next, NetworkPolicies. Each app gets a default-deny for ingress and egress, plus an explicit allow from Traefik. Here is the whoami pair:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: whoami
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

A matching allow-from-traefik policy lets only Traefik reach port 8080. A pod in default now times out against the Service, but still reaches the app through https://test.lab.example.com. Note: once egress is default-denied, a pod that needs DNS needs an explicit allow to CoreDNS on port 53.

Keeping Pods Off My Home Network

This was my own ask: pods may reach the internet, but not the router, other devices or the nodes’ management ports.

First I measured. A throwaway busybox pod could reach the router’s web UI and DNS, SSH on a node and the MicroK8s cluster agent. Not a great baseline.

Plain NetworkPolicy is awkward for "everything private except a few things", so I used Calico. A Calico tier is evaluated before namespace NetworkPolicies, which makes it a good place for a cluster-wide guardrail:

apiVersion: crd.projectcalico.org/v1
kind: Tier
metadata:
  name: lan-guard
spec:
  order: 500
  defaultAction: Pass
---
apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
  name: lan-guard.block-private-egress
spec:
  tier: lan-guard
  order: 100
  selector: has(projectcalico.org/namespace)
  types:
    - Egress
  egress:
    - action: Pass
      protocol: TCP
      destination:
        nets: [192.168.1.101/32, 192.168.1.102/32, 192.168.1.103/32]
        ports: [16443, 10250]
    - action: Pass
      protocol: UDP
      source:
        selector: k8s-app == "kube-dns"
      destination:
        nets: [192.168.1.1/32]
        ports: [53]
    - action: Pass
      protocol: TCP
      source:
        selector: k8s-app == "kube-dns"
      destination:
        nets: [192.168.1.1/32]
        ports: [53]
    - action: Deny
      destination:
        nets: [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, 169.254.0.0/16]
        notNets: [10.1.0.0/16, 10.152.183.0/24]
    - action: Pass

(The node IPs are placeholders. The pod and Service ranges are the MicroK8s defaults.)

Top to bottom: pods may reach the API server and kubelet, CoreDNS may forward to the router, everything else private is denied except our own pod and Service networks, and the rest passes on to later tiers. The first exception looks odd until you know that Calico sees API traffic after kube-proxy rewrites the destination, so it appears as a node IP on port 16443.

After applying it, the router, node SSH and cluster agent all timed out from a pod, while the API, DNS, Services and the internet still worked. Remember that has(projectcalico.org/namespace) selector; it matters in a minute.

Encrypting Secrets and Logging Who Did What

By default Kubernetes stores Secrets unencrypted, and the dqlite backups from Part 3 would carry them too. The Kubernetes docs on encrypting data at rest list the provider options. I picked secretbox, which the docs describe as using XSalsa20 and Poly1305 and as faster than the others with a 32-byte key. They list KMS v2 for external key services, and note that secretbox key material is accessible from the control plane host.

That matters here. I generated the key straight into Key Vault (kv-<name>) and a script pipes it over SSH into a root-only file on each node. So this protects the datastore files and backups, not against root on a node. For a home cluster I am fine with that.

The rollout order matters. Every API server must read a format before any writes it, so the config goes in twice:

# Step 1, "read" mode: still writes plaintext, but can decrypt secretbox.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
    - identity: {}
    - secretbox:
        keys:
          - name: key-2026-11-16
            secret: <32-byte key, base64, from Key Vault>

In step 2 the order flips, with secretbox first so new writes are encrypted. I restarted one API server at a time between steps, then rewrote every Secret with kubectl get secrets -A -o json | kubectl replace -f - (24 of them). A marker Secret was findable in the dqlite files in plaintext before, and not after.

A word from the wise: losing that key makes every Secret unreadable, and a restore needs it. Key Vault holds the only copy, with purge protection on.

Audit logging is the quiet one. The policy works at Metadata level only (the Kubernetes audit docs explain that this logs who, when, what and which verb, but not request or response bodies), so Secret values never reach the log. It records every create, update, patch and delete by anyone, plus who read a Secret, and drops reads, watches, events, leases, health checks and the control plane’s own chatter:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  - level: None
    resources:
      - group: coordination.k8s.io
        resources: [leases]
      - group: ""
        resources: [events]
      - group: events.k8s.io
        resources: [events]
  - level: None
    userGroups: ["system:nodes"]
  - level: None
    users:
      - system:kube-scheduler
      - system:kube-controller-manager
      - system:kube-proxy
      - system:apiserver
  - level: None
    nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version", "/metrics", "/openid/*", "/.well-known/*"]
  - level: Metadata
    resources:
      - group: ""
        resources: [secrets]
    verbs: [get, list]
  - level: Metadata
    verbs: [create, update, patch, delete, deletecollection]
  - level: None

The log lives outside the snap at /var/log/kube-apiserver, about 60 MB per node at most. A ConfigMap I created and deleted as my Entra user showed up with my username. Be warned: it is per node, not shipped anywhere and nothing alerts on it. It is a record, not a defence.

The Host Firewall, and the Line I Forgot

The nodes themselves, last. First I shrank what they listen on: rpcbind and avahi-daemon masked, controller-manager and scheduler bound to localhost. Then a firewall. Calico can do this without another daemon: host endpoints represent a node’s own interfaces, and policy can then apply to traffic into the node. Calico can create them automatically.

The design: the LAN reaches only SSH (22), the Kubernetes API (16443) and Traefik (80 and 443), plus ICMP. Everything else (kubelet, dqlite, the cluster agent, VXLAN) comes only from the three nodes and the pod network. Calico also has failsafe rules so a mistake cannot cut off every host, and I listed SSH and the API port there explicitly.

The policy today:

apiVersion: crd.projectcalico.org/v1
kind: FelixConfiguration
metadata:
  name: default
spec:
  failsafeInboundHostPorts:
    - { protocol: tcp, port: 22 }
    - { protocol: tcp, port: 16443 }
    - { protocol: udp, port: 68 }
---
apiVersion: crd.projectcalico.org/v1
kind: KubeControllersConfiguration
metadata:
  name: default
spec:
  controllers:
    node:
      hostEndpoint:
        autoCreate: Enabled
---
apiVersion: crd.projectcalico.org/v1
kind: Tier
metadata:
  name: host-firewall
spec:
  order: 400
---
apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
  name: host-firewall.node-ingress
spec:
  tier: host-firewall
  order: 100
  selector: has(kubernetes.io/hostname) && !has(projectcalico.org/namespace)
  types: [Ingress]
  ingress:
    - action: Allow
      source:
        nets: [127.0.0.0/8, 192.168.1.101/32, 192.168.1.102/32, 192.168.1.103/32, 10.1.0.0/16]
    - action: Allow
      protocol: TCP
      destination:
        ports: [22, 16443, 80, 443]
    - action: Allow
      protocol: ICMP
    - action: Deny

First, remember the lan-guard selector? It started life as all(). With host endpoints about to exist, all() would have matched the nodes too and cut them off from the router and each other, so I narrowed it to pods only before rolling out the firewall. That was the agent’s call.

Second, I rolled out carefully: canary on node 3, then the rest. From my Mac, ports 22, 16443, 443 and 80 were open and 10250, 19001 and 25000 blocked on every node. A drained node rebooted with the firewall active and rejoined. All green by about two in the afternoon.

Now look at the first line of nets: 127.0.0.0/8. That line was not in the original. The auto host endpoints cover every interface, including loopback, so the final Deny dropped localhost traffic. Calico’s own liveness check (calico-node -felix-live, which calls localhost:9099) timed out and Kubernetes kept restarting the pod.

Why did every test pass? The existing dataplane kept forwarding traffic, so pods ran, ports were blocked correctly and a reboot worked. The thing I had changed was failing in the one place I was not watching, and restarts only appear after the first failed probes. At around nine that night I spotted the restarts in Headlamp.

The fix was one line, adding 127.0.0.0/8 to the allowed sources. I pushed it, deleted one pod stuck in backoff and watched for 14 to 16 minutes with no restarts.

Here is the lesson, now a rule in the repo: after any firewall change, watch the component you changed for longer than its probe period, not just the things it protects. Ten minutes of calico-node is now a written step.

Working with the Agent

I need to be fair here, in both directions.

I asked in plain English: "Security hardening: prevent pods from accessing outside the cluster in my LAN" plus minimal audit logging. The agent proposed the controls, wrote the manifests, ran probe pods before and after, and did the node-by-node rollouts over SSH. It also vendored Calico’s CRD schemas so CI could validate the new objects, because the public catalogue lacks them.

It got a lot right: the lan-guard narrowing, and the Secrets rollout order (read everywhere, then write, then rewrite).

Where it was wrong was verification. It declared the host firewall verified on port scans and a reboot, which are the tests I would have asked for too. I did not catch it either; it took me glancing at Headlamp seven hours later, and I had already set a second agent loose to fix it. The agent laid out the loopback cause and checked the repo for parallel changes before touching anything. I told the other agent to stand down.

Then it wrote the incident into the docs, with the 10-minute watch as a hard rule. In my opinion, an agent that admits fault plainly and writes it down beats one that is defensive.

Wrapping Up

The cluster now has labelled namespaces, default-deny policies, a LAN guard, encrypted Secrets, a minimal audit trail and a host firewall that no longer breaks Calico. It costs almost no memory, but it does cost you a rule: watch what you changed, for longer than you think.

In Part 7 I look at keeping it patched without babysitting it, and the surprise of finding I had flashed the desktop image onto three servers.

As always, feel free to reach out with any questions or comments!

Up Next

  1. Rebuilding My Desk Kubernetes Cluster, Five Years On
  2. GitOps and Secrets with Flux and SOPS
  3. Workload Identity Without AKS
  4. Entra ID for kubectl, Five Years After Okta
  5. Ingress, Certificates and Getting Home Through CGNAT
  6. Hardening a Home Cluster, and How I Broke It (this post)
  7. Keeping It Patched Without Babysitting It
  8. One LLM Gateway for Everything, with a Hard $100 Cap
  9. What I Learnt Building It with an AI Agent

Until next time, stay cloudy!

Leave a Comment