Raspberry Co-PI-lot (Part 5): Ingress, Certificates and Getting Home Through CGNAT

Recently I got to the point of publishing apps on the Pi cluster, and I discovered that the ingress story for Kubernetes had changed underneath me. The Kubernetes project announced that Ingress NGINX would be retired in March 2026, with no further releases, bug fixes or security patches after that. In the same window, MicroK8s 1.35 switched its ingress add-on to Traefik. Perfect timing, I thought.

It was perfect timing right up until I found out what the add-on does by default on my kernel, and right up until I tried to reach the thing from outside my home network. I built this cluster in about a day by directing an AI coding agent in my terminal, so this post also covers where my co-pilot earned its keep.

This is Part 5 of the Raspberry Co-PI-lot series. Part 1 built the three-node MicroK8s cluster, Part 2 added Flux and SOPS, Part 3 covered workload identity (which is what lets cert-manager write DNS records without a stored secret) and Part 4 put Entra ID in front of kubectl. The full list is at the bottom of this post.

This post will cover why the add-on’s default exposure does not work on these Pis, how I exposed Traefik instead, how the Gateway API and wildcard certificates fit together, a small private registry (Zot) that uses all of it, and how I get home through CGNAT. Let’s dive in!

In English Please?

Think of the ingress layer as the front desk of a building. Visitors (browsers) arrive at one door, tell the receptionist (Traefik) who they want (the hostname), and get pointed to the right office (a Service). The padlock is the receptionist’s ID badge: a TLS certificate proving the building is who it says it is.

The old way to describe the desk rules was the Ingress object. The newer way is the Gateway API, which splits the job in two: a Gateway (the doors and which certificates they present) and HTTPRoute objects (which hostnames go to which app). I would rather learn the thing the ecosystem is heading towards than the thing that was just retired.

The path of a request on my cluster:

  1. DNS returns the three node IPs for test.lab.example.com.
  2. The client connects to port 443 on one of them.
  3. kube-proxy forwards that to a Traefik pod.
  4. Traefik picks the right wildcard certificate, then routes by hostname to the app.

Why hostPort Does Not Work Here

The MicroK8s add-on runs Traefik as a DaemonSet and, by default, binds ports 80 and 443 on each node with hostPort. On my cluster those pods sat in ContainerCreating with can't initialize iptables table 'nat'.

You may recognise the culprit from Part 1. The Raspberry Pi kernel on these nodes has no legacy ip_tables module, which is why kube-proxy runs in nftables mode. The hostPort feature is implemented by the CNI portmap plugin, and the Calico build shipped with MicroK8s only speaks legacy iptables. This is what I observed on my kernel and MicroK8s version, not something I found documented.

The fix is to skip hostPort entirely. Kubernetes lets you give a Service externalIPs: addresses on which kube-proxy will accept traffic for that Service, if they reach a node. I set them to the three node addresses, so every node answers on 80 and 443 and kube-proxy (already in nftables mode) forwards to any Traefik pod. There is no load balancer and no floating IP; DNS simply returns all three addresses.

Here is the full set of Traefik values from my Flux HelmRelease, with the overrides that matter:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: traefik
  namespace: ingress
spec:
  interval: 1h
  releaseName: traefik
  chart:
    spec:
      chart: traefik
      version: 39.0.8
      sourceRef:
        kind: HelmRepository
        name: traefik
        namespace: flux-system
  install:
    crds: Skip
  upgrade:
    crds: Skip
  values:
    deployment:
      kind: DaemonSet
    ports:
      web:
        port: 8000
        exposedPort: 80
        protocol: TCP
        http:
          redirections:
            entryPoint:
              to: websecure
              scheme: https
              permanent: true
      websecure:
        port: 8443
        exposedPort: 443
        protocol: TCP
        transport:
          respondingTimeouts:
            readTimeout: 600s
    service:
      type: ClusterIP
      externalIPs:
        - 192.168.1.x   # node 1
        - 192.168.1.x   # node 2
        - 192.168.1.x   # node 3
    ingressRoute:
      dashboard:
        enabled: false
    gateway:
      listeners:
        web:
          port: 8000
          protocol: HTTP
          namespacePolicy:
            from: All
        websecure-prod:
          port: 8443
          protocol: HTTPS
          hostname: "*.example.com"
          certificateRefs:
            - name: wildcard-example-com-tls
          namespacePolicy:
            from: All
        websecure-lab:
          port: 8443
          protocol: HTTPS
          hostname: "*.lab.example.com"
          certificateRefs:
            - name: wildcard-lab-example-com-tls
          namespacePolicy:
            from: All
    providers:
      kubernetesIngress:
        enabled: false
      kubernetesGateway:
        enabled: true
      kubernetesIngressNginx:
        enabled: false
      kubernetesCRD:
        enabled: false
    ingressClass:
      enabled: false
    resources:
      requests: { cpu: 25m, memory: 64Mi }
      limits: { memory: 192Mi }

The service block is the externalIPs fix. The dashboard is off because the add-on enables an unauthenticated one by default. The crds: Skip lines are the Part 2 gotcha: the add-on owns the Gateway API CRDs. The 600 second readTimeout is for the registry, below.

Be warned: re-running microk8s enable ingress resets the release to the add-on defaults, hostPort included. It is now a hard rule in the repo’s AGENTS.md, the file of rules the agent reads: never run it again.

Gateway API, Wildcards and Publishing an App

Once Traefik was running, I turned off everything except the Gateway API provider. The Ingress provider, the NGINX-annotation compatibility provider, the Traefik CRD provider and the add-on’s IngressClasses are all disabled. Nothing needed them, and fewer ways to publish a service means fewer surprises.

The Gateway has two HTTPS listeners, *.example.com and *.lab.example.com (for experiments), and port 80 redirects everything to HTTPS.

Publishing an app is three small steps. Here is the route for whoami, my canary: one request to it exercises DNS, kube-proxy, Traefik, the listener and the certificate at once.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: whoami
  namespace: whoami
spec:
  parentRefs:
    - name: traefik-gateway
      namespace: ingress
      sectionName: websecure-lab
  hostnames:
    - test.lab.example.com
  rules:
    - backendRefs:
        - name: whoami
          port: 80

The parentRefs attach the route to the lab listener. The last step is a DNS record; I create A records by hand in Azure DNS (the cluster has no write access to those zones, deliberately), one per node IP:

for ip in 192.168.1.x 192.168.1.x 192.168.1.x; do   # your three node IPs
  az network dns record-set a add-record \
    -g rg-<name> -z lab.example.com -n test -a "$ip"
done

Yes, my public DNS contains private addresses. In my opinion that is fine: the same name and certificate then work at home and over a VPN. If your router strips private answers, allow the domain.

Verify the whole chain with one command:

curl -sf https://test.lab.example.com

The certificates come from cert-manager and Let’s Encrypt. Wildcard certificates require the DNS-01 challenge, where you prove control of the domain by publishing a TXT record rather than answering an HTTP request. That suits a cluster nobody can reach from the internet. Part 3 covered how cert-manager authenticates to Azure and why it can only write to a dedicated acme.example.com zone.

On the ingress side that is two Certificate objects, one per listener:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-example-com
  namespace: ingress
spec:
  secretName: wildcard-example-com-tls
  issuerRef: { name: letsencrypt-prod, kind: ClusterIssuer }
  dnsNames: ["*.example.com"]
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-lab-example-com
  namespace: ingress
spec:
  secretName: wildcard-lab-example-com-tls
  issuerRef: { name: letsencrypt-prod, kind: ClusterIssuer }
  dnsNames: ["*.lab.example.com", "lab.example.com"]

The secret names match the certificateRefs on the Gateway listeners, so every app on the lab domain gets a trusted certificate without its own request. I left the apex example.com out of the production certificate because the main domain is served elsewhere.

Note: a wildcard covers one label only. myapp.lab.example.com is covered; foo.bar.lab.example.com is not. And per the Part 3 constraint, a standalone certificate for any name without an _acme-challenge CNAME into the challenge zone will sit in pending forever.

A Private Registry for Next to Nothing

With ingress working, I wanted somewhere to push images I build on my Mac. The MicroK8s registry add-on has no authentication and plain HTTP, so I rejected it. Zot, an OCI-native registry, turned out to be an excellent fit for a Pi: the zot-minimal image sat at about 25 Mi of memory in my cluster.

It is a Flux-managed Helm chart with a 10Gi volume, htpasswd authentication from a SOPS secret and two users: pusher (read and write, for my workstation) and puller (read only, for the cluster). No anonymous access. It sits behind the websecure-lab listener, so registry.lab.example.com already has a trusted certificate and the nodes need no extra configuration.

Two small lessons. First, Traefik’s default 60 second read timeout covers the whole request body, which breaks large layer pushes; that is why I raised readTimeout to 600 seconds in the values above. With that in place, a push of about 45 MB took 18 seconds. Second, a pod without a pull secret got ImagePullBackOff, which is exactly what I wanted to see.

A word from the wise: the registry’s data lives on a single node and is not backed up. Everything in it should be rebuildable from source.

Working With the Agent: Where I Had to Steer

I describe an outcome in plain English, my co-pilot does the typing, and I make the calls. Here are a few moments from this part.

The agent did the research on the ingress-nginx retirement and the MicroK8s add-on change, found that hostPort could not work on this kernel, and moved Traefik to externalIPs. That was the right call. It also wrote the "never re-run microk8s enable ingress" rule into AGENTS.md, so a future session cannot undo the fix by accident. My job was to read what it proposed and decide whether I could live with it.

It was wrong in a small way on the redirect. Its first comment said Traefik answers with a 308; when we tested it, Traefik answered 301. The behaviour was fine, but the comment was a confident claim nobody had checked. The agent is quick, but it is only as trustworthy as the evidence it gathers.

The remote access piece is where I steered most. I was away from home, which is when it hit me that I could not reach any of this, so I asked the agent what my options were. It checked the gateway, tested the candidates and wrote up the trade-offs; I picked the one I could live with.

Getting Home Through CGNAT

My router is a UniFi Gateway Pro Max on Aussie Broadband. The agent checked the WAN address, and it was in 100.64.0.0/10, the range set aside for carrier-grade NAT (RFC 6598). In plain terms, my ISP shares one public address between many customers, so nothing can start a connection to my router. A WireGuard server on the gateway, or a port forward, simply cannot work. That is what I observed.

UniFi Teleport is where this gets workable. Ubiquiti describe it as a VPN built on WireGuard that needs no configuration, and state that it can be used when the gateway is behind NAT. You enable it in the UniFi Network settings, generate an invitation, and accept it in the WiFiman app (mobile or desktop). Their docs note the invitation expires in 24 hours and works on a single device at a time. They also note Teleport requires IPv6 on the gateway’s WAN in some circumstances, which is worth knowing if it misbehaves for you.

I tested it from a phone hotspot, which was a fair stand-in for a hostile network:

  • First attempt failed. The tunnel interfaces came up but had no IPv4 address, and traffic went straight out the hotspot. Reconnecting fixed it.
  • After that it worked. The HTTPS canary returned 200, and kubectl and SSH both worked.
  • It is a full tunnel. All traffic goes via home, and DNS still came from the hotspot, which was fine because my names resolve in public DNS.
  • It dropped twice during testing. In my opinion that is tolerable for a lab.

The fallback, documented but not installed, is a Tailscale subnet router on the host OS of two nodes, advertising only the node IPs.

Nothing on the cluster is exposed to the internet, and the router is never port-forwarded.

Wrapping Up

The cluster now serves HTTPS on trusted wildcard certificates through Traefik and the Gateway API, hosts its own private registry, and I can reach it from anywhere without opening a single inbound port. The unexpected cost was learning that a default which works on most clusters cannot work on mine.

Next up is the part where I lock the whole thing down, and spectacularly break it.

As always, feel free to reach out with any questions or comments!

Up Next

  1. Rebuilding My Desk Kubernetes Cluster, Five Years On
  2. GitOps and Secrets with Flux and SOPS
  3. Workload Identity Without AKS
  4. Entra ID for kubectl, Five Years After Okta
  5. Ingress, Certificates and Getting Home Through CGNAT (this post)
  6. Hardening a Home Cluster, and How I Broke It
  7. Keeping It Patched Without Babysitting It
  8. One LLM Gateway for Everything, with a Hard $100 Cap
  9. What I Learnt Building It with an AI Agent

Until next time, stay cloudy!

Leave a Comment