Troubleshooting

Infrastructure Issues

cloudflared won’t start (IPv6-only)

Check /etc/cloudflared/config.yml:

edge-ip-version: "6"

SSH key rotation failed

1. Connect via Hetzner web console (root password)
2. Add the new key manually:

   echo "ssh-ed25519 AAAA..." >> /home/kubernetes-admin/.ssh/authorized_keys
   

Ansible cannot connect

Check:

1. Is cloudflared installed locally?
2. Is the tunnel running? (cloudflared tunnel list)
3. Is the inventory correct? (./scripts/generate-ansible-inventory.sh)
4. Is the ssh-agent running with keys loaded? (ssh-add -l)

State lost / keys gone

With auto-generated keys:

1. Connect via web console (root)
2. Create new keys
3. Add them to authorized_keys
4. Import servers into new state:

   tofu import hcloud_server.master_control_node <server-id>
   

SSH connection via tunnel fails

Checklist:

1. Is cloudflared installed locally? (which cloudflared)
2. Is the ProxyCommand path correct?
3. Is the Cloudflare Access Application configured?
4. Is the tunnel shown as “Connected” in Cloudflare?

Hetzner Web Console does not work

• Use the “Send Clipboard” button above the console
• Use a simple password without special characters for the initial login
• Alternative: Create a temporary Admin Node with a public IP

Cloud-Init password does not work

Cause: The old chpasswd.list syntax is deprecated.

Solution: Use the new syntax:

users:
  - name: root
    plain_text_passwd: 'your-password'
    lock_passwd: false

Kubernetes Issues

Worker nodes not joining

# On the worker node, check kubelet logs
journalctl -xeu kubelet

# Common issues:
# - Swap not disabled: swapoff -a
# - containerd not running: systemctl status containerd
# - Port 6443 not reachable: curl -k https://10.0.0.2:6443
# - Token expired (24h default): kubeadm token create --print-join-command

Cilium pods not ready

# Check Cilium status
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status

# Check Cilium pod logs
kubectl logs -n kube-system -l k8s-app=cilium

# Check Cilium pods
kubectl get pods -n kube-system -l k8s-app=cilium

Network policy blocking traffic unexpectedly

# Check which policies apply
kubectl get ciliumnetworkpolicies -n apps-restricted

# Check if FQDN rules are resolving
kubectl exec -n kube-system -it \
  $(kubectl get pods -n kube-system -l k8s-app=cilium -o name | head -1) \
  -c cilium-agent -- cilium-dbg fqdn cache list

CSI volume not attaching

kubectl describe pvc -n apps-restricted
kubectl get events -n apps-restricted --sort-by='.lastTimestamp'

Pod not starting (general)

kubectl describe pod -n <namespace> <pod-name>
kubectl logs -n <namespace> <pod-name> --previous

OpenClaw Pod not starting

kubectl describe pod -n apps-restricted -l app=openclaw
kubectl logs -n apps-restricted -l app=openclaw --previous

Cloudflare Tunnel not connecting (K8s workload)

kubectl logs -n system-unrestricted -l app=cloudflared