Operations

• 1: Scale Up/Down
• 2: SSH Key Rotation
• 3: Kubernetes Maintenance
• 4: Security
• 5: Password Management
• 6: Troubleshooting

1 - Scale Up/Down

This guide covers scaling your cluster at both the infrastructure and Kubernetes levels.

Scale Infrastructure

Add worker nodes

1. Edit terraform.tfvars – update worker_node_types:

   worker_node_types = [
     { type = "cx23", count = 2 },
     { type = "cx33", count = 1 },  # 3 workers total, mixed types
   ]
   

2. Apply changes:

   tofu apply
   

3. Regenerate Ansible inventory:

   ./scripts/generate-ansible-inventory.sh
   

4. Run Ansible playbooks on the new nodes – see Server Management (Ansible) for the full workflow.

Add replica control nodes

Same pattern using control_node_types:

control_node_types = [
  { type = "cx23", count = 2 },  # 2 replicas → 3 total control nodes
]

Then tofu apply, regenerate inventory, and run Ansible playbooks.

Remove nodes

1. Drain and remove from Kubernetes first (see Remove nodes from Kubernetes below)
2. Update terraform.tfvars to reduce node counts
3. Run tofu apply

Scale Kubernetes

Join new worker nodes

After provisioning and running Ansible playbooks on new nodes, join them to the cluster. Follow Step 2 of the Kubernetes guide to run the kubeadm join command.

Remove nodes from Kubernetes

Before removing infrastructure, drain and delete the node from Kubernetes:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
kubectl delete node <node-name>

Then update terraform.tfvars and run tofu apply to remove the infrastructure.

Examples

Master-only (no replicas, no workers)

# terraform.tfvars
master_control_node_type = "cx23"
control_node_types = []
worker_node_types  = []

Add 2 workers

# terraform.tfvars
worker_node_types = [
  { type = "cx23", count = 2 },
]

Mixed server types

# terraform.tfvars
control_node_types = [
  { type = "cx23", count = 2 },
]
worker_node_types = [
  { type = "cx23", count = 2 },
  { type = "cx33", count = 1 },
]

2 - SSH Key Rotation

When to rotate

• Periodically (e.g. annually)
• If compromise is suspected
• When personnel changes occur

With auto-generated keys

# 1. Mark old key resources for recreation
tofu taint 'tls_private_key.control_node[0]'
tofu taint 'tls_private_key.worker_node[0]'

# 2. Generate new keys and update servers
tofu apply

# 3. Export new keys
./scripts/setup-ssh.sh

With custom keys

# 1. Create new keys
ssh-keygen -t ed25519 -f ~/.ssh/k8s-control-new -C "control-node-new"

# 2. Add the new public key to the server (before removing the old one)
ssh control-node
echo "ssh-ed25519 AAAA... control-node-new" >> ~/.ssh/authorized_keys

# 3. Test the new key
ssh -i ~/.ssh/k8s-control-new kubernetes-admin@<node-ip>

# 4. Remove the old key from authorized_keys
ssh -i ~/.ssh/k8s-control-new control-node
# Edit ~/.ssh/authorized_keys and remove the old key line

# 5. Update terraform.tfvars with the new public key
# control_node_public_key = "ssh-ed25519 AAAA... (new key)"

# 6. Sync OpenTofu state
tofu apply

Repeat for worker node keys if applicable.

3 - Kubernetes Maintenance

Operational procedures for upgrading and managing the Kubernetes cluster. For initial cluster setup, see the Kubernetes (kubeadm) guide.

Upgrade Kubernetes

Kubernetes upgrades follow a strict order: control plane first, then workers. This is the standard CKA upgrade workflow.

Upgrade control plane

# 1. Unhold packages
sudo apt-mark unhold kubeadm

# 2. Upgrade kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm=1.33.*-*

# 3. Check available upgrade
sudo kubeadm upgrade plan

# 4. Apply the upgrade
sudo kubeadm upgrade apply v1.33.0

# 5. Drain the control node (if running workloads)
kubectl drain $(hostname) --ignore-daemonsets --delete-emptydir-data

# 6. Upgrade kubelet and kubectl
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.33.*-* kubectl=1.33.*-*
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# 7. Uncordon the node
kubectl uncordon $(hostname)

Upgrade worker nodes

On each worker node:

# 1. From the control plane: drain the worker
kubectl drain <worker-name> --ignore-daemonsets --delete-emptydir-data

# 2. On the worker: upgrade packages
sudo apt-mark unhold kubeadm kubelet kubectl
sudo apt-get update && sudo apt-get install -y kubeadm=1.33.*-* kubelet=1.33.*-* kubectl=1.33.*-*
sudo apt-mark hold kubeadm kubelet kubectl

# 3. Upgrade node config
sudo kubeadm upgrade node

# 4. Restart kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# 5. From the control plane: uncordon the worker
kubectl uncordon <worker-name>

Backup PVC data

# Create snapshot via Hetzner Console or API
# Hetzner Console → Volumes → Select volume → Create Snapshot

Useful kubectl commands

# Check nodes (dual-stack IPs visible)
kubectl get nodes -o wide

# Check all pods across namespaces
kubectl get pods -A

# Check pod IPs (should show both IPv4 and IPv6)
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.name}: {.status.podIPs}{"\n"}{end}'

# Check storage
kubectl get pvc -A
kubectl get pv

# Check CSI driver
kubectl get pods -n kube-system | grep hcloud

# Check Cilium status
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status

# Check network policies
kubectl get ciliumnetworkpolicies -A

# Check FQDN DNS cache (verify DNS64 synthesized addresses are cached)
kubectl exec -n kube-system -it \
  $(kubectl get pods -n kube-system -l k8s-app=cilium -o name | head -1) \
  -- cilium fqdn cache list

# Logs for cloudflared (if running as K8s workload)
kubectl logs -n system-unrestricted -l app=cloudflared

# Restart a workload
kubectl rollout restart deployment/my-app -n apps-restricted

4 - Security

Security Summary

What this setup protects against

• Direct server attacks – no public IPs, no open inbound ports
• Unauthorized access – Cloudflare Access + SSH key-only auth
• Data exfiltration – FQDN-based egress whitelist per application
• Lateral movement – namespace isolation, per-pod network policies
• Resource abuse – container resource limits

What to monitor

• API key and token compromise – rotate regularly
• Cloudflare Tunnel health – monitor via Zero Trust dashboard
• Node resource utilization – watch for memory pressure on small instances

Security notes

• Passwords in cloud-init are visible in cloud-init logs. Change them after first login.
• SSH keys should be different for each server role.
• Root password is only intended for emergency access via Hetzner Web Console.
• Terraform state contains sensitive data (private keys when auto-generated). Protect state files.
• UFW rules for HTTP/HTTPS on worker nodes can be removed after initial setup:

  sudo ufw delete allow out to any port 80 proto tcp
  sudo ufw delete allow out to any port 443 proto tcp
  

OpenClaw-specific security

When running OpenClaw:

SSH hardening details

All nodes are configured with:

• PermitRootLogin no
• PasswordAuthentication no
• KbdInteractiveAuthentication no
• MaxAuthTries 3
• X11Forwarding no
• AllowAgentForwarding no
• AllowUsers kubernetes-admin
• ClientAliveInterval 300
• ClientAliveCountMax 2

Control nodes additionally allow AllowTcpForwarding yes (needed for ProxyJump and tunnel). Worker nodes set AllowTcpForwarding no.

5 - Password Management

Which credentials exist?

Recommended Dashlane structure

📁 K8s Cluster
├── 🔐 Hetzner API Token
│   └── Token: xxx
├── 🔐 Root Password
│   └── Password: xxx
├── 📝 SSH Keys (Secure Note)
│   ├── Control Node Private Key: ...
│   ├── Control Node Public Key: ...
│   ├── Worker Node Private Key: ...
│   └── Worker Node Public Key: ...
└── 🔐 Cloudflare Tunnel Token
    └── Token: xxx

Securing terraform.tfvars

terraform.tfvars contains sensitive data. Options:

1. Do not commit: exclude via .gitignore (default)
2. Encrypt: with git-crypt or sops
3. Use environment variables instead of tfvars:

   export TF_VAR_hcloud_token="xxx"
   export TF_VAR_root_password="xxx"
   

6 - Troubleshooting

Infrastructure Issues

cloudflared won’t start (IPv6-only)

Check /etc/cloudflared/config.yml:

edge-ip-version: "6"

SSH key rotation failed

1. Connect via Hetzner web console (root password)
2. Add the new key manually:

   echo "ssh-ed25519 AAAA..." >> /home/kubernetes-admin/.ssh/authorized_keys
   

Ansible cannot connect

Check:

1. Is cloudflared installed locally?
2. Is the tunnel running? (cloudflared tunnel list)
3. Is the inventory correct? (./scripts/generate-ansible-inventory.sh)
4. Is the ssh-agent running with keys loaded? (ssh-add -l)

State lost / keys gone

With auto-generated keys:

1. Connect via web console (root)
2. Create new keys
3. Add them to authorized_keys
4. Import servers into new state:

   tofu import hcloud_server.master_control_node <server-id>
   

SSH connection via tunnel fails

Checklist:

1. Is cloudflared installed locally? (which cloudflared)
2. Is the ProxyCommand path correct?
3. Is the Cloudflare Access Application configured?
4. Is the tunnel shown as “Connected” in Cloudflare?

Hetzner Web Console does not work

• Use the “Send Clipboard” button above the console
• Use a simple password without special characters for the initial login
• Alternative: Create a temporary Admin Node with a public IP

Cloud-Init password does not work

Cause: The old chpasswd.list syntax is deprecated.

Solution: Use the new syntax:

users:
  - name: root
    plain_text_passwd: 'your-password'
    lock_passwd: false

Kubernetes Issues

Worker nodes not joining

# On the worker node, check kubelet logs
journalctl -xeu kubelet

# Common issues:
# - Swap not disabled: swapoff -a
# - containerd not running: systemctl status containerd
# - Port 6443 not reachable: curl -k https://10.0.0.2:6443
# - Token expired (24h default): kubeadm token create --print-join-command

Cilium pods not ready

# Check Cilium status
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status

# Check Cilium pod logs
kubectl logs -n kube-system -l k8s-app=cilium

# Check Cilium pods
kubectl get pods -n kube-system -l k8s-app=cilium

Network policy blocking traffic unexpectedly

# Check which policies apply
kubectl get ciliumnetworkpolicies -n apps-restricted

# Check if FQDN rules are resolving
kubectl exec -n kube-system -it \
  $(kubectl get pods -n kube-system -l k8s-app=cilium -o name | head -1) \
  -c cilium-agent -- cilium-dbg fqdn cache list

CSI volume not attaching

kubectl describe pvc -n apps-restricted
kubectl get events -n apps-restricted --sort-by='.lastTimestamp'

Pod not starting (general)

kubectl describe pod -n <namespace> <pod-name>
kubectl logs -n <namespace> <pod-name> --previous

OpenClaw Pod not starting

kubectl describe pod -n apps-restricted -l app=openclaw
kubectl logs -n apps-restricted -l app=openclaw --previous

Cloudflare Tunnel not connecting (K8s workload)

kubectl logs -n system-unrestricted -l app=cloudflared