Operations on KubeClaw

Scale Up/Down

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers scaling your cluster at both the infrastructure and Kubernetes levels.

Scale Infrastructure

Add worker nodes

Edit terraform.tfvars – update worker_node_types:

worker_node_types = [
 { type = "cx23", count = 2 },
 { type = "cx33", count = 1 }, # 3 workers total, mixed types
]

Apply changes:
```
tofu apply
```

Regenerate Ansible inventory:

./scripts/generate-ansible-inventory.sh

Run Ansible playbooks on the new nodes – see Server Management (Ansible) for the full workflow.

Add replica control nodes

Same pattern using control_node_types:

SSH Key Rotation

Mon, 01 Jan 0001 00:00:00 +0000

When to rotate

Periodically (e.g. annually)
If compromise is suspected
When personnel changes occur

With auto-generated keys

# 1. Mark old key resources for recreation
tofu taint 'tls_private_key.control_node[0]'
tofu taint 'tls_private_key.worker_node[0]'

# 2. Generate new keys and update servers
tofu apply

# 3. Export new keys
./scripts/setup-ssh.sh

Warning

During rotation SSH access may be briefly interrupted. Keep Hetzner web console root access available as a fallback.

With custom keys

# 1. Create new keys
ssh-keygen -t ed25519 -f ~/.ssh/k8s-control-new -C "control-node-new"

# 2. Add the new public key to the server (before removing the old one)
ssh control-node
echo "ssh-ed25519 AAAA... control-node-new" >> ~/.ssh/authorized_keys

# 3. Test the new key
ssh -i ~/.ssh/k8s-control-new kubernetes-admin@<node-ip>

# 4. Remove the old key from authorized_keys
ssh -i ~/.ssh/k8s-control-new control-node
# Edit ~/.ssh/authorized_keys and remove the old key line

# 5. Update terraform.tfvars with the new public key
# control_node_public_key = "ssh-ed25519 AAAA... (new key)"

# 6. Sync OpenTofu state
tofu apply

Repeat for worker node keys if applicable.

Kubernetes Maintenance

Mon, 01 Jan 0001 00:00:00 +0000

Operational procedures for upgrading and managing the Kubernetes cluster. For initial cluster setup, see the Kubernetes (kubeadm) guide.

Upgrade Kubernetes

Kubernetes upgrades follow a strict order: control plane first, then workers. This is the standard CKA upgrade workflow.

Upgrade control plane

# 1. Unhold packages
sudo apt-mark unhold kubeadm

# 2. Upgrade kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm=1.33.*-*

# 3. Check available upgrade
sudo kubeadm upgrade plan

# 4. Apply the upgrade
sudo kubeadm upgrade apply v1.33.0

# 5. Drain the control node (if running workloads)
kubectl drain $(hostname) --ignore-daemonsets --delete-emptydir-data

# 6. Upgrade kubelet and kubectl
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.33.*-* kubectl=1.33.*-*
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# 7. Uncordon the node
kubectl uncordon $(hostname)

Upgrade worker nodes

On each worker node:

Security

Mon, 01 Jan 0001 00:00:00 +0000

Design intent, not a security guarantee

This page describes what the design *intends* to protect against. KubeClaw is a [prototype](/kubeclaw/docs/project-status/) and has not been adversarially tested or audited -- assume gaps exist. Treat the table below as a statement of intent, not of assurance.

Security Summary

Layer	Protection
Network (Hetzner)	Firewall blocks all inbound; IPv6-only, no public IPv4
Network (K8s)	Cilium egress whitelist per namespace/app (FQDN-based)
Access	Cloudflare Tunnel (outbound-only connection, no open ports)
Authentication	Cloudflare Access policies + SSH key-only auth
Container	Non-root user, dropped capabilities, resource limits
SSH	Key-only auth, fail2ban, no TCP forwarding on workers
Storage	Isolated PVCs per workload

What this setup protects against

Direct server attacks – no public IPs, no open inbound ports
Unauthorized access – Cloudflare Access + SSH key-only auth
Data exfiltration – FQDN-based egress whitelist per application
Lateral movement – namespace isolation, per-pod network policies
Resource abuse – container resource limits

What to monitor

API key and token compromise – rotate regularly
Cloudflare Tunnel health – monitor via Zero Trust dashboard
Node resource utilization – watch for memory pressure on small instances

Security notes

Passwords in cloud-init are visible in cloud-init logs. Change them after first login.
SSH keys should be different for each server role.
Root password is only intended for emergency access via Hetzner Web Console.
Terraform state contains sensitive data (private keys when auto-generated). Protect state files.

UFW rules for HTTP/HTTPS on worker nodes can be removed after initial setup:

sudo ufw delete allow out to any port 80 proto tcp
sudo ufw delete allow out to any port 443 proto tcp

OpenClaw-specific security

When running OpenClaw:

Password Management

Mon, 01 Jan 0001 00:00:00 +0000

Which credentials exist?

Credential	Purpose	Storage
Hetzner API Token	Create infrastructure	`terraform.tfvars`
Root password	Emergency web console	`terraform.tfvars`
SSH private keys	Server access	`~/.ssh/` or password manager
Cloudflare Tunnel token	Tunnel auth	Cloudflare Dashboard

Recommended Dashlane structure

📁 K8s Cluster
├── 🔐 Hetzner API Token
│ └── Token: xxx
├── 🔐 Root Password
│ └── Password: xxx
├── 📝 SSH Keys (Secure Note)
│ ├── Control Node Private Key: ...
│ ├── Control Node Public Key: ...
│ ├── Worker Node Private Key: ...
│ └── Worker Node Public Key: ...
└── 🔐 Cloudflare Tunnel Token
 └── Token: xxx

Securing terraform.tfvars

terraform.tfvars contains sensitive data. Options:

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Infrastructure Issues

cloudflared won’t start (IPv6-only)

Check /etc/cloudflared/config.yml:

edge-ip-version: "6"

Important

The value must be a string in quotes (`"6"`, not `6`). Cloudflared defaults to IPv4 connections to Cloudflare's edge servers.

SSH key rotation failed

Connect via Hetzner web console (root password)

Add the new key manually:

echo "ssh-ed25519 AAAA..." >> /home/kubernetes-admin/.ssh/authorized_keys

Ansible cannot connect

Check:

Is cloudflared installed locally?
Is the tunnel running? (cloudflared tunnel list)
Is the inventory correct? (./scripts/generate-ansible-inventory.sh)
Is the ssh-agent running with keys loaded? (ssh-add -l)

State lost / keys gone

With auto-generated keys: