KubeClaw

Project Status

Mon, 01 Jan 0001 00:00:00 +0000

KubeClaw is a prototype

KubeClaw is a learning project. Its objective is to explore what a secure, scalable environment for running AI agents on Kubernetes actually requires – by building one end to end and finding out where the difficulties are.

It is not production ready. It is not beta. There is no supported release, no stability guarantee, and no security review.

What that means concretely

	Status
Maturity	Prototype / experiment
Suitable for production	No
Suitable for handling real secrets or customer data	No
Security reviewed or audited	No
API / variable stability	None – variables and layouts change without notice
Support	None – issues may go unanswered
Breaking changes	Expected, without a deprecation period

Why it exists

Agentic AI workloads execute arbitrary code with tool access. Running them without an isolation boundary is genuinely risky, and the interesting question is what a correct boundary looks like: which network controls actually hold, how much egress restriction is practical, what the operational cost is, and where the design breaks under load.

Quick Start

Mon, 01 Jan 0001 00:00:00 +0000

This provisions real, billable infrastructure

KubeClaw is a [prototype](/kubeclaw/docs/project-status/) -- a learning project, not production software. The steps below create servers on Hetzner Cloud that cost real money, and the resulting cluster has not been security reviewed. Do not use it for production traffic, real credentials, or third-party data.

Prerequisites

Accounts

Hetzner Cloud account with an API token (console.hetzner.cloud)
Cloudflare account with a configured domain (free tier is sufficient)

Local Machine

Docker and an IDE with Dev Container support (e.g. VS Code + Dev Containers extension)

That’s it. All project tools (OpenTofu, Ansible, SSH, cloudflared, Hugo, Kubernetes clients, and AI assistants) are available in the Dev Container. No local installation is required beyond Docker and an IDE with Dev Container support.

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Overview

KubeClaw creates a secure, IPv6-only Kubernetes cluster on Hetzner Cloud. The design prioritizes security through network isolation: no public IPv4 addresses, SSH access exclusively via Cloudflare Tunnel, and per-namespace egress control with Cilium network policies.

┌─────────────────────────────────────────────────────────────────┐
│ Internet │
└─────────────────────────────────────────────────────────────────┘
 │ │
 ▼ ▼
 ┌──────────────┐ ┌───────────────┐
 │ Cloudflare │ │ Admin Node │
 │ Tunnel │ │ 10.0.0.254 │
 │ (permanent) │ │ (temporary, │
 └──────┬───────┘ │ public IPv6) │
 │ └───────┬───────┘
 ▼ ▼
┌─────────────────────────────────────────────────────────────────┐
│ Hetzner Cloud │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Private Network (10.0.0.0/24) │ │
│ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ control-01 │ │ control-02+ │ │ │
│ │ │ 10.0.0.2 │ │ 10.0.0.3+ │ │ │
│ │ │ (master, │◄►│ (replicas, │ │ │
│ │ │ cloudflared) │ │ 0-n instances) │ │ │
│ │ └────────┬────────┘ └─────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────┐ │ │
│ │ │ worker-nodes │ │ │
│ │ │ (0-n instances) │ │ │
│ │ └─────────────────┘ │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘

Node Roles

Master Control Node (control-01, 10.0.0.2)

The master control node always exists and serves as:

Cilium Dual-Stack Pod Network

Mon, 01 Jan 0001 00:00:00 +0000

Implemented

This feature has been implemented. The [Kubernetes guide](/kubeclaw/docs/guide/kubernetes/) now uses dual-stack `kubeadm init` and Cilium with `ipv6.enabled=true` as the standard configuration.

Summary

The cluster uses dual-stack networking (IPv4 + IPv6) in the Cilium pod network. Every pod gets both an IPv4 address (from 10.244.0.0/16) for internal cluster communication and an IPv6 address (from fd00:10:244::/48) for external connectivity via DNS64/NAT64.

This eliminates the need for hostNetwork: true on any pod, which means:

Dev Container

Mon, 01 Jan 0001 00:00:00 +0000

The project uses an aibox-generated Dev Container (Debian Trixie) as a self-contained, reproducible environment. Nothing needs to be set up on the host machine beyond Docker and an IDE with Dev Container support.

What the Dev Container Provides

OpenTofu – infrastructure provisioning (Terraform-compatible)
Ansible – server management (runs inside the container, no host install needed)
cloudflared – SSH ProxyCommand via Cloudflare Tunnel
ssh-agent – start inside the container to use passphrase-protected keys with Ansible
AI CLI tools – Codex and the aibox toolchain
Hugo + Docsy – documentation site (./scripts/serve-docs.sh at port 1313)
Persistence – .aibox-home/.ssh/ is bind-mounted read-write by this project, so setup-ssh.sh output, SSH config, and exported keys survive container rebuilds

Step 1: Prepare Persistent Directories

Create the persistent directories before opening the Dev Container:

Development Setup

Mon, 01 Jan 0001 00:00:00 +0000

Getting started

Follow the Dev Container guide to set up your development environment. The aibox Dev Container includes OpenTofu, Ansible, cloudflared, Hugo, Node.js, Kubernetes clients, and AI assistants.

Documentation development

Preview the documentation site locally:

./scripts/serve-docs.sh
# Open http://localhost:1313

When using a remote Dev Container environment, forward port 1313 from your editor to view the preview.

Build the site:

./scripts/build-docs.sh

The build uses Hugo’s strict template and content validation. The first run installs the pinned Docsy asset dependencies locally and initializes the pinned Docsy theme submodule when needed.

Scale Up/Down

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers scaling your cluster at both the infrastructure and Kubernetes levels.

Scale Infrastructure

Add worker nodes

Edit terraform.tfvars – update worker_node_types:

worker_node_types = [
 { type = "cx23", count = 2 },
 { type = "cx33", count = 1 }, # 3 workers total, mixed types
]

Apply changes:
```
tofu apply
```

Regenerate Ansible inventory:

./scripts/generate-ansible-inventory.sh

Run Ansible playbooks on the new nodes – see Server Management (Ansible) for the full workflow.

Add replica control nodes

Same pattern using control_node_types:

SSH Keys with Passphrase

Mon, 01 Jan 0001 00:00:00 +0000

Why use a passphrase?

An SSH key without a passphrase is like a house key without a lock on the key cabinet. If your laptop or key file is stolen, the attacker gains immediate access to your cluster.

Creating keys with a passphrase

# Control node key
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_control-node_key -C "k8s-control"
# Enter a strong passphrase when prompted

# Worker node key
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_worker-node_key -C "k8s-worker"
# Enter a passphrase when prompted

Using ssh-agent

Since Terraform and Ansible cannot directly use encrypted keys, you must use ssh-agent:

Variables Reference

Mon, 01 Jan 0001 00:00:00 +0000

All configurable inputs for the OpenTofu infrastructure. Set these in terraform.tfvars.

Required Variables

Variable	Type	Description
`hcloud_token`	`string`	Hetzner Cloud API Token (sensitive)

Cluster Configuration

Variable	Type	Default	Description
`cluster_name`	`string`	`"k8s-cluster"`	Name prefix for all resources
`location`	`string`	`"fsn1"`	Hetzner Cloud location (`fsn1`, `nbg1`, `hel1`, `ash`, `hil`)

Network Configuration

Variable	Type	Default	Description
`network_ip_range`	`string`	`"10.0.0.0/8"`	IP range for the private network
`subnet_ip_range`	`string`	`"10.0.0.0/24"`	IP range for the subnet
`network_zone`	`string`	`"eu-central"`	Network zone (`eu-central`, `us-east`, `us-west`)
`enable_public_ipv6`	`bool`	`true`	Enable public IPv6 for replica control nodes and worker nodes. The master always has public IPv6 (required for cloudflared). Setting to `false` air-gaps replicas and workers.

Server Configuration

Variable	Type	Default	Description
`server_image`	`string`	`"debian-13"`	Server image to use
`master_control_node_type`	`string`	`"cx23"`	Server type for the master control node (runs cloudflared)
`control_node_types`	`list(object({type, count}))`	`[]`	Server types and counts for replica control nodes
`worker_node_types`	`list(object({type, count}))`	`[]`	Server types and counts for worker nodes

The cost-optimized x86 line is cx23, cx33, cx43, and cx53; the Arm equivalents are cax11 through cax41. See Cost Estimate for specifications, current prices, and the constraints on the Arm line.

Code of Conduct

Mon, 01 Jan 0001 00:00:00 +0000

This project follows the Contributor Covenant v2.1.

Please read the full text at the link above. In summary: be respectful, be constructive, and create a welcoming environment for everyone.

Reporting

If you experience or witness unacceptable behavior, please contact the project maintainer at bnaard@gmx.net.

Infrastructure (OpenTofu)

Mon, 01 Jan 0001 00:00:00 +0000

KubeClaw uses OpenTofu (Terraform-compatible) to provision all infrastructure on Hetzner Cloud. This page walks through the provisioning workflow step by step. See Variables Reference for all configurable options.

Step 1: Configure terraform.tfvars

cp terraform.tfvars.example terraform.tfvars

Edit terraform.tfvars with your settings:

hcloud_token = "your-hcloud-api-token"
root_password = "a-strong-root-password"
cluster_name = "k8s-cluster"

# Optional: auto-configure Cloudflare Tunnel on the master control node
# cloudflare_tunnel_token = "eyJ..."

SSH key options

No configuration needed. OpenTofu generates ED25519 keys and stores them in the state file.

| Advantage | Disadvantage |
|-----------|--------------|
| No manual key creation | State file contains private keys |
| Works out of the box | Keys lost if state is lost |

Create your own keys and reference them in `terraform.tfvars`:

```bash
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_control-node_key -C "control-node"
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_worker-node_key -C "worker-node"
```

```hcl
control_node_public_key = "ssh-ed25519 AAAA... control-node"
worker_node_public_key = "ssh-ed25519 AAAA... worker-node"
```

| Advantage | Disadvantage |
|-----------|--------------|
| Full control over key storage | Manual key management |
| State has no private keys | Must create keys before provisioning |
| Easy password manager integration | |

Step 2: Provision Infrastructure

tofu init # First time only
tofu apply

This creates:

Outputs Reference

Mon, 01 Jan 0001 00:00:00 +0000

All outputs exposed by the OpenTofu configuration. These are consumed by the setup scripts and can be queried manually.

Network Information

Output	Description
`network_id`	ID of the private network
`network_name`	Name of the private network

Cluster Metadata

Output	Description
`cluster_name`	Name of the cluster
`admin_user`	Admin user name
`ssh_key_prefix`	Prefix used for SSH key filenames

Master Control Node

Output	Description
`master_control_node_id`	ID of the master control node
`master_control_node_name`	Name of the master control node
`master_control_node_private_ip`	Private IP of the master control node

All Control Nodes (master + replicas)

Output	Description
`control_node_count`	Total number of control nodes (master + replicas)
`control_node_ids`	IDs of all control nodes
`control_node_names`	Names of all control nodes
`control_node_private_ips`	Private IPs of all control nodes

Worker Nodes

Output	Description
`worker_node_count`	Number of worker nodes
`worker_node_ids`	IDs of worker nodes
`worker_node_names`	Names of worker nodes
`worker_node_private_ips`	Private IPs of worker nodes

Admin Node

Output	Description
`admin_node_id`	ID of the admin node (null if disabled)
`admin_node_name`	Name of the admin node (null if disabled)
`admin_node_ipv6`	Public IPv6 address of the admin node (null if disabled)
`admin_node_private_ip`	Private IP of the admin node (null if disabled)
`enable_admin_node`	Whether the admin node is enabled

SSH Keys

These outputs are sensitive when auto-generated keys are used.

Security Model

Mon, 01 Jan 0001 00:00:00 +0000

KubeClaw’s security is layered across three levels: Hetzner Cloud firewalls, host-level hardening, and Kubernetes network policies. See Architecture for the overall network design.

Firewall Rules (Hetzner)

Hetzner Cloud firewalls are the first line of defense. Each node role has its own firewall:

Control nodes: SSH from private network + localhost (for tunnel), ICMP from private network
Worker nodes: SSH from private network, ICMP from private network
Admin node: SSH from anywhere (temporary), ICMP from private network

Kubernetes ports (6443, 10250, 2379-2380, 30000-32767) are intentionally excluded from the Hetzner firewall and added only when deploying Kubernetes.

SSH Key Rotation

Mon, 01 Jan 0001 00:00:00 +0000

When to rotate

Periodically (e.g. annually)
If compromise is suspected
When personnel changes occur

With auto-generated keys

# 1. Mark old key resources for recreation
tofu taint 'tls_private_key.control_node[0]'
tofu taint 'tls_private_key.worker_node[0]'

# 2. Generate new keys and update servers
tofu apply

# 3. Export new keys
./scripts/setup-ssh.sh

Warning

During rotation SSH access may be briefly interrupted. Keep Hetzner web console root access available as a fallback.

With custom keys

# 1. Create new keys
ssh-keygen -t ed25519 -f ~/.ssh/k8s-control-new -C "control-node-new"

# 2. Add the new public key to the server (before removing the old one)
ssh control-node
echo "ssh-ed25519 AAAA... control-node-new" >> ~/.ssh/authorized_keys

# 3. Test the new key
ssh -i ~/.ssh/k8s-control-new kubernetes-admin@<node-ip>

# 4. Remove the old key from authorized_keys
ssh -i ~/.ssh/k8s-control-new control-node
# Edit ~/.ssh/authorized_keys and remove the old key line

# 5. Update terraform.tfvars with the new public key
# control_node_public_key = "ssh-ed25519 AAAA... (new key)"

# 6. Sync OpenTofu state
tofu apply

Repeat for worker node keys if applicable.

Store SSH Keys in Password Manager

Mon, 01 Jan 0001 00:00:00 +0000

Storing SSH keys in a password manager provides a secure backup that survives hardware failures and makes it easy to restore access from a new machine.

General workflow

This workflow applies to any password manager that supports secure notes or file attachments (Dashlane, 1Password, Bitwarden, etc.).

1. Create keys locally

ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_control-node_key -C "control-node"
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_worker-node_key -C "worker-node"

2. Save to your password manager

Create a secure note or vault entry:

Cloud-Init Templates

Mon, 01 Jan 0001 00:00:00 +0000

Cloud-init templates are located in cloud-init/ and rendered by OpenTofu via templatefile() in main.tf. They configure each server on first boot.

admin-node.yaml.tpl

Purpose: Minimal jump host for initial SSH access.

Used by: hcloud_server.admin_node

Template variables:

Variable	Source
`ssh_public_key`	`local.admin_node_public_key`
`root_password`	`var.root_password`
`admin_user`	`var.admin_user`
`keyboard_layout`	`var.keyboard_layout`

What it configures:

Admin user with sudo NOPASSWD
SSH hardening with AllowTcpForwarding yes (needed for ProxyJump)
fail2ban for SSH protection
UFW allowing public SSH (port 22 from anywhere)

control-node.yaml.tpl

Purpose: Kubernetes control plane node with optional Cloudflare Tunnel.

Cloudflare Tunnel Setup

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks through creating and configuring a Cloudflare Tunnel that provides secure SSH access to your KubeClaw cluster. The tunnel replaces the temporary admin node as the permanent access path – no open ports, no public SSH, outbound-only connectivity.

Why Cloudflare Tunnel?

KubeClaw nodes have no public IPv4 and no inbound SSH ports. Access works through one of two paths:

Path	When to use	How it works
Admin node (temporary)	Initial setup, before tunnel is ready	Jump host with public IPv6; SSH via `ProxyJump`
Cloudflare Tunnel (permanent)	After tunnel is configured	`cloudflared` on the master node connects outbound to Cloudflare’s edge; SSH proxied via `ProxyCommand cloudflared access ssh` on your local machine

After the tunnel is working, you disable the admin node (enable_admin_node = false in terraform.tfvars) and all SSH flows through Cloudflare.

DNS and NAT64

Mon, 01 Jan 0001 00:00:00 +0000

The cluster is IPv6-only – but many services (GitHub CDN, container registries, package repos) are IPv4-only. NAT64/DNS64 provides transparent IPv4 reachability at the network layer, no application changes needed. This page covers how DNS64/NAT64 works, how it integrates with Kubernetes CoreDNS, and how to configure it.

The Problem: IPv4 Internet from IPv6-Only Nodes

The nodes have no public IPv4 addresses. Most internet services (GitHub, Docker Hub, package repos) have IPv4 addresses. How does an IPv6-only node reach them?

Kubernetes Maintenance

Mon, 01 Jan 0001 00:00:00 +0000

Operational procedures for upgrading and managing the Kubernetes cluster. For initial cluster setup, see the Kubernetes (kubeadm) guide.

Upgrade Kubernetes

Kubernetes upgrades follow a strict order: control plane first, then workers. This is the standard CKA upgrade workflow.

Upgrade control plane

# 1. Unhold packages
sudo apt-mark unhold kubeadm

# 2. Upgrade kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm=1.33.*-*

# 3. Check available upgrade
sudo kubeadm upgrade plan

# 4. Apply the upgrade
sudo kubeadm upgrade apply v1.33.0

# 5. Drain the control node (if running workloads)
kubectl drain $(hostname) --ignore-daemonsets --delete-emptydir-data

# 6. Upgrade kubelet and kubectl
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.33.*-* kubectl=1.33.*-*
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# 7. Uncordon the node
kubectl uncordon $(hostname)

Upgrade worker nodes

On each worker node:

Manual Setup (Alternative)

Mon, 01 Jan 0001 00:00:00 +0000

This guide is an alternative to OpenTofu

This guide shows the manual steps that OpenTofu automates. Use this if you want to understand what happens behind the scenes, or if you prefer to set up infrastructure manually via the Hetzner Cloud Console. For the automated approach, see [Infrastructure (OpenTofu)](/kubeclaw/docs/guide/infrastructure/).

Overview

This setup creates a secure server infrastructure with the following properties:

No public IPv4/IPv6 addresses (after setup)
SSH access exclusively via Cloudflare Tunnel
Internal communication via Hetzner Private Network
Hardened SSH configuration with fail2ban and UFW

Architecture

┌─────────────────────────────────────────────────────────────────┐
│ Internet │
└─────────────────────────────────────────────────────────────────┘
 │
 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Cloudflare Tunnel │
│ console.yourdomain.org │
└─────────────────────────────────────────────────────────────────┘
 │
 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Hetzner Cloud │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Private Network (10.0.0.0/24) │ │
│ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ control-node │ │ worker-node │ │ │
│ │ │ 10.0.0.2 │◄─────►│ 10.0.0.3 │ │ │
│ │ │ (cloudflared) │ │ (isolated) │ │ │
│ │ └─────────────────┘ └─────────────────┘ │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘

Prerequisites

Hetzner Cloud account
Cloudflare account with your own domain
macOS/Linux machine with SSH
cloudflared installed locally (brew install cloudflared)

Step 1: Create Hetzner Private Network

Open the Hetzner Cloud Console
Select your project
Go to Networks > Create Network
Configure:
- Name: k8s-network (or any name)
- IP Range: 10.0.0.0/8 (Hetzner requires /8 for the network object)
Click Create Network
Add a Subnet: 10.0.0.0/24 in zone eu-central (this is the actual range used by nodes)

Step 2: Generate SSH keys

Create a separate SSH key for each server:

Ansible Playbooks Reference

Mon, 01 Jan 0001 00:00:00 +0000

All playbooks are in ansible/playbooks/ and target the k8s_cluster host group by default.

update-system.yml

Purpose: Run apt update && apt upgrade on all nodes.

Usage:

cd ansible
ansible-playbook playbooks/update-system.yml
ansible-playbook playbooks/update-system.yml --limit control_nodes
ansible-playbook playbooks/update-system.yml -e "reboot_after_update=true"

Variables:

Variable	Default	Description
`reboot_after_update`	`false`	Reboot after update if the system requires it
`reboot_timeout`	`300`	Timeout (seconds) to wait for reboot

Tasks:

Update apt cache
Upgrade all packages (dist-upgrade with autoremove)
Check if reboot is required (/var/run/reboot-required)
Reboot if required and enabled
Wait for system to come back online

security-hardening.yml

Purpose: Apply security measures on all nodes.

Security

Mon, 01 Jan 0001 00:00:00 +0000

Design intent, not a security guarantee

This page describes what the design *intends* to protect against. KubeClaw is a [prototype](/kubeclaw/docs/project-status/) and has not been adversarially tested or audited -- assume gaps exist. Treat the table below as a statement of intent, not of assurance.

Security Summary

Layer	Protection
Network (Hetzner)	Firewall blocks all inbound; IPv6-only, no public IPv4
Network (K8s)	Cilium egress whitelist per namespace/app (FQDN-based)
Access	Cloudflare Tunnel (outbound-only connection, no open ports)
Authentication	Cloudflare Access policies + SSH key-only auth
Container	Non-root user, dropped capabilities, resource limits
SSH	Key-only auth, fail2ban, no TCP forwarding on workers
Storage	Isolated PVCs per workload

What this setup protects against

Direct server attacks – no public IPs, no open inbound ports
Unauthorized access – Cloudflare Access + SSH key-only auth
Data exfiltration – FQDN-based egress whitelist per application
Lateral movement – namespace isolation, per-pod network policies
Resource abuse – container resource limits

What to monitor

API key and token compromise – rotate regularly
Cloudflare Tunnel health – monitor via Zero Trust dashboard
Node resource utilization – watch for memory pressure on small instances

Security notes

Passwords in cloud-init are visible in cloud-init logs. Change them after first login.
SSH keys should be different for each server role.
Root password is only intended for emergency access via Hetzner Web Console.
Terraform state contains sensitive data (private keys when auto-generated). Protect state files.

UFW rules for HTTP/HTTPS on worker nodes can be removed after initial setup:

sudo ufw delete allow out to any port 80 proto tcp
sudo ufw delete allow out to any port 443 proto tcp

OpenClaw-specific security

When running OpenClaw:

Server Management (Ansible)

Mon, 01 Jan 0001 00:00:00 +0000

Ansible handles ongoing server management: system updates, security hardening, NAT64 configuration, and Kubernetes prerequisites. All playbooks run from the Dev Container.

Step 1: Set Up Ansible

1.1 Generate inventory

./scripts/generate-ansible-inventory.sh

The inventory (ansible/inventory.ini) is auto-generated. It defines three groups:

control_nodes – all control plane nodes
worker_nodes – all worker nodes
k8s_cluster – union of control and worker nodes

Warning

Never hand-edit `inventory.ini`. Regenerate after infrastructure changes with `./scripts/generate-ansible-inventory.sh`.

1.2 Load SSH keys

source ./scripts/ssh-agent-setup.sh

Important

The ssh-agent must be running with the cluster keys loaded before Ansible can connect. Run `source ./scripts/ssh-agent-setup.sh` in every new terminal session.

1.3 Test connectivity

cd ansible
ansible all -m ping

Step 2: Apply System Updates

ansible-playbook playbooks/update-system.yml

Target specific node groups or enable automatic reboots:

Kubernetes (kubeadm)

Mon, 01 Jan 0001 00:00:00 +0000

After provisioning the infrastructure with OpenTofu and configuring SSH access, deploy a standard Kubernetes cluster using kubeadm with Cilium as the CNI.

Why kubeadm + Cilium?

kubeadm: The official Kubernetes bootstrapper. Produces a standard, upstream cluster – exactly what the CKA exam expects. Full control over every component (etcd, kube-apiserver, kube-scheduler, kube-controller-manager).
Cilium: eBPF-based CNI providing advanced network policies with FQDN-based egress filtering – critical for restricting outbound traffic per namespace.

Step 1: Initialize the Control Plane (Dual-Stack)

SSH into the master control node. Prerequisites (containerd, kubeadm, kubelet, kubectl) are already installed via cloud-init when enable_k8s_prereqs = true (default).

OS Images

Mon, 01 Jan 0001 00:00:00 +0000

Hetzner Cloud offers the following Debian/Ubuntu images:

Image	Name	Recommendation
`debian-13`	Debian 13 Trixie	Recommended for K8s
`debian-12`	Debian 12 Bookworm	Stable, well-proven
`ubuntu-24.04`	Ubuntu 24.04 LTS	Good for K8s
`ubuntu-22.04`	Ubuntu 22.04 LTS	Well-proven

Why Debian 13?

Stability: long support cycles
Compatibility: the pkgs.k8s.io apt repository used for kubeadm, kubelet, and kubectl ships Debian packages
Small footprint: leaner than Ubuntu, but not as small as Alpine
No glibc/musl issues: unlike Alpine, no compatibility problems

Hetzner Cloud doesn’t provide dedicated “slim” or “minimal” variants. The standard images are fairly compact already.

Password Management

Mon, 01 Jan 0001 00:00:00 +0000

Which credentials exist?

Credential	Purpose	Storage
Hetzner API Token	Create infrastructure	`terraform.tfvars`
Root password	Emergency web console	`terraform.tfvars`
SSH private keys	Server access	`~/.ssh/` or password manager
Cloudflare Tunnel token	Tunnel auth	Cloudflare Dashboard

Recommended Dashlane structure

📁 K8s Cluster
├── 🔐 Hetzner API Token
│ └── Token: xxx
├── 🔐 Root Password
│ └── Password: xxx
├── 📝 SSH Keys (Secure Note)
│ ├── Control Node Private Key: ...
│ ├── Control Node Public Key: ...
│ ├── Worker Node Private Key: ...
│ └── Worker Node Public Key: ...
└── 🔐 Cloudflare Tunnel Token
 └── Token: xxx

Securing terraform.tfvars

terraform.tfvars contains sensitive data. Options:

OpenClaw Deployment

Mon, 01 Jan 0001 00:00:00 +0000

A step-by-step guide to deploying OpenClaw as an isolated AI assistant on the KubeClaw cluster, accessible via Telegram, WhatsApp, or Signal, with a web-based Control UI exposed through Cloudflare Tunnel.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐
│ Hetzner Private Network │
│ (10.0.0.0/24) │
│ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ Kubernetes Cluster (kubeadm) + Cilium │ │
│ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ control-01 │ │ worker-01 │ │ │
│ │ │ (control plane)│◄─────►│ (worker node) │ │ │
│ │ │ 10.0.0.2 │ │ 10.0.0.3 │ │ │
│ │ │ │ │ │ │ │
│ │ │ ┌───────────┐ │ │ ┌───────────┐ │ │ │
│ │ │ │Block Vol │ │ │ │Block Vol │ │ │ │
│ │ │ │ 10 GB │ │ │ │ 10 GB │ │ │ │
│ │ │ └───────────┘ │ │ └───────────┘ │ │ │
│ │ └────────┬────────┘ └─────────────────┘ │ │
│ │ │ │ │
│ │ system-unrestricted namespace: │ │
│ │ └─ cloudflared (egress: ANY) │ │
│ │ │ │
│ │ apps-restricted namespace: │ │
│ │ └─ OpenClaw (Cilium FQDN egress whitelist) │ │
│ │ ├─ Telegram / WhatsApp / Signal │ │
│ │ └─ Control UI (:18789) │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ cloudflared │
│ (outbound only) │
└──────────────────────────────┼──────────────────────────────────────┘
 │
 ▼
 ┌───────────────────┐
 │ Cloudflare │
 │ Edge + Access │
 └─────────┬─────────┘
 │
 ┌─────────┼─────────┐
 │ │ │
 ▼ ▼ ▼
 [Telegram] [WhatsApp] [Signal]
 │
 ▼
 [Control UI]
 (browser dashboard)

Prerequisites

A running KubeClaw cluster (see Kubernetes guide)
Anthropic API key (console.anthropic.com)
At least one messaging channel:

Channel	What You Need
Telegram	Bot Token from @BotFather, your Telegram User ID
WhatsApp	A phone number with WhatsApp, access to scan a QR code
Signal	A dedicated phone number, `signal-cli` installed in the container

Start with Telegram

Telegram is the simplest channel to set up — it only requires a bot token and user ID, with no interactive pairing or additional dependencies.

Step 1: Infrastructure Setup

If you haven’t already set up the Hetzner Cloud infrastructure, follow the Quick Start and Kubernetes guide.

Project Structure

Mon, 01 Jan 0001 00:00:00 +0000

kubeclaw/
├── main.tf # Infrastructure (providers, network, firewalls, servers)
├── variables.tf # All configurable inputs
├── outputs.tf # IPs, SSH keys, ssh_config_snippet, next-steps banner
├── terraform.tfvars.example # Example configuration
├── hugo.yaml # Hugo + Docsy configuration
├── package.json # Docsy asset dependencies
├── themes/docsy/ # Pinned Docsy theme submodule
├── aibox.toml # Dev Container source of truth (aibox apply)
├── aibox.lock # Pinned aibox toolchain versions
├── CLAUDE.md # Claude Code project instructions
├── README.md # Project overview (concise)
├── LICENSE # MIT License
├── .gitignore
├── .devcontainer/
│ ├── devcontainer.json # Generated by aibox; do not edit directly
│ ├── Dockerfile # Generated by aibox; do not edit directly
│ ├── docker-compose.yml # Generated by aibox; do not edit directly
│ ├── Dockerfile.local # Project layer: installs cloudflared
│ └── docker-compose.override.yml # Project override: writable SSH mount
├── cloud-init/
│ ├── admin-node.yaml.tpl # Admin node (temporary jump host with public IPv6)
│ ├── control-node.yaml.tpl # Control node (cloudflared on master, UFW, fail2ban, k8s prereqs)
│ └── worker-node.yaml.tpl # Worker node (isolated, outbound DNS/HTTP/S only, k8s prereqs)
├── scripts/
│ ├── setup-ssh.sh # Export SSH keys from tofu state, generate ~/.ssh/config
│ ├── ssh-agent-setup.sh # Fix SSH permissions, start ssh-agent, load keys
│ ├── generate-ansible-inventory.sh # Build ansible/inventory.ini from tofu state
│ ├── build-docs.sh # Build Hugo site locally
│ ├── serve-docs.sh # Preview Hugo site locally
│ └── deploy-docs.sh # Build and deploy Hugo to gh-pages
├── ansible/
│ ├── ansible.cfg # Ansible defaults (user, pipelining, SSH args)
│ ├── inventory.ini # Auto-generated inventory (do not hand-edit)
│ └── playbooks/
│ ├── update-system.yml # System updates with optional reboot
│ ├── security-hardening.yml # Unattended upgrades, fail2ban, sysctl hardening
│ ├── configure-nat64.yml # NAT64/DNS64 for IPv4 reachability on running nodes
│ └── prepare-k8s-nodes.yml # Kubernetes prerequisites (container runtime, kubeadm) on running nodes
├── assets/ # Project brand layer for the Hugo site
│ ├── icons/logo.svg # projectious mark used in the navbar
│ └── scss/ # Brand tokens and Docsy surface styles
├── layouts/ # Project template overrides (favicons, head hooks)
├── static/favicons/ # Brand favicon set
├── content/ # Hugo documentation source
│ ├── _index.md # Branded landing page
│ └── docs/ # Documentation section
│ ├── _index.md # Documentation overview
│ ├── quick-start.md # Combined prerequisites + deployment steps
│ ├── introduction/ # Concepts: architecture, security, DNS/NAT64
│ ├── guide/ # Ordered deployment path (dev container -> OpenClaw)
│ ├── how-to/ # Focused procedures, incl. the manual setup alternative
│ ├── reference/ # Variables, outputs, templates, playbooks, structure
│ ├── operations/ # Day-two: scaling, rotation, upgrades, troubleshooting
│ ├── roadmap/ # Planned work
│ └── contributing/ # Development setup and code of conduct
└── public/ # Generated site output (gitignored)

Key Files

Infrastructure (root level)

File	Purpose
`main.tf`	Core infrastructure: providers, SSH keys, network/subnet, firewalls, servers, cloud-init rendering
`variables.tf`	All configurable inputs with types, defaults, and descriptions
`outputs.tf`	Exposes IPs, SSH keys (sensitive), ssh_config_snippet, and next-steps banner
`terraform.tfvars.example`	Reference configuration (actual `.tfvars` is gitignored)

Cloud-Init Templates

File	Purpose
`cloud-init/admin-node.yaml.tpl`	Minimal jump host: admin user, SSH hardening, fail2ban, public SSH
`cloud-init/control-node.yaml.tpl`	Control plane: cloudflared (master only), UFW, fail2ban, optional NAT64 + K8s prereqs
`cloud-init/worker-node.yaml.tpl`	Worker: restrictive UFW, no TCP forwarding, optional NAT64 + K8s prereqs

Scripts

File	Purpose
`scripts/setup-ssh.sh`	Export SSH keys, generate `~/.ssh/config` with backup
`scripts/ssh-agent-setup.sh`	Fix SSH permissions, start ssh-agent, load keys
`scripts/generate-ansible-inventory.sh`	Build Ansible inventory from OpenTofu state
`scripts/build-docs.sh`	Build the Hugo site locally
`scripts/serve-docs.sh`	Preview the site locally
`scripts/deploy-docs.sh`	Build Hugo and push to `gh-pages`

Dev Container

The aibox configuration in aibox.toml is the source of truth for generated Dev Container files. Run aibox apply after changing it; do not hand-edit .devcontainer/Dockerfile, docker-compose.yml, or devcontainer.json. Dockerfile.local and docker-compose.override.yml are project-owned extensions and are safe to edit.

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Infrastructure Issues

cloudflared won’t start (IPv6-only)

Check /etc/cloudflared/config.yml:

edge-ip-version: "6"

Important

The value must be a string in quotes (`"6"`, not `6`). Cloudflared defaults to IPv4 connections to Cloudflare's edge servers.

SSH key rotation failed

Connect via Hetzner web console (root password)

Add the new key manually:

echo "ssh-ed25519 AAAA..." >> /home/kubernetes-admin/.ssh/authorized_keys

Ansible cannot connect

Check:

Is cloudflared installed locally?
Is the tunnel running? (cloudflared tunnel list)
Is the inventory correct? (./scripts/generate-ansible-inventory.sh)
Is the ssh-agent running with keys loaded? (ssh-add -l)

State lost / keys gone

With auto-generated keys:

Cost Estimate

Mon, 01 Jan 0001 00:00:00 +0000

All figures are list prices in EUR excluding VAT, current as of the Hetzner price adjustment of 15 June 2026. Hetzner bills hourly; the monthly figure is the cap you will not exceed. Always check Hetzner Cloud pricing before committing – prices changed several times during 2026.

Server types

KubeClaw defaults to the cost-optimized CX line (shared vCPU, x86). The CAX line offers the same resources on Ampere Arm cores, but is only available in the German and Finnish locations (fsn1, nbg1, hel1) and requires arm64 container images throughout the cluster.